[gmx-users] Gromacs and Software RDMA over Converged Ethernet -- is there a point ?

[Christopher Neale]

Dear Gromacs users:

I have access to a new cluster that has GigE interconnect (selected vs. IB for reasons other than cost). As expected, systems that scale nicely to two nodes with IB end up running faster on 1 node than they do in 2 nodes when using GigE. SysAdmins are wondering if software RoCE (Software RDMA over Converged Ethernet) will help. Anybody have any experience with this?

here is what the sysadmin said:

"
For large message sizes (>64k), SoftRoCE can provide performance comparable to hardware RoCE.  Latency improvements are more modest, ~50% better than straight ethernet but still about 3x higher than hardware RoCE.

Some references:

http://www.lanl.gov/projects/national-security-education-center/information-science-technology/_assets/docs/2010-si-docs/Team_CYAN_Implementation_and_Comparison_of_RDMA_Over_Ethernet_Presentation.pdf

http://www.iosrjournals.org/iosr-jce/papers/Vol15-issue4/N01548187.pdf?id=7557
"

I found this: http://quick.hcs.ufl.edu/pubs/UF_HPIDC.pdf but that is suggesting that there is a speedup when going to multiple nodes even for GigE and that is not what I see.

Thank you,
Chris.