[gmx-users] No improvement in scaling on introducing flow control

[Carsten Kutzner]

himanshu khandelia wrote:
> Hi Carsten,
> 
> Thank you very much for the prompt reply.
> 
> I know very little  about network architecture, and therefore
> understand your explanation only partly. Based on what you say,
> however, would it be fair to conclude that on the quad core
> woodcrests, it will not be possible to improve scaleup without
> altering the network architecture hardware itself ? Do you think it
> will be worthwhile to test an all-to-all optimization at all ?
If my guess is right and bandwidth is the problem here, the patch will
not improve the scaling. Are the benchmarks made with 1 or 2 NICs/node?
If they are for 1 NIC/node then there should be no network congestion
for the case of 8 CPUs (=2 nodes). You could try a back-to-back
connection between two nodes to be absolutely shure that the rest of the
network (switch etc.) does not play a role. I would try that and repeat
the benchmark for 8 CPUs. See if you get a different value.

Regards,
  Carsten


> 
> Thank you,
> 
> -Himanshu
> 
> 
> 
> On 10/25/07, Carsten Kutzner <ckutzne at gwdg.de> wrote:
>> Hi Himanshu,
>>
>> maybe your problem is not even flow control, but the limited network
>> bandwidth which is shared among 4 CPUs in your case. I also have done
>> benchmarks on Woodcrests (2.33 GHz) and was not able to scale an 80000
>> atom system beyond 1 node with Gbit Ethernet. Looking in more detail,
>> the time gained by the additional 4 CPUs of a second node was exactly
>> balanced by the extra communication. I used only 1 network interface for
>> that benchmark, leaving effectively only 1/4 th of the bandwidth for
>> each CPU. Using two interfaces with OpenMPI did not double the network
>> performance on our cluster. In my tests nodes with 2 CPUs sharing one
>> NIC were faster than nodes with 4 CPUs sharing two NICs. Could be
>> on-node contention, since both interfaces probably end up on the same
>> bus internally.
>>
>> Regards,
>>   Carsten
>>
>>
>> himanshu khandelia wrote:
>>> Hi,
>>>
>>> We tried turning on switch control on our local cluster
>>> (www.dcsc.sdu.dk) but were unable to achieve any improvement in scale
>>> up whatsoever. I was wondering if you folks could shed light upon how
>>> we should go ahead with this. (We have not installed the all-to-all
>>> patch yet)
>>>
>>> The cluster architecture is as follows:
>>> ##########
>>> * Computing nodes
>>> 160x Dell PowerEdge 1950 1U rackmountable servers with 2 2,66Ghz Intel
>>> Woodcrest CPUs, 4 GB Ram, 2x160 GB HDD (7200rpm, 8 MB buffer,
>>> SATA150), 2x Gigabit Ethernet
>>> 40x Dell PowerEdge 1950 1U rackmountable servers with 2 2,66Ghz Intel
>>> Woodcrest CPUs, 8 GB Ram, 2x160 GB HDD (7200rpm, 8 MB buffer,
>>> SATA150), 2x Gigabit Ethernet
>>> ##########
>>> * Switches
>>> 9 D-link SR3324
>>> 2 D-link SRi3324
>>> The switches are organised in two stacks, each connected to the
>>> infrastracture switch with an 8 Gb/s LACP trunk.Firmware Build on the
>>> switches
>>> ##########
>>> * Firmware Build on the switches: 3.00-B16
>>> There are newer firmware builds available, but according to the update
>>> logs, there is not update on the IEEE flow control protocol in the new
>>> firmware
>>> ##########
>>> * Tests (were run using OPENMPI, not LAMMPI)
>>> DPPC-bilayer system of ~ 40000 atoms, with PME and cutoffs, 1fs time
>>> step. The scaleup data is as follows. We are also currently running
>>> some tests with larger systems.
>>>
>>> # Procs     nanoseconds/day     Scaleup
>>> 1              0.526                       1
>>> 2              1.0                           1.90
>>> 4              1.768                        3.36
>>> 8              1.089                        2.07
>>> 16            0.39                          0.74
>>>
>>> Any inputs will be very helpful, thank you
>>>
>>> Best,
>>>
>>> -himanshu
>>> _______________________________________________
>>> gmx-users mailing list    gmx-users at gromacs.org
>>> http://www.gromacs.org/mailman/listinfo/gmx-users
>>> Please search the archive at http://www.gromacs.org/search before posting!
>>> Please don't post (un)subscribe requests to the list. Use the
>>> www interface or send it to gmx-users-request at gromacs.org.
>>> Can't post? Read http://www.gromacs.org/mailing_lists/users.php
>> --
>> Dr. Carsten Kutzner
>> Max Planck Institute for Biophysical Chemistry
>> Theoretical and Computational Biophysics Department
>> Am Fassberg 11
>> 37077 Goettingen, Germany
>> Tel. +49-551-2012313, Fax: +49-551-2012302
>> http://www.mpibpc.mpg.de/research/dep/grubmueller/
>> http://www.gwdg.de/~ckutzne
>> _______________________________________________
>> gmx-users mailing list    gmx-users at gromacs.org
>> http://www.gromacs.org/mailman/listinfo/gmx-users
>> Please search the archive at http://www.gromacs.org/search before posting!
>> Please don't post (un)subscribe requests to the list. Use the
>> www interface or send it to gmx-users-request at gromacs.org.
>> Can't post? Read http://www.gromacs.org/mailing_lists/users.php
>>
> _______________________________________________
> gmx-users mailing list    gmx-users at gromacs.org
> http://www.gromacs.org/mailman/listinfo/gmx-users
> Please search the archive at http://www.gromacs.org/search before posting!
> Please don't post (un)subscribe requests to the list. Use the 
> www interface or send it to gmx-users-request at gromacs.org.
> Can't post? Read http://www.gromacs.org/mailing_lists/users.php

-- 
Dr. Carsten Kutzner
Max Planck Institute for Biophysical Chemistry
Theoretical and Computational Biophysics Department
Am Fassberg 11
37077 Goettingen, Germany
Tel. +49-551-2012313, Fax: +49-551-2012302
http://www.mpibpc.mpg.de/research/dep/grubmueller/
http://www.gwdg.de/~ckutzne