[gmx-users] Improving scaling - Gromacs 4.0 RC2

[David van der Spoel]

Justin A. Lemkul wrote:
> 
> Hi,
> 
> I've been playing around with the latest release candidate of version 
> 4.0, and I was hoping someone out there more knowledgeable than me might 
> tell me how to improve a bit on the performance I'm seeing.  To clarify, 
> the performance I'm seeing is a ton faster than 3.3.x, but I still seem 
> to be getting bogged down with the PME/PP balance.  I'm using mostly the 
> default options with the new mdrun:
> 
> mdrun_mpi -s test.tpr -np 64 -npme 32

try -dlb auto


> 
> The system contains about 150,000 atoms - a membrane protein surrounded 
> by several hundred lipids and solvent (water).  The protein parameters 
> are GROMOS, lipids are Berger, and water is SPC.  My .mdp file (adapted 
> from a generic 3.3.x file that I always used to use for such 
> simulations) is attached at the end of this mail.  It seems that my 
> system runs fastest on 64 CPU's.  Almost all tests with 128 or 256 seem 
> to run slower.  The nodes are dual-core 2.3 GHz Xserve G5, connected by 
> Infiniband.
> 
> Here's a summary of some of the tests I've run:
> 
> -np    -npme    -ddorder    ns/day    % performance loss from imbalance
> 64    16    interleave    5.760    19.6
> 64    32    interleave    9.600    40.9
> 64    32    pp_pme        5.252    3.9
> 64    32    cartesian    5.383    4.7
> 
> All other mdrun command line options are defaults.
> 
> I get ~10.3 ns/day with -np 256 -npme 64, but since -np 64 -npme 32 
> seems to give almost that same performance there seems to be no 
> compelling reason to tie up that many nodes.
> 
> Any hints on how to speed things up any more?  Is it possible?  Not that 
> I'm complaining...the same system under GMX 3.3.3 gives just under 1 
> ns/day :)  I'm really curious about the 40.9% performance loss I'm 
> seeing with -np 64 -npme 32, even though it gives the best overall 
> performance in terms of ns/day.
> 
> Thanks in advance for your attention, and any comments.
> 
> -Justin
> 
> =======test.mdp=========
> title        = NPT simulation for a membrane protein
> ; Run parameters
> integrator    = md
> dt        = 0.002
> nsteps        = 10000        ; 20 ps
> nstcomm        = 1
> ; Output parameters
> nstxout        = 500
> nstvout        = 500
> nstfout        = 500
> nstlog        = 500
> nstenergy    = 500
> ; Bond parameters
> constraint_algorithm     = lincs
> constraints        = all-bonds
> continuation     = no        ; starting up
> ; Twin-range cutoff scheme, parameters for Gromos96
> nstlist        = 5
> ns_type        = grid
> rlist        = 0.8
> rcoulomb    = 0.8
> rvdw        = 1.4
> ; PME electrostatics parameters
> coulombtype    = PME
> fourierspacing  = 0.24
> pme_order    = 4
> ewald_rtol    = 1e-5
> optimize_fft    = yes
> ; V-rescale temperature coupling is on in three groups
> Tcoupl         = V-rescale
> tc_grps        = Protein POPC SOL_NA+_CL-
> tau_t        = 0.1 0.1 0.1
> ref_t        = 310 310 310
> ; Pressure coupling is on
> Pcoupl        = Berendsen
> pcoupltype    = semiisotropic
> tau_p        = 2.0       
> compressibility    = 4.5e-5 4.5e-5
> ref_p        = 1.0 1.0
> ; Generate velocities is on
> gen_vel        = yes       
> gen_temp    = 310
> gen_seed    = 173529
> ; Periodic boundary conditions are on in all directions
> pbc        = xyz
> ; Long-range dispersion correction
> DispCorr    = EnerPres
> 
> ========end test.mdp==========
> 


-- 
David.
________________________________________________________________________
David van der Spoel, PhD, Professor of Biology
Dept. of Cell and Molecular Biology, Uppsala University.
Husargatan 3, Box 596,  	75124 Uppsala, Sweden
phone:	46 18 471 4205		fax: 46 18 511 755
spoel at xray.bmc.uu.se	spoel at gromacs.org   http://folding.bmc.uu.se
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++