[gmx-users] Timing variability

[chris.neale at utoronto.ca]

Dear Users:

Has anybody else looked at simulation speed (ns/day) over the segments  
of long runs? I always benchmark and optimize my systems carefully,  
but it was only recently that I realized how much variability I am  
obtaining over long runs. Perhaps this is specific to my cluster,  
which is one reason for my post.

Here is the performance (in ns/day) that I obtain with a single run.  
This is representative of what I see with 30 other runs (see list  
below). First, the distribution is bimodal, with peaks at around 80  
and 90 ns/day. Second, the values go as low as 70 ns/day (and I have  
seen as low as 50 ns/day when I look through all 30 run directories  
that differ only by the position of the umbrella restraint).

I am using gromacs 4.5.3 and I run it like this:
mpirun mdrun_mpi -deffnm md1 -dlb yes -npme 16 -cpt 30 -maxh 47 -cpi  
md1.cpt -cpo md1.cpt -px coord.xvg -pf force.xvg -noappend

I have obtained similar behaviour also with another system.

I have attempted to correlate timing with the node on which the job  
runs, the output of the "^Grid:" in the .log file, the DD division of  
PME and real-space and the value of pme mesh/force, all to no avail. I  
have found one correlation whereby the very slow runs also indicate a  
high relative load for PME.

I suspect that it is some value that is being determined at run start  
time that is affecting my performance, but I am not sure what this  
could be. Perhaps the fourier-spacing is leading to different initial  
grids?

Thank you (timing information and my .mdp file follows),
Chris

########### Output the ns/day obtained by the run segments
$ grep Perf *log|awk '{print $4}'

78.286
82.345
81.573
83.418
92.423
90.863
85.833
91.131
91.820
71.246
76.844
91.805
92.037
85.702
92.251
89.706
88.590
89.381
90.446
81.142
76.365
76.968
76.037
79.286
79.895
79.047
78.273
79.406
78.018
78.645
78.172
80.255
81.032
81.047
77.414
78.414
80.167
79.278
80.892
82.796
81.300
77.392
71.350
73.134
76.519
75.879
80.684
81.076
87.821
90.064
88.409
80.803
88.435

########### My .mdp file

constraints = all-bonds
lincs-iter =  1
lincs-order =  6
constraint_algorithm =  lincs
integrator = sd
dt = 0.004
tinit = 1000000
init_step            =  0
nsteps = 1000000000
nstcomm = 1
nstxout = 1000000000
nstvout = 1000000000
nstfout = 1000000000
nstxtcout = 25000
nstenergy = 25000
nstlist = 5
nstlog=0 ; reduce log file size
ns_type = grid
rlist = 1
rcoulomb = 1
rvdw = 1
coulombtype = PME
ewald-rtol = 1e-5
optimize_fft = yes
fourierspacing = 0.12
fourier_nx = 0
fourier_ny = 0
fourier_nz = 0
pme_order = 4
tc_grps             =  System
tau_t               =  1.0
ld_seed             =  -1
ref_t = 300
gen_temp = 300
gen_vel = yes
unconstrained_start = no
gen_seed = -1
Pcoupl = berendsen
pcoupltype = semiisotropic
tau_p = 4 4
compressibility = 4.5e-5 4.5e-5
ref_p = 1.0 1.0

; COM PULLING
pull                     = umbrella
pull_geometry            = position
pull_dim                 = N N Y
pull_start               = no
pull_nstxout             = 250
pull_nstfout             = 250
pull_ngroups             = 1
pull_group0              = POPC
pull_pbcatom0            = 338
pull_group1              = KSC
pull_pbcatom1            = 0
pull_init1               = 0 0 0.0
pull_rate1               = 0
pull_k1                  = 3000.0
pull_vec1                = 0 0 0
;;;EOF