[gmx-users] 35% Variability in Simulation Performance

[Kutzner, Carsten]

Hi Mitchell,

> On 30 Apr 2015, at 03:04, Mitchell Dorrell <mwd at udel.edu> wrote:
> 
> Hi all, I just ran the same simulation twice (ignore the difference in
> filenames), and got very different results. Obviously, I'd like to
> reproduce the faster simulation. I expect this probably has to do with
> automatically-tuned parameters, but I'm not sure what is really going on.
> 
> Links to the two log files:
> http://pastebin.com/Wq3uAyUv
> http://pastebin.com/sU5Qhs5h
> 
> Also, any tips for further boosting the performance?
there are several things you can try out here.

>From the log files I see that in one case the tuning of PME grid spacing
versus Coulomb cutoff goes on until after time step 2000. During this tuning
time, performance varies a lot and can be far from optimal. To get 
trustworthy performance numbers at the end of your run, you should therefore
exclude the first 3000 steps or so from the performance measurement using
the -resetstep 3000 command line parameter to mdrun (or, alternatively,
-resethway).

Another issue that is present in both simulations is that your 256 PME nodes
have way more work to do than the PP (direct-space) nodes. So you could try to increase
the fraction of PME nodes. Or try switching of dynamic load balancing, so
that the Coulomb cutoff can grow larger (and the PME grid smaller) during 
PME/PP load balancing. With your system you seem to be near the parallelization
limit, so it should help to use more than 2 OpenMP threads per MPI rank 
(thus a smaller number of MPI ranks). Try 4 or more, the reduced number
of PME MPI ranks will greatly reduce the PME communication. At the same time, the
PP domains will be larger, which is beneficial for load balancing.

Carsten


> 
> Thank you for your help!
> Mitchell Dorrell
> -- 
> Gromacs Users mailing list
> 
> * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!
> 
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> 
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a mail to gmx-users-request at gromacs.org.

--
Dr. Carsten Kutzner
Max Planck Institute for Biophysical Chemistry
Theoretical and Computational Biophysics
Am Fassberg 11, 37077 Goettingen, Germany
Tel. +49-551-2012313, Fax: +49-551-2012302
http://www.mpibpc.mpg.de/grubmueller/kutzner
http://www.mpibpc.mpg.de/grubmueller/sppexa