[gmx-users] Different energies of GPU and CPU-only runs with Gromacs 5.x

[Mark Abraham]

On Mon, Sep 22, 2014 at 11:02 AM, Machtens, Jan-Philipp <
j.machtens at fz-juelich.de> wrote:

> Thanks for the reply!
> However, the differences between GPU runs and CPU-only runs for
> Gromacs 5.x that I observe are statistically significant (10-100 ns for
> each test sim) (and reproducible).
> In contrast, with Gromacs 4.6.x I do not have significant differences that
> I can apparently attribute to the use of GPUs.
>

OK. I would expect to observe a *total* energy that was constant across 4.6
and 5 and CPU-only vs with-GPUs. Please open an issue at
http://redmine.gromacs.org/ and describe as much detail as you can. A
tarball with .tpr, grompp-input, and whole .log files would let us analyze
and reproduce your observations and work out what is going on. Thanks!

Is there any other reason for the deviation with GPU and GROMAC 5.x ?


Not by design :-) All the relevant changes that I can think of either
removed code paths for old hardware, or tweaked performance (in various
kinds of ways). There were a bunch of fixes introduced into 4.6.6 and 5.0
that could be relevant, but we need the gory details to know.

Mark

Best,
> JP
> ________________________________________
> From: gromacs.org_gmx-users-bounces at maillist.sys.kth.se [
> gromacs.org_gmx-users-bounces at maillist.sys.kth.se] on behalf of Mark
> Abraham [mark.j.abraham at gmail.com]
> Sent: Monday, September 22, 2014 10:25 AM
> To: Discussion list for GROMACS users
> Subject: Re: [gmx-users] Different energies of GPU and CPU-only runs with
> Gromacs 5.x
>
> On Mon, Sep 22, 2014 at 9:44 AM, Machtens, Jan-Philipp <
> j.machtens at fz-juelich.de> wrote:
>
> > Dear all,
> > if I compare GPU-accelerated and CPU-only runs, I get significantly
> > different results/energies with
> > Gromacs 5.x. In contrast, with Gromacs 4.x everything is fine, i.e. no
> > difference between CPU-only runs and with GPU.
> >
> > In short, with Gromacs 5.x using GPUs, I have lower total energy, higher
> > short-range LJ, higher short-range Coulomb and
> > lower reciprocal Coulomb values.
>
>
> This is a normal consequence of different PME load balancing between CPU
> and GPU. You can use the UNIX diff tool to observe that mdrun is making
> different decisions.
>
> This also leads to altered density and box dimensions, etc....
> >
>
> MD trajectories are generally not reproducible even if the PME load is
> balanced the same way. Do replicates show the same kind of variation?
>
> Mark
>
> The simulations were run on a single node with either 2x GeForce GTX 780 Ti
> > or with 1x GTX 770.
> > Compilers tested: gcc 4.8 and icc 14
> > Cuda Versions: 6.0 or 6.5
> > ForceField: Amber99sb-ildn
> >
> > The topol.tpr was identical for each simulation. Verlet cut-off scheme
> was
> > used.
> > The mdp file can be accessed here:
> > https://www.dropbox.com/s/kjkglu2p8p11ajn/parameter.mdp?dl=0
> > Regression tests of the various compilations were always passed.
> >
> > Does someone has the some problems or has a clue what might cause the
> > problems I have with Gromacs5.x when using GPUs ?
> > Many thanks!
> > Best,
> > Jan-Philipp Machtens
> > Postdoctoral Research Fellow
> > Institute of Complex Systems - Cellular Biophysics (ICS-4)
> > Forschungszentrum Jülich, Germany
> >
> >
> >
> >
> ------------------------------------------------------------------------------------------------
> >
> >
> ------------------------------------------------------------------------------------------------
> > Forschungszentrum Juelich GmbH
> > 52425 Juelich
> > Sitz der Gesellschaft: Juelich
> > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
> > Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
> > Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
> > Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
> > Prof. Dr. Sebastian M. Schmidt
> >
> >
> ------------------------------------------------------------------------------------------------
> >
> >
> ------------------------------------------------------------------------------------------------
> >
> > --
> > Gromacs Users mailing list
> >
> > * Please search the archive at
> > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> > posting!
> >
> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >
> > * For (un)subscribe requests visit
> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> > send a mail to gmx-users-request at gromacs.org.
> >
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
>