[gmx-users] Looking for the source of the H-REMD slowdown when decoupling a lot of atoms

[Szilárd Páll]

Chris,

I can suggest two possible tweaks, these won't do miracles, but may give
you a bit better performance.

* icc is better than gcc at optimizing the naiive free energy kernel code.
I observed in the past up to 1.5x faster free energy kernel performance
with icc 15 vs gcc 4.9 or so.

* The default nstlist heuristic/suggestion assumes fast non-bondeds. In
your case, depending on how long is the list buffer with the nstlist you
use (25?), you may be able to gain a bit of performance by shifting load
back to the search with a smaller nstlist, e.g. 10.

These two combined could give in best case 2x, I guess.

What I find slightly unusual in the logs you posted on redmine is the 3-4x
slowdown in PME and constraints (I'd expect ~2x in PME and less in
constraints), but it could be that this too is simply because most
interaction in the system are perturbed which trigger non-optimized
code-paths.

Cheers,
--
Szilárd

On Wed, Mar 30, 2016 at 3:05 PM, Mark Abraham <mark.j.abraham at gmail.com>
wrote:

> Hi,
>
> Yeah, unfortunately Michael's pretty much right - the free-energy kernel is
> currently that cousin nobody talks about (or to). It's essentially
> unchanged since GROMACS 4.0 days, except that the Verlet scheme has some
> kludge so that it can call the same kernel that the group scheme used to
> call. But it has none of the optimizations and also some simplifying
> pessimizations. The result is highly unsuitable for Chris's use case, but
> not horrible for the more normal case of perturbing a small part of a
> system. For now, I can only suggest trying a run with more than one OpenMP
> thread per rank, but there's nothing in the log file snippets that fills me
> with any hope that it would be noticeably faster.
>
> We have a pile of infrastructure built and in the works that will lead to
> being able to offer similarly optimized free-energy kernels, but they won't
> see the light of day until next year, I'm afraid. A set of sample .tprs at
> Redmine 742 would be most welcome, however - it's very good for us to know
> when/that we're optimizing a workflow someone actually wants to run, but
> currently has reason to avoid.
>
> Mark
>
> On Wed, Mar 30, 2016 at 8:05 AM Michael Shirts <mrshirts at gmail.com> wrote:
>
> > Hi, Chris: I'm pretty sure that it's because the nonbonded free
> > energies are much slower than the standard free energies.  You state:
> >
> > > I took a look at gmxlib/nonbonded/nb_free_energy.c in v.5.1.2, but I
> was
> > unable to find a function called "gmx_waste_time_here()" and beyond that
> I
> > was out of my depth.
> >
> > But it's much more the fact that the non-free energy nonbondeds are
> > SUPER optimized.
> >
> > I don't see a particularly viable way around this for now. The only
> > thing I can think of splitting the neighborlists into two force calls,
> > and scaling the forces and energies coming out of those.  That would
> > be a huge pain.
> >
> > On Tue, Mar 29, 2016 at 9:50 PM, Christopher Neale
> > <chris.neale at alum.utoronto.ca> wrote:
> > > Dear Users:
> > >
> > > I am trying to do some Hamiltonian replica exchange (H-REMD) in gromacs
> > 5.1.2 and am running up against really large slowdowns when decoupling a
> > large number of atoms. I am decoupling 5360 atoms out of the 15520 atoms
> in
> > my system. The goal is not to get a PMF, but to enhance sampling using
> the
> > REST approach to partially decouple lipids in a bilayer. This approach
> > enhances lipid relaxation times (
> > http://pubs.acs.org/doi/pdf/10.1021/ct500305u ) though the authors of
> > that paper modified the gromacs code to do their own H-REMD in order to
> > avoid the really slow speed they also got when decoupling lots of atoms
> via
> > the free energy code.
> > >
> > > I have already posted details here
> http://redmine.gromacs.org/issues/742
> > , which includes .mdp options and some timing output. I compare the
> timing
> > output to a standard temperature REMD (T-REMD) run. For my usage, the
> > slowdown is about 12x for H-REMD vs. T-REMD.
> > >
> > > I am motivated to find a solution within gromacs because the
> alternative
> > is to use gromacs 4.6.7 with plumed (or with the aforementioned modified
> > code, which is also gromacs v4). Normally that would be a viable option,
> > but I am using the charmm force field and the charmm TIP3P water and I
> > would rather not give up the speed boost that I see in gromacs v5.1.2,
> > which allows the use of the verlet cutoff scheme and has been tested and
> > shown to give the correct reproduction of charmm forces (vs. the forces
> one
> > would get using the charmm simulation software).
> > >
> > > I took a look at gmxlib/nonbonded/nb_free_energy.c in v.5.1.2, but I
> was
> > unable to find a function called "gmx_waste_time_here()" and beyond that
> I
> > was out of my depth.
> > >
> > > Thank you for any pointers,
> > > Chris.
> > >
> > >
> > > --
> > > Gromacs Users mailing list
> > >
> > > * Please search the archive at
> > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> > posting!
> > >
> > > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> > >
> > > * For (un)subscribe requests visit
> > > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> > send a mail to gmx-users-request at gromacs.org.
> > --
> > Gromacs Users mailing list
> >
> > * Please search the archive at
> > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> > posting!
> >
> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >
> > * For (un)subscribe requests visit
> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> > send a mail to gmx-users-request at gromacs.org.
> >
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
>