[gmx-users] Gromacs 2018 and GPU PME

[Daniel Kozuch]

Szilárd,

If I may jump in on this conversation, I am having the reverse problem
(which I assume others may encounter also) where I am attempting a large
REMD run (84 replicas) and I have access to say 12 GPUs and 84 CPUs.

Basically I have less GPUs than simulations. Is there a logical approach to
using gputasks and other new options in GROMACS 2018 for this setup? I read
through the available documentation,but as you mentioned it seems to be
targeted for a single-GPU runs.

Thanks so much,
Dan



On Fri, Feb 9, 2018 at 10:27 AM, Szilárd Páll <pall.szilard at gmail.com>
wrote:

> On Fri, Feb 9, 2018 at 4:25 PM, Szilárd Páll <pall.szilard at gmail.com>
> wrote:
>
> > Hi,
> >
> > First of all,have you read the docs (admittedly somewhat brief):
> > http://manual.gromacs.org/documentation/2018/user-guide/
> > mdrun-performance.html#types-of-gpu-tasks
> >
> > The current PME GPU was optimized for single-GPU runs. Using multiple
> GPUs
> > with PME offloaded works, but this mode hasn't been an optimization
> target
> > and it will often not give very good performance. Using multiple GPUs
> > requires a separate PME rank (as you have realized), only one can be used
> > (as we don't support PME decomposition on GPUs) and it comes some
> > inherent scaling drawbacks. For this reason, unless you _need_ your
> single
> > run to be as fast as possible, you'll be better off running multiple
> > simulations side-by side.
> >
>
> PS: You can of course also run on two GPUs and run two simulations
> side-by-side (on half of the cores for each) to improve the overall
> aggregate throughput you get out of the hardware.
>
>
> >
> > A few tips for tuning the performance of a multi-GPU run with PME
> offload:
> > * expect to get at best 1.5 scaling to 2 GPUs (rarely 3 if the tasks
> allow)
> > * generally it's best to use about the same decomposition that you'd use
> > with nonbonded-only offload, e.g. in your case 6-8 ranks
> > * map the GPU task alone or at most together with 1 PP rank to a GPU,
> i.e.
> > use the new -gputasks option
> > e.g. for your case I'd expect the following to work ~best:
> > gmx mdrun -v -deffnm md -pme gpu -nb gpu -ntmpi 8 -ntomp 6 -npme 1
> > -gputasks 00000001
> > or
> > gmx mdrun -v -deffnm md -pme gpu -nb gpu -ntmpi 8 -ntomp 6 -npme 1
> > -gputasks 00000011
> >
> >
> > Let me know if that gave some improvement.
> >
> > Cheers,
> >
> > --
> > Szilárd
> >
> > On Fri, Feb 9, 2018 at 8:51 AM, Gmx QA <gmxquestions at gmail.com> wrote:
> >
> >> Hi list,
> >>
> >> I am trying out the new gromacs 2018 (really nice so far), but have a
> few
> >> questions about what command line options I should specify, specifically
> >> with the new gnu pme implementation.
> >>
> >> My computer has two CPUs (with 12 cores each, 24 with hyper threading)
> and
> >> two GPUs, and I currently (with 2018) start simulations like this:
> >>
> >> $ gmx mdrun -v -deffnm md -pme gpu -nb gpu -ntmpi 2 -npme 1 -ntomp 24
> >> -gpu_id 01
> >>
> >> this works, but gromacs prints the message that 24 omp threads per mpi
> >> rank
> >> is likely inefficient. However, trying to reduce the number of omp
> threads
> >> I see a reduction in performance. Is this message no longer relevant
> with
> >> gpu pme or am I overlooking something?
> >>
> >> Thanks
> >> /PK
> >> --
> >> Gromacs Users mailing list
> >>
> >> * Please search the archive at http://www.gromacs.org/Support
> >> /Mailing_Lists/GMX-Users_List before posting!
> >>
> >> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >>
> >> * For (un)subscribe requests visit
> >> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> >> send a mail to gmx-users-request at gromacs.org.
> >>
> >
> >
> --
> Gromacs Users mailing list
>
> * Please search the archive at http://www.gromacs.org/
> Support/Mailing_Lists/GMX-Users_List before posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
>