[gmx-users] GPU and MPI

Mon Sep 1 14:30:31 CEST 2014

I have added " -resethway" but still the same result. Use two GPU and 12
cores distributed in 2 nodes will result 33 ns/day, that is, it is about 3
time slower than MD run on one node (2GPU+12core).

I have no idea what is wrong.

dawei

On Mon, Sep 1, 2014 at 5:34 AM, Carsten Kutzner <ckutzne at gwdg.de> wrote:

> Hi,
>
> take a look at mdrun’s hidden but sometimes useful options:
>
> mdrun -h -hiddden
>
> Carsten
>
>
> On 01 Sep 2014, at 11:07, Oliver Schillinger <o.schillinger at fz-juelich.de>
> wrote:
>
> > Hi,
> > I did not know about the -resethway command line switch to mdrun.
> > Why is it nowhere documented?
> > Or am I blind/stupid?
> > Cheers,
> > Oliver
> >
> > On 08/29/2014 05:15 PM, Carsten Kutzner wrote:
> >> Hi Dawei,
> >>
> >> On 29 Aug 2014, at 16:52, Da-Wei Li <lidawei at gmail.com> wrote:
> >>
> >>> Dear Carsten
> >>>
> >>> Thanks for the clarification. Here it is my benchmark for a small
> protein
> >>> system (18k atoms).
> >>>
> >>> (1) 1 node (12 cores/node, no GPU):   50 ns/day
> >>> (2) 2 nodes (12 cores/node, no GPU): 80 ns/day
> >>> (3) 1 node (12 cores/node, 2 K40 GPUs/node): 100 ns/day
> >>> (4) 2 nodes (12 cores/node, 2 K40 GPUs/node): 40 ns/day
> >>>
> >>>
> >>> I send out this question because the benchmark 4 above is very
> suspicious.
> >> Indeed, if you get 80 ns/day without GPUs, then it should not be less
> >> with GPUs. For how many time steps do you run each of the
> >> benchmarks? Do you use the -resethway command line switch to mdrun
> >> to disregard the first half of the run (where initialization and
> >> balancing is done, you don’t want to count that in a benchmark)?
> >>
> >> Carsten
> >>
> >>> But I agree size of my system may play a role.
> >>>
> >>> best,
> >>>
> >>> dawei
> >>>
> >>>
> >>> On Fri, Aug 29, 2014 at 10:36 AM, Carsten Kutzner <ckutzne at gwdg.de>
> wrote:
> >>>
> >>>> Hi Dawei,
> >>>>
> >>>> the mapping of GPUs to PP ranks is printed for the Master node only,
> >>>> but if this node reports two GPUs, then all other PP ranks will also
> >>>> use two GPUs (or an error is reported).
> >>>>
> >>>> The scaling will depend also on your system size, if this is too
> small,
> >>>> then you might be better off by using a single node.
> >>>>
> >>>> Carsten
> >>>>
> >>>>
> >>>> On 29 Aug 2014, at 16:24, Da-Wei Li <lidawei at gmail.com> wrote:
> >>>>
> >>>>> Dear users,
> >>>>>
> >>>>> I recently try to run Gromacs on two nodes, each of them has 12 cores
> >>>> and 2
> >>>>> GPUs. The nodes are connected with infiniband and scaling is pretty
> good
> >>>>> when no GPU is evolved.
> >>>>>
> >>>>> My command is like this:
> >>>>>
> >>>>> mpiexec  -npernode 2 -np 4 mdrun_mpi -ntomp 6
> >>>>>
> >>>>>
> >>>>> However, it looks like Gromacs only detected 2 GPUs on node 0, then
> skip
> >>>>> node 1. Part of the output looks like:
> >>>>>
> >>>>>
> >>>>> ************************
> >>>>>
> >>>>> Using 4 MPI processes
> >>>>>
> >>>>> Using 6 OpenMP threads per MPI process
> >>>>>
> >>>>> 2 GPUs detected on host n0316.ten:
> >>>>>
> >>>>> #0: NVIDIA Tesla M2070, compute cap.: 2.0, ECC: yes, stat: compatible
> >>>>>
> >>>>> #1: NVIDIA Tesla M2070, compute cap.: 2.0, ECC: yes, stat: compatible
> >>>>>
> >>>>> 2 GPUs user-selected for this run.
> >>>>>
> >>>>> Mapping of GPUs to the 2 PP ranks in this node: #0, #1
> >>>>>
> >>>>> ****************************
> >>>>>
> >>>>>
> >>>>> The performance is about only 40% of the run, where I use only 1
> node (12
> >>>>> cores+2GPUs).
> >>>>>
> >>>>>
> >>>>> Does I miss something?
> >>>>>
> >>>>>
> >>>>> thanks.
> >>>>>
> >>>>>
> >>>>> dawei
> >>>>> --
> >>>>> Gromacs Users mailing list
> >>>>>
> >>>>> * Please search the archive at
> >>>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> >>>> posting!
> >>>>>
> >>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >>>>>
> >>>>> * For (un)subscribe requests visit
> >>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users
> or
> >>>> send a mail to gmx-users-request at gromacs.org.
> >>>>
> >>>> --
> >>>> Gromacs Users mailing list
> >>>>
> >>>> * Please search the archive at
> >>>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> >>>> posting!
> >>>>
> >>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >>>>
> >>>> * For (un)subscribe requests visit
> >>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> >>>> send a mail to gmx-users-request at gromacs.org.
> >>>>
> >>> --
> >>> Gromacs Users mailing list
> >>>
> >>> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
> >>>
> >>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >>>
> >>> * For (un)subscribe requests visit
> >>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
> >>
> >>
> >> --
> >> Dr. Carsten Kutzner
> >> Max Planck Institute for Biophysical Chemistry
> >> Theoretical and Computational Biophysics
> >> Am Fassberg 11, 37077 Goettingen, Germany
> >> Tel. +49-551-2012313, Fax: +49-551-2012302
> >> http://www.mpibpc.mpg.de/grubmueller/kutzner
> >> http://www.mpibpc.mpg.de/grubmueller/sppexa
> >>
> >
> > --
> > Oliver Schillinger
> > PhD student
> >
> > ICS-6 - Structural Biochemistry
> > Building 5.8v, Room 3010
> > Phone:  +49 2461-61-9532
> > Mobile: +49 172 53 27 914
> >
> > Forschungszentrum Juelich GmbH
> > 52425 Juelich
> > Sitz der Gesellschaft: Juelich
> > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
> > Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
> > Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender),
> > Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
> > Prof. Dr. Sebastian M. Schmidt
> > --
> > Gromacs Users mailing list
> >
> > * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
> >
> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >
> > * For (un)subscribe requests visit
> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
>
>
> --
> Dr. Carsten Kutzner
> Max Planck Institute for Biophysical Chemistry
> Theoretical and Computational Biophysics
> Am Fassberg 11, 37077 Goettingen, Germany
> Tel. +49-551-2012313, Fax: +49-551-2012302
> http://www.mpibpc.mpg.de/grubmueller/kutzner
> http://www.mpibpc.mpg.de/grubmueller/sppexa
>
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
>