[gmx-users] GPU and MPI
Da-Wei Li
lidawei at gmail.com
Mon Sep 1 15:58:26 CEST 2014
No. With GPU, both domain and PME domain are decomposited by 4X1X1, because
I use 4 MPI ranks, in line with 4 GPUs. W/o GPU, domain decomposition is
20X1X1 and PME is 4X1X1.
dawei
On Mon, Sep 1, 2014 at 8:39 AM, Carsten Kutzner <ckutzne at gwdg.de> wrote:
> Hi Dawei,
>
> on two nodes, regarding the cases with and without GPUs,
> do you use the same domain decomposition in both cases?
>
> Carsten
>
>
> On 01 Sep 2014, at 14:30, Da-Wei Li <lidawei at gmail.com> wrote:
>
> > I have added " -resethway" but still the same result. Use two GPU and 12
> > cores distributed in 2 nodes will result 33 ns/day, that is, it is about
> 3
> > time slower than MD run on one node (2GPU+12core).
> >
> > I have no idea what is wrong.
> >
> >
> > dawei
> >
> >
> > On Mon, Sep 1, 2014 at 5:34 AM, Carsten Kutzner <ckutzne at gwdg.de> wrote:
> >
> >> Hi,
> >>
> >> take a look at mdrun’s hidden but sometimes useful options:
> >>
> >> mdrun -h -hiddden
> >>
> >> Carsten
> >>
> >>
> >> On 01 Sep 2014, at 11:07, Oliver Schillinger <
> o.schillinger at fz-juelich.de>
> >> wrote:
> >>
> >>> Hi,
> >>> I did not know about the -resethway command line switch to mdrun.
> >>> Why is it nowhere documented?
> >>> Or am I blind/stupid?
> >>> Cheers,
> >>> Oliver
> >>>
> >>> On 08/29/2014 05:15 PM, Carsten Kutzner wrote:
> >>>> Hi Dawei,
> >>>>
> >>>> On 29 Aug 2014, at 16:52, Da-Wei Li <lidawei at gmail.com> wrote:
> >>>>
> >>>>> Dear Carsten
> >>>>>
> >>>>> Thanks for the clarification. Here it is my benchmark for a small
> >> protein
> >>>>> system (18k atoms).
> >>>>>
> >>>>> (1) 1 node (12 cores/node, no GPU): 50 ns/day
> >>>>> (2) 2 nodes (12 cores/node, no GPU): 80 ns/day
> >>>>> (3) 1 node (12 cores/node, 2 K40 GPUs/node): 100 ns/day
> >>>>> (4) 2 nodes (12 cores/node, 2 K40 GPUs/node): 40 ns/day
> >>>>>
> >>>>>
> >>>>> I send out this question because the benchmark 4 above is very
> >> suspicious.
> >>>> Indeed, if you get 80 ns/day without GPUs, then it should not be less
> >>>> with GPUs. For how many time steps do you run each of the
> >>>> benchmarks? Do you use the -resethway command line switch to mdrun
> >>>> to disregard the first half of the run (where initialization and
> >>>> balancing is done, you don’t want to count that in a benchmark)?
> >>>>
> >>>> Carsten
> >>>>
> >>>>> But I agree size of my system may play a role.
> >>>>>
> >>>>> best,
> >>>>>
> >>>>> dawei
> >>>>>
> >>>>>
> >>>>> On Fri, Aug 29, 2014 at 10:36 AM, Carsten Kutzner <ckutzne at gwdg.de>
> >> wrote:
> >>>>>
> >>>>>> Hi Dawei,
> >>>>>>
> >>>>>> the mapping of GPUs to PP ranks is printed for the Master node only,
> >>>>>> but if this node reports two GPUs, then all other PP ranks will also
> >>>>>> use two GPUs (or an error is reported).
> >>>>>>
> >>>>>> The scaling will depend also on your system size, if this is too
> >> small,
> >>>>>> then you might be better off by using a single node.
> >>>>>>
> >>>>>> Carsten
> >>>>>>
> >>>>>>
> >>>>>> On 29 Aug 2014, at 16:24, Da-Wei Li <lidawei at gmail.com> wrote:
> >>>>>>
> >>>>>>> Dear users,
> >>>>>>>
> >>>>>>> I recently try to run Gromacs on two nodes, each of them has 12
> cores
> >>>>>> and 2
> >>>>>>> GPUs. The nodes are connected with infiniband and scaling is pretty
> >> good
> >>>>>>> when no GPU is evolved.
> >>>>>>>
> >>>>>>> My command is like this:
> >>>>>>>
> >>>>>>> mpiexec -npernode 2 -np 4 mdrun_mpi -ntomp 6
> >>>>>>>
> >>>>>>>
> >>>>>>> However, it looks like Gromacs only detected 2 GPUs on node 0, then
> >> skip
> >>>>>>> node 1. Part of the output looks like:
> >>>>>>>
> >>>>>>>
> >>>>>>> ************************
> >>>>>>>
> >>>>>>> Using 4 MPI processes
> >>>>>>>
> >>>>>>> Using 6 OpenMP threads per MPI process
> >>>>>>>
> >>>>>>> 2 GPUs detected on host n0316.ten:
> >>>>>>>
> >>>>>>> #0: NVIDIA Tesla M2070, compute cap.: 2.0, ECC: yes, stat:
> compatible
> >>>>>>>
> >>>>>>> #1: NVIDIA Tesla M2070, compute cap.: 2.0, ECC: yes, stat:
> compatible
> >>>>>>>
> >>>>>>> 2 GPUs user-selected for this run.
> >>>>>>>
> >>>>>>> Mapping of GPUs to the 2 PP ranks in this node: #0, #1
> >>>>>>>
> >>>>>>> ****************************
> >>>>>>>
> >>>>>>>
> >>>>>>> The performance is about only 40% of the run, where I use only 1
> >> node (12
> >>>>>>> cores+2GPUs).
> >>>>>>>
> >>>>>>>
> >>>>>>> Does I miss something?
> >>>>>>>
> >>>>>>>
> >>>>>>> thanks.
> >>>>>>>
> >>>>>>>
> >>>>>>> dawei
> >>>>>>> --
> >>>>>>> Gromacs Users mailing list
> >>>>>>>
> >>>>>>> * Please search the archive at
> >>>>>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> >>>>>> posting!
> >>>>>>>
> >>>>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >>>>>>>
> >>>>>>> * For (un)subscribe requests visit
> >>>>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users
> >> or
> >>>>>> send a mail to gmx-users-request at gromacs.org.
> >>>>>>
> >>>>>> --
> >>>>>> Gromacs Users mailing list
> >>>>>>
> >>>>>> * Please search the archive at
> >>>>>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> >>>>>> posting!
> >>>>>>
> >>>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >>>>>>
> >>>>>> * For (un)subscribe requests visit
> >>>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users
> or
> >>>>>> send a mail to gmx-users-request at gromacs.org.
> >>>>>>
> >>>>> --
> >>>>> Gromacs Users mailing list
> >>>>>
> >>>>> * Please search the archive at
> >> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> >> posting!
> >>>>>
> >>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >>>>>
> >>>>> * For (un)subscribe requests visit
> >>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users
> or
> >> send a mail to gmx-users-request at gromacs.org.
> >>>>
> >>>>
> >>>> --
> >>>> Dr. Carsten Kutzner
> >>>> Max Planck Institute for Biophysical Chemistry
> >>>> Theoretical and Computational Biophysics
> >>>> Am Fassberg 11, 37077 Goettingen, Germany
> >>>> Tel. +49-551-2012313, Fax: +49-551-2012302
> >>>> http://www.mpibpc.mpg.de/grubmueller/kutzner
> >>>> http://www.mpibpc.mpg.de/grubmueller/sppexa
> >>>>
> >>>
> >>> --
> >>> Oliver Schillinger
> >>> PhD student
> >>>
> >>> ICS-6 - Structural Biochemistry
> >>> Building 5.8v, Room 3010
> >>> Phone: +49 2461-61-9532
> >>> Mobile: +49 172 53 27 914
> >>>
> >>> Forschungszentrum Juelich GmbH
> >>> 52425 Juelich
> >>> Sitz der Gesellschaft: Juelich
> >>> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
> >>> Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
> >>> Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender),
> >>> Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
> >>> Prof. Dr. Sebastian M. Schmidt
> >>> --
> >>> Gromacs Users mailing list
> >>>
> >>> * Please search the archive at
> >> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> >> posting!
> >>>
> >>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >>>
> >>> * For (un)subscribe requests visit
> >>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> >> send a mail to gmx-users-request at gromacs.org.
> >>
> >>
> >> --
> >> Dr. Carsten Kutzner
> >> Max Planck Institute for Biophysical Chemistry
> >> Theoretical and Computational Biophysics
> >> Am Fassberg 11, 37077 Goettingen, Germany
> >> Tel. +49-551-2012313, Fax: +49-551-2012302
> >> http://www.mpibpc.mpg.de/grubmueller/kutzner
> >> http://www.mpibpc.mpg.de/grubmueller/sppexa
> >>
> >> --
> >> Gromacs Users mailing list
> >>
> >> * Please search the archive at
> >> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> >> posting!
> >>
> >> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >>
> >> * For (un)subscribe requests visit
> >> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> >> send a mail to gmx-users-request at gromacs.org.
> >>
> > --
> > Gromacs Users mailing list
> >
> > * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
> >
> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >
> > * For (un)subscribe requests visit
> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
>
>
> --
> Dr. Carsten Kutzner
> Max Planck Institute for Biophysical Chemistry
> Theoretical and Computational Biophysics
> Am Fassberg 11, 37077 Goettingen, Germany
> Tel. +49-551-2012313, Fax: +49-551-2012302
> http://www.mpibpc.mpg.de/grubmueller/kutzner
> http://www.mpibpc.mpg.de/grubmueller/sppexa
>
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
>
More information about the gromacs.org_gmx-users
mailing list