[gmx-users] GPU and MPI

[Carsten Kutzner]

Hi,

take a look at mdrun’s hidden but sometimes useful options:

mdrun -h -hiddden

Carsten


On 01 Sep 2014, at 11:07, Oliver Schillinger <o.schillinger at fz-juelich.de> wrote:

> Hi,
> I did not know about the -resethway command line switch to mdrun.
> Why is it nowhere documented?
> Or am I blind/stupid?
> Cheers,
> Oliver
> 
> On 08/29/2014 05:15 PM, Carsten Kutzner wrote:
>> Hi Dawei,
>> 
>> On 29 Aug 2014, at 16:52, Da-Wei Li <lidawei at gmail.com> wrote:
>> 
>>> Dear Carsten
>>> 
>>> Thanks for the clarification. Here it is my benchmark for a small protein
>>> system (18k atoms).
>>> 
>>> (1) 1 node (12 cores/node, no GPU):   50 ns/day
>>> (2) 2 nodes (12 cores/node, no GPU): 80 ns/day
>>> (3) 1 node (12 cores/node, 2 K40 GPUs/node): 100 ns/day
>>> (4) 2 nodes (12 cores/node, 2 K40 GPUs/node): 40 ns/day
>>> 
>>> 
>>> I send out this question because the benchmark 4 above is very suspicious.
>> Indeed, if you get 80 ns/day without GPUs, then it should not be less
>> with GPUs. For how many time steps do you run each of the
>> benchmarks? Do you use the -resethway command line switch to mdrun
>> to disregard the first half of the run (where initialization and
>> balancing is done, you don’t want to count that in a benchmark)?
>> 
>> Carsten
>> 
>>> But I agree size of my system may play a role.
>>> 
>>> best,
>>> 
>>> dawei
>>> 
>>> 
>>> On Fri, Aug 29, 2014 at 10:36 AM, Carsten Kutzner <ckutzne at gwdg.de> wrote:
>>> 
>>>> Hi Dawei,
>>>> 
>>>> the mapping of GPUs to PP ranks is printed for the Master node only,
>>>> but if this node reports two GPUs, then all other PP ranks will also
>>>> use two GPUs (or an error is reported).
>>>> 
>>>> The scaling will depend also on your system size, if this is too small,
>>>> then you might be better off by using a single node.
>>>> 
>>>> Carsten
>>>> 
>>>> 
>>>> On 29 Aug 2014, at 16:24, Da-Wei Li <lidawei at gmail.com> wrote:
>>>> 
>>>>> Dear users,
>>>>> 
>>>>> I recently try to run Gromacs on two nodes, each of them has 12 cores
>>>> and 2
>>>>> GPUs. The nodes are connected with infiniband and scaling is pretty good
>>>>> when no GPU is evolved.
>>>>> 
>>>>> My command is like this:
>>>>> 
>>>>> mpiexec  -npernode 2 -np 4 mdrun_mpi -ntomp 6
>>>>> 
>>>>> 
>>>>> However, it looks like Gromacs only detected 2 GPUs on node 0, then skip
>>>>> node 1. Part of the output looks like:
>>>>> 
>>>>> 
>>>>> ************************
>>>>> 
>>>>> Using 4 MPI processes
>>>>> 
>>>>> Using 6 OpenMP threads per MPI process
>>>>> 
>>>>> 2 GPUs detected on host n0316.ten:
>>>>> 
>>>>> #0: NVIDIA Tesla M2070, compute cap.: 2.0, ECC: yes, stat: compatible
>>>>> 
>>>>> #1: NVIDIA Tesla M2070, compute cap.: 2.0, ECC: yes, stat: compatible
>>>>> 
>>>>> 2 GPUs user-selected for this run.
>>>>> 
>>>>> Mapping of GPUs to the 2 PP ranks in this node: #0, #1
>>>>> 
>>>>> ****************************
>>>>> 
>>>>> 
>>>>> The performance is about only 40% of the run, where I use only 1 node (12
>>>>> cores+2GPUs).
>>>>> 
>>>>> 
>>>>> Does I miss something?
>>>>> 
>>>>> 
>>>>> thanks.
>>>>> 
>>>>> 
>>>>> dawei
>>>>> --
>>>>> Gromacs Users mailing list
>>>>> 
>>>>> * Please search the archive at
>>>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
>>>> posting!
>>>>> 
>>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>>>> 
>>>>> * For (un)subscribe requests visit
>>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>>>> send a mail to gmx-users-request at gromacs.org.
>>>> 
>>>> --
>>>> Gromacs Users mailing list
>>>> 
>>>> * Please search the archive at
>>>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
>>>> posting!
>>>> 
>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>>> 
>>>> * For (un)subscribe requests visit
>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>>>> send a mail to gmx-users-request at gromacs.org.
>>>> 
>>> --
>>> Gromacs Users mailing list
>>> 
>>> * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!
>>> 
>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>> 
>>> * For (un)subscribe requests visit
>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a mail to gmx-users-request at gromacs.org.
>> 
>> 
>> --
>> Dr. Carsten Kutzner
>> Max Planck Institute for Biophysical Chemistry
>> Theoretical and Computational Biophysics
>> Am Fassberg 11, 37077 Goettingen, Germany
>> Tel. +49-551-2012313, Fax: +49-551-2012302
>> http://www.mpibpc.mpg.de/grubmueller/kutzner
>> http://www.mpibpc.mpg.de/grubmueller/sppexa
>> 
> 
> -- 
> Oliver Schillinger
> PhD student
> 
> ICS-6 - Structural Biochemistry
> Building 5.8v, Room 3010
> Phone:  +49 2461-61-9532
> Mobile: +49 172 53 27 914
> 
> Forschungszentrum Juelich GmbH
> 52425 Juelich
> Sitz der Gesellschaft: Juelich
> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
> Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
> Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender),
> Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
> Prof. Dr. Sebastian M. Schmidt
> -- 
> Gromacs Users mailing list
> 
> * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!
> 
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> 
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a mail to gmx-users-request at gromacs.org.


--
Dr. Carsten Kutzner
Max Planck Institute for Biophysical Chemistry
Theoretical and Computational Biophysics
Am Fassberg 11, 37077 Goettingen, Germany
Tel. +49-551-2012313, Fax: +49-551-2012302
http://www.mpibpc.mpg.de/grubmueller/kutzner
http://www.mpibpc.mpg.de/grubmueller/sppexa