[gmx-users] Commands to run simulations using multiple GPU's in version 5.0.1

Wed Sep 24 16:50:49 CEST 2014

found it.
http://www.gromacs.org/Documentation/Acceleration_and_parallelization

GPUs are assigned to PP ranks within the same physical node in a sequential
order, that is GPU 0 to the (thread-)MPI rank 0, GPU 1 to rank 1. In order
to manually specify which GPU(s) to be used by mdrun, the respective device
ID(s) can be passed with the -gpu_id XYZ command line option or with the
GMX_GPU_ID=XYZ environment variable. Here, XYZ is a sequence of digits
representing the numeric ID-s of available GPUs (the numbering starts from
0) . The environment variable is particularly useful when running on
multiple compute nodes with different GPU configurations.

Taking the above example of 8-core machine with two compatible GPUs, we can
manually specify the GPUs and get the same launch configuration as in the
above examples by:

mdrun -ntmpi 2 -ntomp 4 -gpu_id 01

On Wed, Sep 24, 2014 at 10:49 AM, Johnny Lu <johnny.lu128 at gmail.com> wrote:

> Actually i am trying to find the answer to the same question now.
>
> manual 4.6.7/appendix D/mdrun
> says
>
> -gpu_id
> string
> List of GPU device id-s to use, specifies the per-node PP rank to GPU
> mapping
>
> On Tue, Sep 23, 2014 at 11:07 PM, Siva Dasetty <sdasett at g.clemson.edu>
> wrote:
>
>> Thank you Lu for the reply.
>>
>> As I have mentioned in the post, I have already tried those options but it
>> didn't work. Kindly please let me know if you have anymore suggestions.
>>
>> Thank you,
>>
>> On Tue, Sep 23, 2014 at 8:41 PM, Johnny Lu <johnny.lu128 at gmail.com>
>> wrote:
>>
>> > Try -nt, -ntmpi, -ntomp, -np  (one at a time) ?
>> > I forget about what I tried now.... But I just stop the mdrun, and then
>> > read the log file.
>> > Also can look for the mdrun page in the offical manual (pdf) and try
>> this
>> > page:
>> >
>> >
>> http://www.gromacs.org/Documentation/Gromacs_Utilities/mdrun?highlight=mdrun
>> >
>> >
>> > On Mon, Sep 22, 2014 at 6:46 PM, Siva Dasetty <sdasett at g.clemson.edu>
>> > wrote:
>> >
>> > > Dear  All,
>> > >
>> > > I am trying to run NPT simulations using GROMACS version 5.0.1 of a
>> > system
>> > > of size 140k atoms (protein+water systems) with 2 or more GPU's
>> > > (model=k20); 8 cores (or more); and 1 or more nodes. I am trying to
>> > > understand how to run simulations using multiple gpus on more than one
>> > > node. I  get the following errors/output when I run the simulation
>> using
>> > > the following commands:-
>> > >
>> > > Note: time-step used = 2 fs and total number of steps = 20000
>> > >
>> > > First 4 cases are using single GPU and cases 5-8 are using 2 GPU's.
>> > >
>> > > 1. 1 node, 8 cpus, 1 gpu
>> > > export OMP_NUM_THREADS = 8
>> > > command used-  mdrun -s topol.tpr  -gpu_id 0
>> > > Speed - 5.8 ns/day
>> > >
>> > > 2.  1 node, 8 cpus, 1 gpu
>> > > export OMP_NUM_THREADS = 16
>> > > command used-  mdrun -s topol.tpr   -gpu_id 0
>> > > Speed - 4.7 ns/day
>> > >
>> > > 3. 1 node, 8cpus, 1gpu
>> > > mdrun -s topol.tpr -ntomp 8  -gpu_id 0
>> > > Speed- 5.876 ns/day
>> > >
>> > > 4. 1 node, 8cpus, 1gpu
>> > > mdrun -s topol.tpr -ntomp 16  -gpu_id 0
>> > > Fatal Error: Environment variable OMP_NUM_THREADS (8) and the number
>> of
>> > > threads requested on  the command line (16) have different values.
>> Either
>> > > omit one, or set them both
>> > >  to the same value.
>> > >
>> > > Question for 3 and 4 : Do I need to always use OMP_NUM_THREADS or is
>> > there
>> > > a way ntomp overwrites the environment settings?
>> > >
>> > >
>> > > 5. 1 node, 8cpus , 2gpus
>> > > export OMP_NUM_THREADS = 8
>> > > mpirun -np 2 mdrun -s topol.tpr -pin on  -gpu_id 01
>> > > Speed - 4.044 ns/day
>> > >
>> > > 6. 2 nodes, 8cpus , 2 gpus
>> > > export OMP_NUM_THREADS = 8
>> > > mpirun -np 2 mdrun -s topol.tpr -pin on  -gpu_id 01
>> > > Speed - 3.0 ns/day
>> > >
>> > > Are the commands that I used for 5 and 6 correct?
>> > >
>> > > 7. I also used (1node, 8 cpus, 2 gpus)
>> > >  mdrun -s topol.tpr -ntmpi 2 -ntomp 8  -gpu_id 01
>> > > but this time I get a fatal error: thread mpi's are requested but
>> gromacs
>> > > is not compiled with thread MPI.
>> > >
>> > > Question: Isn't thread MPI enabled by default?
>> > >
>> > > 8. Finally, I recompiled Gromacs without OpenMP and re-ran case 1 but
>> > this
>> > > time there is a fatal error "More than 1 OpenMP thread requested, but
>> > > Gromacs was compiled without OpenMP support."
>> > > command : mdrun -s topol.tpr  (no environment settings)  -gpu_id 0
>> > > Question: Here again, I assumed thread MPI is enabled by default and I
>> > > think Gromacs still assumes OpenMp thread settings. Am i doing
>> something
>> > > wrong here?
>> > >
>> > > Thanks in advance for your help
>> > >
>> > > --
>> > > Siva
>> > > --
>> > > Gromacs Users mailing list
>> > >
>> > > * Please search the archive at
>> > > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
>> > > posting!
>> > >
>> > > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>> > >
>> > > * For (un)subscribe requests visit
>> > > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>> > > send a mail to gmx-users-request at gromacs.org.
>> > >
>> > --
>> > Gromacs Users mailing list
>> >
>> > * Please search the archive at
>> > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
>> > posting!
>> >
>> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>> >
>> > * For (un)subscribe requests visit
>> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>> > send a mail to gmx-users-request at gromacs.org.
>> >
>>
>>
>>
>> --
>> Siva
>> --
>> Gromacs Users mailing list
>>
>> * Please search the archive at
>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
>> posting!
>>
>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>
>> * For (un)subscribe requests visit
>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>> send a mail to gmx-users-request at gromacs.org.
>>
>
>