[gmx-users] Commands to run simulations using multiple GPU's in version 5.0.1

Wed Sep 24 18:17:54 CEST 2014

On Wed, Sep 24, 2014 at 5:57 PM, Siva Dasetty <sdasett at g.clemson.edu> wrote:
> Thank you again for the reply.
>
> ntmpi is for threadMPI but I am using OpenMPI for MPI as I am planning to
> use multiple nodes.
> As I have pointed in case 7 of my post that if I use ntmpi, i get a fatal
> error that says :thread mpi's are requested but gromacs is not compiled
> with thread MPI.
>
> For which my questions are,
> 1. Isnt threadMPI enabled by default?

Yes it is, but you could have checked that by running cmake and then
checking the resulting cache or mdrun binary.

> 2. Are threadMPI and OpenMPI mutually incompatible?

Yes. Again, that's not hard to figure out either; e.g.
* from the wiki page on parallelization: "Acting as a drop-in
replacement for MPI, thread-MPI enables..."
* from the cmake output:
$ cmake -DGMX_MPI=ON .
-- MPI is not compatible with thread-MPI. Disabling thread-MPI.

> In any case if I use mpirun -np 2 instead of ntmpi, I still cannot use
> ntomp because gromacs now automatically detects environment settings for
> the OpenMP threads which is equal to the number of hardware threads that is
> available and this resulted in case 4 (please check above) of my post.

Your case 4) doesn't work simply because your environment contains
OMP_NUM_THREADS (set e.g. by your job script or scheduler) set to a
value different from what you pass to -ntomp. You should do exactly as
the error message says.

> Is there any other command similar to the one you posted above that I can
> use with OpenMPI? Because it looks like threadMPI and OpenMPI are not
> compatible to me.

The command line interface, with the exception of the "-ntmpi"
argument which is thread-MPI specific, is always the same.

--
Szilárd

> Thanks,
>
>
>
>
> On Wed, Sep 24, 2014 at 10:50 AM, Johnny Lu <johnny.lu128 at gmail.com> wrote:
>
>> found it.
>> http://www.gromacs.org/Documentation/Acceleration_and_parallelization
>>
>> GPUs are assigned to PP ranks within the same physical node in a sequential
>> order, that is GPU 0 to the (thread-)MPI rank 0, GPU 1 to rank 1. In order
>> to manually specify which GPU(s) to be used by mdrun, the respective device
>> ID(s) can be passed with the -gpu_id XYZ command line option or with the
>> GMX_GPU_ID=XYZ environment variable. Here, XYZ is a sequence of digits
>> representing the numeric ID-s of available GPUs (the numbering starts from
>> 0) . The environment variable is particularly useful when running on
>> multiple compute nodes with different GPU configurations.
>>
>> Taking the above example of 8-core machine with two compatible GPUs, we can
>> manually specify the GPUs and get the same launch configuration as in the
>> above examples by:
>>
>> mdrun -ntmpi 2 -ntomp 4 -gpu_id 01
>>
>>
>>
>> On Wed, Sep 24, 2014 at 10:49 AM, Johnny Lu <johnny.lu128 at gmail.com>
>> wrote:
>>
>> > Actually i am trying to find the answer to the same question now.
>> >
>> > manual 4.6.7/appendix D/mdrun
>> > says
>> >
>> > -gpu_id
>> > string
>> > List of GPU device id-s to use, specifies the per-node PP rank to GPU
>> > mapping
>> >
>> > On Tue, Sep 23, 2014 at 11:07 PM, Siva Dasetty <sdasett at g.clemson.edu>
>> > wrote:
>> >
>> >> Thank you Lu for the reply.
>> >>
>> >> As I have mentioned in the post, I have already tried those options but
>> it
>> >> didn't work. Kindly please let me know if you have anymore suggestions.
>> >>
>> >> Thank you,
>> >>
>> >> On Tue, Sep 23, 2014 at 8:41 PM, Johnny Lu <johnny.lu128 at gmail.com>
>> >> wrote:
>> >>
>> >> > Try -nt, -ntmpi, -ntomp, -np  (one at a time) ?
>> >> > I forget about what I tried now.... But I just stop the mdrun, and
>> then
>> >> > read the log file.
>> >> > Also can look for the mdrun page in the offical manual (pdf) and try
>> >> this
>> >> > page:
>> >> >
>> >> >
>> >>
>> http://www.gromacs.org/Documentation/Gromacs_Utilities/mdrun?highlight=mdrun
>> >> >
>> >> >
>> >> > On Mon, Sep 22, 2014 at 6:46 PM, Siva Dasetty <sdasett at g.clemson.edu>
>> >> > wrote:
>> >> >
>> >> > > Dear  All,
>> >> > >
>> >> > > I am trying to run NPT simulations using GROMACS version 5.0.1 of a
>> >> > system
>> >> > > of size 140k atoms (protein+water systems) with 2 or more GPU's
>> >> > > (model=k20); 8 cores (or more); and 1 or more nodes. I am trying to
>> >> > > understand how to run simulations using multiple gpus on more than
>> one
>> >> > > node. I  get the following errors/output when I run the simulation
>> >> using
>> >> > > the following commands:-
>> >> > >
>> >> > > Note: time-step used = 2 fs and total number of steps = 20000
>> >> > >
>> >> > > First 4 cases are using single GPU and cases 5-8 are using 2 GPU's.
>> >> > >
>> >> > > 1. 1 node, 8 cpus, 1 gpu
>> >> > > export OMP_NUM_THREADS = 8
>> >> > > command used-  mdrun -s topol.tpr  -gpu_id 0
>> >> > > Speed - 5.8 ns/day
>> >> > >
>> >> > > 2.  1 node, 8 cpus, 1 gpu
>> >> > > export OMP_NUM_THREADS = 16
>> >> > > command used-  mdrun -s topol.tpr   -gpu_id 0
>> >> > > Speed - 4.7 ns/day
>> >> > >
>> >> > > 3. 1 node, 8cpus, 1gpu
>> >> > > mdrun -s topol.tpr -ntomp 8  -gpu_id 0
>> >> > > Speed- 5.876 ns/day
>> >> > >
>> >> > > 4. 1 node, 8cpus, 1gpu
>> >> > > mdrun -s topol.tpr -ntomp 16  -gpu_id 0
>> >> > > Fatal Error: Environment variable OMP_NUM_THREADS (8) and the number
>> >> of
>> >> > > threads requested on  the command line (16) have different values.
>> >> Either
>> >> > > omit one, or set them both
>> >> > >  to the same value.
>> >> > >
>> >> > > Question for 3 and 4 : Do I need to always use OMP_NUM_THREADS or is
>> >> > there
>> >> > > a way ntomp overwrites the environment settings?
>> >> > >
>> >> > >
>> >> > > 5. 1 node, 8cpus , 2gpus
>> >> > > export OMP_NUM_THREADS = 8
>> >> > > mpirun -np 2 mdrun -s topol.tpr -pin on  -gpu_id 01
>> >> > > Speed - 4.044 ns/day
>> >> > >
>> >> > > 6. 2 nodes, 8cpus , 2 gpus
>> >> > > export OMP_NUM_THREADS = 8
>> >> > > mpirun -np 2 mdrun -s topol.tpr -pin on  -gpu_id 01
>> >> > > Speed - 3.0 ns/day
>> >> > >
>> >> > > Are the commands that I used for 5 and 6 correct?
>> >> > >
>> >> > > 7. I also used (1node, 8 cpus, 2 gpus)
>> >> > >  mdrun -s topol.tpr -ntmpi 2 -ntomp 8  -gpu_id 01
>> >> > > but this time I get a fatal error: thread mpi's are requested but
>> >> gromacs
>> >> > > is not compiled with thread MPI.
>> >> > >
>> >> > > Question: Isn't thread MPI enabled by default?
>> >> > >
>> >> > > 8. Finally, I recompiled Gromacs without OpenMP and re-ran case 1
>> but
>> >> > this
>> >> > > time there is a fatal error "More than 1 OpenMP thread requested,
>> but
>> >> > > Gromacs was compiled without OpenMP support."
>> >> > > command : mdrun -s topol.tpr  (no environment settings)  -gpu_id 0
>> >> > > Question: Here again, I assumed thread MPI is enabled by default
>> and I
>> >> > > think Gromacs still assumes OpenMp thread settings. Am i doing
>> >> something
>> >> > > wrong here?
>> >> > >
>> >> > > Thanks in advance for your help
>> >> > >
>> >> > > --
>> >> > > Siva
>> >> > > --
>> >> > > Gromacs Users mailing list
>> >> > >
>> >> > > * Please search the archive at
>> >> > > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
>> >> > > posting!
>> >> > >
>> >> > > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>> >> > >
>> >> > > * For (un)subscribe requests visit
>> >> > > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users
>> or
>> >> > > send a mail to gmx-users-request at gromacs.org.
>> >> > >
>> >> > --
>> >> > Gromacs Users mailing list
>> >> >
>> >> > * Please search the archive at
>> >> > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
>> >> > posting!
>> >> >
>> >> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>> >> >
>> >> > * For (un)subscribe requests visit
>> >> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>> >> > send a mail to gmx-users-request at gromacs.org.
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Siva
>> >> --
>> >> Gromacs Users mailing list
>> >>
>> >> * Please search the archive at
>> >> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
>> >> posting!
>> >>
>> >> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>> >>
>> >> * For (un)subscribe requests visit
>> >> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>> >> send a mail to gmx-users-request at gromacs.org.
>> >>
>> >
>> >
>> --
>> Gromacs Users mailing list
>>
>> * Please search the archive at
>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
>> posting!
>>
>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>
>> * For (un)subscribe requests visit
>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>> send a mail to gmx-users-request at gromacs.org.
>>
> --
> Gromacs Users mailing list
>
> * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a mail to gmx-users-request at gromacs.org.