[gmx-users] Commands to run simulations using multiple GPU's in version 5.0.1

[Siva Dasetty]

Dear  All,

I am trying to run NPT simulations using GROMACS version 5.0.1 of a system
of size 140k atoms (protein+water systems) with 2 or more GPU's
(model=k20); 8 cores (or more); and 1 or more nodes. I am trying to
understand how to run simulations using multiple gpus on more than one
node. I  get the following errors/output when I run the simulation using
the following commands:-

Note: time-step used = 2 fs and total number of steps = 20000

First 4 cases are using single GPU and cases 5-8 are using 2 GPU's.

1. 1 node, 8 cpus, 1 gpu
export OMP_NUM_THREADS = 8
command used-  mdrun -s topol.tpr  -gpu_id 0
Speed - 5.8 ns/day

2.  1 node, 8 cpus, 1 gpu
export OMP_NUM_THREADS = 16
command used-  mdrun -s topol.tpr   -gpu_id 0
Speed - 4.7 ns/day

3. 1 node, 8cpus, 1gpu
mdrun -s topol.tpr -ntomp 8  -gpu_id 0
Speed- 5.876 ns/day

4. 1 node, 8cpus, 1gpu
mdrun -s topol.tpr -ntomp 16  -gpu_id 0
Fatal Error: Environment variable OMP_NUM_THREADS (8) and the number of
threads requested on  the command line (16) have different values. Either
omit one, or set them both
 to the same value.

Question for 3 and 4 : Do I need to always use OMP_NUM_THREADS or is there
a way ntomp overwrites the environment settings?


5. 1 node, 8cpus , 2gpus
export OMP_NUM_THREADS = 8
mpirun -np 2 mdrun -s topol.tpr -pin on  -gpu_id 01
Speed - 4.044 ns/day

6. 2 nodes, 8cpus , 2 gpus
export OMP_NUM_THREADS = 8
mpirun -np 2 mdrun -s topol.tpr -pin on  -gpu_id 01
Speed - 3.0 ns/day

Are the commands that I used for 5 and 6 correct?

7. I also used (1node, 8 cpus, 2 gpus)
 mdrun -s topol.tpr -ntmpi 2 -ntomp 8  -gpu_id 01
but this time I get a fatal error: thread mpi's are requested but gromacs
is not compiled with thread MPI.

Question: Isn't thread MPI enabled by default?

8. Finally, I recompiled Gromacs without OpenMP and re-ran case 1 but this
time there is a fatal error "More than 1 OpenMP thread requested, but
Gromacs was compiled without OpenMP support."
command : mdrun -s topol.tpr  (no environment settings)  -gpu_id 0
Question: Here again, I assumed thread MPI is enabled by default and I
think Gromacs still assumes OpenMp thread settings. Am i doing something
wrong here?

Thanks in advance for your help

-- 
Siva