[gmx-developers] Running Gromacs on GPUs on multiple machines

[Vedran Miletić]

Hi,

can one use both MPI and CUDA to run Gromacs on multiple machines with
mutliple GPUs? Wiki page on Acceleration and parallelization [1]
doesn't handle this particular scenario, and comments on bug 1135 [2]
suggest GPU acceleration is meant to be used on single machine only. I
might be missing something, because when I actually try to run it on a
cluster with 5 machines, 1 GPU per machine like, assuming -gpu_id
takes local GPU IDs

mpirun -np 5 -hostfile ... mdrun_mpi -v -deffnm ... -gpu_id 00000

it seems to run, and nvidia-smi indicates gmx_mpi running a process.
Very nice! However, the output of mdrun_mpi doesn't seem to be
correct:

1 GPU detected on host akston:
  #0: NVIDIA GeForce GTX 660, compute cap.: 3.0, ECC:  no, stat: compatible

1 GPU user-selected for this run.
Mapping of GPU ID to the 5 PP ranks in this node: 0,0,0,0,0
NOTE: You assigned GPUs to multiple MPI processes.

I certainly did not assign GPU to multiple MPI processes, because
there are 5 GPUs which have no info about them printed on screen. So,
is mdrun output wrong here and the code just works? Or is it that
mdrun should warn even harder that this scenario is unsupported?

Thanks in advance.

Regards,
Vedran

[1] http://www.gromacs.org/Documentation/Acceleration_and_parallelization
[2] http://bugzilla.gromacs.org/issues/1135