[gmx-users] multiple nodes, GPUs on a Cray system

[Michael Weiner]

Hello.  I am trying to run Gromacs on ORNL’s Titan supercomputer (Cray XK7).  Specifically, I wish to run a large system across multiple nodes while employing GPUs with Gromacs 4.6.6.  However, I cannot figure out the appropriate way to call mdrun in order for this to work.  Within a submission script, I have tried many combinations of flags for aprun and mdrun, none of which work.  
Most specifically, if I request 50 nodes of 16 cores each and use this command:
aprun -n 800 -N 16 mdrun_mpi
I get the following error: “Incorrect launch configuration: mismatching number of PP MPI processes and GPUs per node.  mdrun_mpi was started with 16 PP MPI processes per node, but only 1 GPU were detected.”  
I have also tried with the -n flag equal to the number of nodes, but then it decomposes only to that many domains, rather than one per processor.  The “mdrun_mpi” command seems to be the only way of calling mdrun on the machine.  
I would appreciate any help with figuring out the appropriate flags to use for aprun and mdrun.  
Thank you.
Michael Weiner