[gmx-users] multiple nodes, GPUs on a Cray system

[Szilárd Páll]

Hi,

Michael, instead of using brute-forcing to tweak command line options, I
suggest consulting the docs on heterogeneous parallelization. (I hope) From
there it should be easy to learn the correct way to launch GPU accelerated
runs. Additionally, learning about the best practices will allow you not
only to launch jobs without a lot of hassle, but also to avoid wasting
performance by using suboptimal launch configurations (you can loose up to
50% performance!).

Except in a few corner cases, you should always use multiple MPI ranks,
especially on machines with >=8-10 CPU cores/GPU. The docs explain briefly
[1].

Our recent publication [2] gives further details and shows the effect of
the number of ranks per GPU (for the impatient, see Fig. 3 and Tables
10-11):.

On Cray XK7s, as the AMD CPUs have two NUMA regions and pretty slow caches
as well as only PCI-E 2.0, using "narrower" ranks that share a GPU has in
my experience quite serious performance benefit.  Hence, I suggest trying
 2-4 ranks/node (if DD and imbalance allows sometimes 7-8 is eve better).

On Tue, Jan 19, 2016 at 6:44 PM, Smith, Micholas D. <smithmd at ornl.gov>
wrote:

> Try this:
>
> aprun -n (number of nodes) -N 1 (1 mpi-process per node) mdrun_mpi -gpu_id
> 0 (now your job stuff here)
>

Considering the above, that becomes for 4 GPUs/node:
aprun -n NNODES -N 4 mdrun_mpi -gpu_id 0000


Let us know if that worked. Feel free to post results of your
experimentation (i.e. log files), we *may* be able to suggest improvements
to your settings.

Cheers,
--
Szilárd

[1].
http://www.gromacs.org/Documentation/Acceleration_and_parallelization#Multiple_MPI_ranks_per_GPU
[2]. http://doi.wiley.com/10.1002/jcc.24030


>
>
> ===================
> Micholas Dean Smith, PhD.
> Post-doctoral Research Associate
> University of Tennessee/Oak Ridge National Laboratory
> Center for Molecular Biophysics
>
> ________________________________________
> From: gromacs.org_gmx-users-bounces at maillist.sys.kth.se <
> gromacs.org_gmx-users-bounces at maillist.sys.kth.se> on behalf of Michael
> Weiner <mdw224 at cornell.edu>
> Sent: Tuesday, January 19, 2016 12:14 PM
> To: gromacs.org_gmx-users at maillist.sys.kth.se
> Subject: [gmx-users] multiple nodes, GPUs on a Cray system
>
> Hello.  I am trying to run Gromacs on ORNL’s Titan supercomputer (Cray
> XK7).  Specifically, I wish to run a large system across multiple nodes
> while employing GPUs with Gromacs 4.6.6.  However, I cannot figure out the
> appropriate way to call mdrun in order for this to work.  Within a
> submission script, I have tried many combinations of flags for aprun and
> mdrun, none of which work.
> Most specifically, if I request 50 nodes of 16 cores each and use this
> command:
> aprun -n 800 -N 16 mdrun_mpi
> I get the following error: “Incorrect launch configuration: mismatching
> number of PP MPI processes and GPUs per node.  mdrun_mpi was started with
> 16 PP MPI processes per node, but only 1 GPU were detected.”
> I have also tried with the -n flag equal to the number of nodes, but then
> it decomposes only to that many domains, rather than one per processor.
> The “mdrun_mpi” command seems to be the only way of calling mdrun on the
> machine.
> I would appreciate any help with figuring out the appropriate flags to use
> for aprun and mdrun.
> Thank you.
> Michael Weiner
>
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
>