[gmx-users] Efficiently running multiple simulations

[Mark Abraham]

Hi,


On Wed, Sep 16, 2015 at 4:41 PM Zimmerman, Maxwell <mizimmer at wustl.edu>
wrote:

> Hi Mark,
>
> Sorry for the confusion, what I meant to say was that each node on the
> cluster has 8 GPUs and 16 CPUs.
>

OK. Please note that "CPU" is ambiguous, so you should prefer not to use it
without clarification.

Unless the GPUs are weak and the CPU is strong, 2 CPU cores per GPU will
likely be under-powered for PME simulations in GROMACS.

When I attempt to specify the GPU IDs for running 8 simulations on a node
> using the "-multi" and "-gpu_id", each .log file has the following:
>
> "8 GPUs user-selected for this run.
> Mapping of GPUs to the 8 PP ranks in this node: #0, #1, #2, #3, #4, #5,
> #6, #7"
>
> This makes me think that each simulation is competing for each of the GPUs


You are running 8 simulations, each of which has a single domain, each of
which is mapped to a single PP rank, each of which is mapped to a different
single GPU. Perfect.

explaining my performance loss per simulation compared to running 1
> simulation on 1 GPU and 2 CPUs.


Very likely you are not comparing with what you think you are, e.g. you
need to compare with an otherwise empty node running something like

mpirun -np 1 mdrun_mpi -ntomp 2 -gpu_id 0 -pin on

so that you actually have a single process running on two pinned CPU cores
and a single GPU. This should be fairly comparable with the mdrun -multi
setup

A side-by-side diff of that log file and the log file of the 0th member of
the multi-sim should show very few differences until the simulation starts,
and comparable performance. If not, please share your .log files on a
file-sharing service.

If this interpretation is correct, is there a better way to pin each
> simulation to a single GPU and 2 CPUs? If my interpretation is incorrect,
> is there a more efficient way to use the "-multi" option to match the
> performance I see of running a single simulation * 8?
>

mdrun will handle all of that correctly if it hasn't been crippled by how
the MPI library has organized life. You want it to assign ranks to cores
that are close to each other and their matching GPU. That tends to be the
default behaviour, but clusters intended for node sharing can do weird
things. (It is not yet clear that any of this is a problem.)

Mark


> Regards,
> -Maxwell
>
>
> ________________________________________
> From: gromacs.org_gmx-users-bounces at maillist.sys.kth.se <
> gromacs.org_gmx-users-bounces at maillist.sys.kth.se> on behalf of Mark
> Abraham <mark.j.abraham at gmail.com>
> Sent: Wednesday, September 16, 2015 3:52 AM
> To: gmx-users at gromacs.org; gromacs.org_gmx-users at maillist.sys.kth.se
> Subject: Re: [gmx-users] Efficiently running multiple simulations
>
> Hi,
>
> I'm confused by your description of the cluster as having 8 GPUs and 16
> CPUs. The relevant parameters are the number of GPUs and CPU cores per
> node. See the examples at
>
> http://manual.gromacs.org/documentation/5.1/user-guide/mdrun-features.html#running-multi-simulations
>
> Mark
>
> On Tue, Sep 15, 2015 at 11:38 PM Zimmerman, Maxwell <mizimmer at wustl.edu>
> wrote:
>
> > Hello,
> >
> >
> > I am having some troubles efficiently running simulations in parallel on
> a
> > gpu-cluster. The cluster has 8 GPUs and 16 CPUs. Currently, the command
> > that I am using is:
> >
> >
> > mpirun -np 8 mdrun_mpi -multi 8 -nice 4 -s md -o md -c after_md -v -x
> > frame -pin on
> >
> >
> > Per-simulation, the performance I am getting with this command is
> > significantly lower than running 1 simulation that uses 1 GPU and 2 CPUs
> > alone. This command seems to use all 8 GPUs and 16 CPUs on the 8 parallel
> > simulations, although I think this would be faster if I could pin each
> > simulation to a specific GPU and pair of CPUs. The -gpu_id option does
> not
> > seem to change anything when I am using the mpirun. Is there a way that I
> > can efficiently run the 8 simulations on the cluster by specifying the
> GPU
> > and CPUs to run with each simulation?
> >
> >
> > Thank you in advance!
> >
> >
> > Regards,
> >
> > -Maxwell
> > --
> > Gromacs Users mailing list
> >
> > * Please search the archive at
> > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> > posting!
> >
> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >
> > * For (un)subscribe requests visit
> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> > send a mail to gmx-users-request at gromacs.org.
> >
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
>