[gmx-users] How to assign PME ranks to particular nodes?

[Mark Abraham]

Hi,

On Thu., 12 Dec. 2019, 20:27 Marcin Mielniczuk, <marcin at golem.network>
wrote:

> Hi,
>
> I'm running Gromacs on a heterogenous cluster, with one node
> significantly faster than the other. Therefore, I'd like to achieve the
> following setup:
> * run 2 or 3 PP processes and 1 PME process on the faster node (with a
> lower number of OpenMP threads)
> * run 2 PP processes on the slower node (with a lower number of OpenMP
> threads)
>

The special case of n PP ranks and 1 PME rank is relatively easy. Just
arrange for the node to get the PME rank to be the last node to be
allocated ranks, and then both the relevant -ddorder settings produce the
same mapping of duty to rank. As always, if the nodes have multiple
sockets, then you want to avoid splitting any single rank over the socket
boundary; sometimes that can severely restrict the solution space.

Setting the number of threads is easy: one may just create a wrapper
> script on every node setting OMP_NUM_THREADS and GMX_PME_NUM_THREADS and
> use it instead of gmx_mpi. The rank assignment is more difficult.
>
> It's possible to use the OpenMPI --map-by option to control how ranks
> are mapped to nodes and use -ddorder pp_pme in Gromacs, so that the last
> rank is the PME rank, but this may possibly negatively affect the
> topology of the PP ranks.


I don't think there's a reason why that might be expected, do you know of
one?

An alternative is to use an OpenMPI rankfile,
> which requires me to specify precisely how all the ranks are mapped, not
> just the rank counts.
>
> Is there any better way to set the number of PP/PME ranks on particular
> nodes?
>

No, heterogeneous cases are up to you to solve in the kind of way you are
doing!

Mark

Thanks,
> Marcin
>
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
>