[gmx-users] How can I control PME domain decomposition

[Mark Abraham]

Hi,

On Thu, Mar 10, 2016 at 3:27 AM 柯谱 <kepu at csrc.ac.cn> wrote:

> Hi everyone,
>                           I am working on PP/PME optimization.  But I
> found while PP ranks could be controlled by, say -dd  9 8 8, PME node
> decomposition CANNOT be tuned.
>                           For instance,  below is the basic information
> for optimization:
>
>     - Using a fourier grid of 216x192x192, spacing 0.065 0.062 0.062
>     - Estimate for the relative computational load of the PME mesh part:
>     0.24
>
>
>                          I would like to run this simulation by -dd 9 8
> 8 and the PME nodes setting 24 * 8, in which PME grid_x (216) and grid_y
> (192) is divisible by #PME_ranks_x (24)
>        and PME grid_y (192) and grid_z (192) is divisible by
> #PME_ranks_y (8).  However, when I used -npme=192, gromacs automatically
> set
>
>     Domain decomposition grid *9 x 8 x 8*, separate PME ranks *192*
>     PME domain decomposition: *192 x 1 x 1*
>
> which significantly slowed down the simulation.  Then my question is
> what can I do to control PME node decomposition?  Thanks for your


Make sure you've read in the reference manual section 3.17.5
"Multiple-Program, Multiple-Data PME parallelization"

There, you will see "2-D PME decomposition has to have the number of
domains along x equal to the number of the PP decomposition." So by fixing
the choice of 9 PP domains in the x direction, you cannot have a 2-D PME
decomposition at all. Offhand, I would expect 8x8x9 to combine fine with
8x8x3, however.

Note that there are some anisotropies in the DD implementation that mean
that an inhomogeous simulation system often has a preferred orientation for
best performance, e.g. membrane proteins should have the membrane surface
normal to the z direction.

Mark

help!
>          Best wishes,
>                  Pu,
> **
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
>