[gmx-users] Domain decomposition for parallel simulations

[Kevin C Chan]

Dear Users,

I have encountered the problem of "There is no domain decomposition for n
nodes that is compatible with the given box and a minimum cell size of x
nm" and by reading through the gromacs website and some threads I
understand that the problem might be caused by breaking the system into too
small boxes by too many ranks. However, I have no idea how to get the
correct estimation of suitable paralleling parameters. Hope someone could
share his experience.

Here are information stated in the log file:
*Initializing Domain Decomposition on 4000 ranks*
*Dynamic load balancing: on*
*Will sort the charge groups at every domain (re)decomposition*
*Initial maximum inter charge-group distances:*
*    two-body bonded interactions: 0.665 nm, Dis. Rest., atoms 23558 23590*
*  multi-body bonded interactions: 0.425 nm, Proper Dih., atoms 12991 12999*
*Minimum cell size due to bonded interactions: 0.468 nm*
*Maximum distance for 5 constraints, at 120 deg. angles, all-trans: 0.819
nm*
*Estimated maximum distance required for P-LINCS: 0.819 nm*
*This distance will limit the DD cell size, you can override this with
-rcon*
*Guess for relative PME load: 0.11*
*Will use 3500 particle-particle and 500 PME only ranks*
*This is a guess, check the performance at the end of the log file*
*Using 500 separate PME ranks, as guessed by mdrun*
*Scaling the initial minimum size with 1/0.8 (option -dds) = 1.25*
*Optimizing the DD grid for 3500 cells with a minimum initial size of 1.024
nm*
*The maximum allowed number of cells is: X 17 Y 17 Z 17*

And I got this afterwards:
*Fatal error:*
*There is no domain decomposition for 3500 ranks that is compatible with
the given box and a minimum cell size of 1.02425 nm*

Here are some questions:
1. the maximum allowed number of cells is 17x17x17 which is 4913 and seems
to be larger than the requested 3500 particle-particle ranks, so the
minimum cell size is not causing the problem?
2. Where does this 1.024 nm comes from? We can see the inter charge-group
distances are listed as 0.665 and 0.425 nm
3. The distance restraint between atoms 23558 23590 was set explicitly (or
added manually) in the topology file and should be around 0.32 nm by using
[intermolecular_interactions]. How could I know my manual setting is
working or not? As it has shown a different value.


Thanks in advance,
Kevin
OSU