[gmx-users] Too few cells to run on multiple cores

[Adrian Devitt-Lee]

Hi,

I'm having an issue using mdrun in parallel on 48 cores. I'm trying to
figure out which options I can include in the .mdp file to increase the
number of cells in my system. The full error message is:

Initializing Domain Decomposition on 48 ranks
Dynamic load balancing: auto
Will sort the charge groups at every domain (re)decomposition
Initial maximum inter charge-group distances:
    two-body bonded interactions: 1.368 nm, LJC Pairs NB, atoms 2123 2152
  multi-body bonded interactions: 0.428 nm, Proper Dih., atoms 1105 1113
*Minimum cell size due to bonded interactions: 1.505 nm*
Maximum distance for 7 constraints, at 120 deg. angles, all-trans: 0.219 nm
Estimated maximum distance required for P-LINCS: 0.219 nm
Guess for relative PME load: 0.65
Using 0 separate PME ranks, as guessed by mdrun
Scaling the initial minimum size with 1/0.8 (option -dds) = 1.25

*Optimizing the DD grid for 48 cells with a minimum initial size of 1.881
nmThe maximum allowed number of cells is: X 3 Y 3 Z 3*

-------------------------------------------------------
Program mdrun_mpi, VERSION 5.1
Source code file:
/work/y07/y07/gmx/5.1-phase2/source/src/gromacs/domdec/domdec.cpp, line:
6969

Fatal error:
There is no domain decomposition for 48 ranks that is compatible with the
given box and a minimum cell size of 1.88133 nm


I understand the problem -- the system wants to assign one core to each
grid cell, but there are only 3x3x3 = 27 cells. I don't know what I can do
to fix this problem. The system is a solvated protein bound to a ligand in
a ~15 nm box, and it has > 40,000 atoms. I have tried changing the
lincs-order and fourier-spacing to no avail.

I was able to get the system to run by adding the following flags to mdrun:
  -rdd 1.2 -dds 0.9

But when I did this, the force imbalance went to > 140%, and 10-30% of the
cpu time was lost due to load imbalance.

Can someone suggest how I could edit my .mdp file to increase the number of
allowed cells?