[gmx-users] domain decomposition error in the energy minimization step

[Kutzner, Carsten]

Dear Qasim,

those kinds of domain decomposition 'errors' can happen when you
try to distibute an MD system among too many MPI ranks. There is 
a minimum cell length for each domain decomposition cell in each
dimension, which depends on the chosen cutoff radii and possibly
other inter-atomic constraints. So this is normally just a technical 
limitation and not a problem with the MD system.

You can do the following steps to circumvent that issue:

a) use less ranks (at the domain decomposition limit, the parallel
   efficiency suffers anyway)

b) use separate PME ranks, so that you get less and larger domains
   on the MPI ranks that do domain decomposition
   (use mdrun -nt 20 -npme 4 ... for example, you will have to try
   around a bit with the exact number of PME ranks for optimal
   performance - or use the gmx tune_pme tool for that!)

c) In case you haven't done so already, compile GROMACS with
   MPI *and* OpenMP. Then, by using MPI domain decomposition plus
   OpenMP parallelism within each MPI rank, you will be
   able to use more cores in parallel even for smaller MD systems.

   Use mdrun -ntmpi 10 -ntomp 2 for 10 ranks * 2 threads or
       mdrun -ntmpi 4  -ntomp 5 for 4 ranks * 5 threads

   With real MPI, you would use something like

   mpirun -np 10 gmx mdrun -ntomp 2 ... 

   Don't forget to check your simulation performance, there will 
   be better and worse choices in terms of these decomposition parameters.

Happy simulating!
  Carsten


> On 11 Jan 2017, at 08:33, Qasim Pars <qasimpars at gmail.com> wrote:
> 
> Dear users,
> 
> I am trying to simulate a protein-ligand system including ~20000 atoms with
> waters using GROMACS-2016.1. The protocol I tried is forward state for the
> free energy calculation. The best ligand pose used in the simulations was
> got by AutoDock. At the beginning of the simulation GROMACS suffers from
> domain decomposition error in the energy minimization step:
> 
> Fatal error:
> There is no domain decomposition for 20 ranks that is compatible with the
> given box and a minimum cell size of 1.7353 nm
> Change the number of ranks or mdrun option -rdd
> Look in the log file for details on the domain decomposition
> 
> I checked the complex structure visually. I don't see any distortion in the
> structure. To check whether the problem is the number of nodes, I used 16
> nodes:
> 
> gmx mdrun -v -deffnm em -nt 16
> 
> The energy minimization step was completed successfully. For the NVT step I
> used 16 nodes again.
> 
> gmx mdrun -v -deffnm nvt -nt 16
> 
> After ~2000000 steps the system gave too many lincs warning.
> 
> Whereas there is no problem with wild type protein when it is simulated
> without using -nt 16. These domain decomposition error and lincs warning
> arise for complex structure.
> 
> By the way to keep the ligand in the active site of protein I use the bond,
> angle and dihedral restraints under [ intermolecular_interactions ] section
> in the top file.
> 
> [ intermolecular_interactions ]
> [ bonds ]
>  313    17 10     0.294     0.294    10.000      0.000     0.294
> 0.294    10.000   4184.000
> 
> [ angle_restraints ]
>  312   313    17   313          1   140.445      0.000          1
> 140.445     41.840          1
>  313    17    19    17          1   107.175      0.000          1
> 107.175     41.840          1
> 
> [ dihedral_restraints ]
>  300   312   313    17          1    56.245     0.000      0.000
> 56.245     0.000     41.840
>  312   313    17    19          1    -3.417     0.000      0.000
> -3.417     0.000     41.840
>  313    17    19    14          1  -110.822     0.000      0.000
> -110.822     0.000     41.840
> 
> The mdp file used for the energy minimization is follows:
> 
> define              = -DFLEXIBLE
> integrator          = steep
> nsteps              = 50000
> emtol               = 1000.0
> emstep              = 0.01
> nstcomm             = 100
> 
> ; OUTPUT CONTROL
> nstxout-compressed        = 500
> compressed-x-precision    = 1000
> nstlog                    = 500
> nstenergy                 = 500
> nstcalcenergy             = 100
> 
> ; BONDS
> constraints         = none
> 
> ; NEIGHBOUR SEARCHING
> 
> cutoff-scheme    = verlet
> ns-type          = grid
> nstlist          = 10
> pbc              = xyz
> 
> ; ELECTROSTATICS & EWALD
> coulombtype      = PME
> rcoulomb         = 1.0
> ewald_geometry   = 3d
> pme-order        = 4
> fourierspacing   = 0.12
> 
> ; VAN DER WAALS
> vdwtype         = Cut-off
> vdw_modifier    = Potential-switch
> rvdw            = 1.0
> rvdw-switch     = 0.9
> DispCorr        = EnerPres
> 
> ; FREE ENERGY
> free-energy              = yes
> init-lambda              = 0
> delta-lambda             = 0
> sc-alpha                 = 0.3
> sc-power                 = 1
> sc-sigma                 = 0.25
> sc-coul                  = yes
> couple-moltype           = ligand
> couple-intramol          = no
> couple-lambda0           = vdw-q
> couple-lambda1           = none
> nstdhdl                  = 100
> 
> I removed the free energy lines in the em.mdp and [
> intermolecular_interactions ] section in the top file but GROMACS still
> gives the domain decomposition error for the complex structure.
> 
> Will you please give suggestions on getting rid of the lincs warning and
> domain decomposition messages?
> 
> I would appreciate any kind of help.
> 
> Thanks.
> 
> -- 
> Qasim Pars
> -- 
> Gromacs Users mailing list
> 
> * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!
> 
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> 
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a mail to gmx-users-request at gromacs.org.