[gmx-users] Multi-GPU optimization, "DD without halo exchange is not supported"

[Leandro Bortot]

Dear users,

     I'm trying to optimize the execution of a system composed by 10
million atoms on a multi-GPU machine with GROMACS 2020.1.
     I've followed the instructions given at
https://devblogs.nvidia.com/creating-faster-molecular-dynamics-simulations-with-gromacs-2020/
. However, when I run my simulation, mdrun tells me this:

"

*Update task on the GPU was required, by the GMX_FORCE_UPDATE_DEFAULT_GPU
environment variable, but the following condition(s) were not
satisfied:Domain decomposition without GPU halo exchange is not supported.*
"

     My understanding was that exporting *GMX_GPU_DD_COMMS=true* would
enable such halo communications.
     My simulation runs, however the performance is not scaling well with
the number of GPUs.

     I've done the following:
"

*export GMX_GPU_DD_COMMS=trueexport GMX_GPU_PME_PP_COMMS=trueexport
GMX_FORCE_UPDATE_DEFAULT_GPU=true*"

     And my execution line is:
"*mpirun -np 4 gmx_mpi mdrun -s eq.1.tpr -v -deffnm eq.1 -pin on -ntomp 6
-npme 1 -nb gpu -bonded gpu -pme gpu -nstlist 400*"

     If I add "-update gpu" to this same line I have the following error:
"




*Inconsistency in user input:Update task on the GPU was required,but the
following condition(s) were not satisfied:Domain decomposition without GPU
halo exchange is not supported. With separate PME rank(s), PME must use
direct communication.*"

     Also, I'm using constraints = h-bonds in my .mdp file.

     Am I doing something wrong?

Thank you for your attention,
Leandro
-------
Leandro Oliveira Bortot
Postdoctoral Fellow
<http://linkedin.com> https://www.linkedin.com/in/leandro-obt/
Laboratory of Computational Biology
Brazilian Biosciences National Laboratory (LNBio)
Brazilian Center for Research in Energy and Materials (CNPEM)
Zip Code 13083-970, Campinas, São Paulo, Brazil.