[gmx-users] Multi-GPU optimization, "DD without halo exchange is not supported"

[Szilárd Páll]

On Fri, Mar 27, 2020 at 8:30 PM Leandro Bortot <leandro.obt at gmail.com>
wrote:

> Dear users,
>
>      I'm trying to optimize the execution of a system composed by 10
> million atoms on a multi-GPU machine with GROMACS 2020.1.
>      I've followed the instructions given at
>
> https://devblogs.nvidia.com/creating-faster-molecular-dynamics-simulations-with-gromacs-2020/
> . However, when I run my simulation, mdrun tells me this:
>
> "
>
> *Update task on the GPU was required, by the GMX_FORCE_UPDATE_DEFAULT_GPU
> environment variable, but the following condition(s) were not
> satisfied:Domain decomposition without GPU halo exchange is not supported.*
> "
>
>      My understanding was that exporting *GMX_GPU_DD_COMMS=true* would
> enable such halo communications.
>      My simulation runs, however the performance is not scaling well with
> the number of GPUs.
>
>      I've done the following:
> "
>
> *export GMX_GPU_DD_COMMS=trueexport GMX_GPU_PME_PP_COMMS=trueexport
> GMX_FORCE_UPDATE_DEFAULT_GPU=true*"
>

You are getting the error below because you did not export all three
variables there. Those exports are separate commands and need to be issued
with some separator, e.g. semicolon or newline.


>      And my execution line is:
> "*mpirun -np 4 gmx_mpi mdrun -s eq.1.tpr -v -deffnm eq.1 -pin on -ntomp 6
> -npme 1 -nb gpu -bonded gpu -pme gpu -nstlist 400*"
>

Beware that this command line is specific for one use-case presented in
your source (i.e. that very hardware and input system) and may not be fully
transferable (e.g. "-ntomp 6" or "-nstlist 400").

Cheers,
--
Szilárd


>
>      If I add "-update gpu" to this same line I have the following error:
> "
>
>
>
>
> *Inconsistency in user input:Update task on the GPU was required,but the
> following condition(s) were not satisfied:Domain decomposition without GPU
> halo exchange is not supported. With separate PME rank(s), PME must use
> direct communication.*"
>
>      Also, I'm using constraints = h-bonds in my .mdp file.
>
>      Am I doing something wrong?
>
> Thank you for your attention,
> Leandro
> -------
> Leandro Oliveira Bortot
> Postdoctoral Fellow
> <http://linkedin.com> https://www.linkedin.com/in/leandro-obt/
> Laboratory of Computational Biology
> Brazilian Biosciences National Laboratory (LNBio)
> Brazilian Center for Research in Energy and Materials (CNPEM)
> Zip Code 13083-970, Campinas, São Paulo, Brazil.
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.