[gmx-users] Multi-GPU optimization, "DD without halo exchange is not supported"

Leandro Bortot leandro.obt at gmail.com
Mon Apr 13 22:39:15 CEST 2020


Dear Szilárd,
     Thank you for your answer.

     I'm already exporting the three variables as separate commands (maybe
what you saw  was some formatting problem with the email? I'm not sure).
     For clarity, I just tested the following line:

          export GMX_GPU_DD_COMMS=true ; export GMX_GPU_PME_PP_COMMS=true ;
export GMX_FORCE_UPDATE_DEFAULT_GPU=true ; mpirun -np 16 gmx_mpi mdrun -s
md.tpr -v -deffnm md -ntomp 3 -nstlist 400 -nb gpu -bonded gpu -pme gpu
-pin on -npme 1

     And I still have the same problem:
"




*Update task on the GPU was required, by the GMX_FORCE_UPDATE_DEFAULT_GPU
environment variable, but the following condition(s) were not
satisfied:Domain decomposition without GPU halo exchange is not
supported. With separate PME rank(s), PME must use direct
communication.Will use CPU version of update.*
"

    Any ideas about what may be happening?


Best regards,
Leandro

-------
Leandro Oliveira Bortot
Postdoctoral Fellow
<http://linkedin.com> https://www.linkedin.com/in/leandro-obt/
Laboratory of Computational Biology
Brazilian Biosciences National Laboratory (LNBio)
Brazilian Center for Research in Energy and Materials (CNPEM)
Zip Code 13083-970, Campinas, São Paulo, Brazil.



On Mon, Apr 6, 2020 at 10:25 AM Szilárd Páll <pall.szilard at gmail.com> wrote:

> On Fri, Mar 27, 2020 at 8:30 PM Leandro Bortot <leandro.obt at gmail.com>
> wrote:
>
> > Dear users,
> >
> >      I'm trying to optimize the execution of a system composed by 10
> > million atoms on a multi-GPU machine with GROMACS 2020.1.
> >      I've followed the instructions given at
> >
> >
> https://devblogs.nvidia.com/creating-faster-molecular-dynamics-simulations-with-gromacs-2020/
> > . However, when I run my simulation, mdrun tells me this:
> >
> > "
> >
> > *Update task on the GPU was required, by the GMX_FORCE_UPDATE_DEFAULT_GPU
> > environment variable, but the following condition(s) were not
> > satisfied:Domain decomposition without GPU halo exchange is not
> supported.*
> > "
> >
> >      My understanding was that exporting *GMX_GPU_DD_COMMS=true* would
> > enable such halo communications.
> >      My simulation runs, however the performance is not scaling well with
> > the number of GPUs.
> >
> >      I've done the following:
> > "
> >
> > *export GMX_GPU_DD_COMMS=trueexport GMX_GPU_PME_PP_COMMS=trueexport
> > GMX_FORCE_UPDATE_DEFAULT_GPU=true*"
> >
>
> You are getting the error below because you did not export all three
> variables there. Those exports are separate commands and need to be issued
> with some separator, e.g. semicolon or newline.
>
>
> >      And my execution line is:
> > "*mpirun -np 4 gmx_mpi mdrun -s eq.1.tpr -v -deffnm eq.1 -pin on -ntomp 6
> > -npme 1 -nb gpu -bonded gpu -pme gpu -nstlist 400*"
> >
>
> Beware that this command line is specific for one use-case presented in
> your source (i.e. that very hardware and input system) and may not be fully
> transferable (e.g. "-ntomp 6" or "-nstlist 400").
>
> Cheers,
> --
> Szilárd
>
>
> >
> >      If I add "-update gpu" to this same line I have the following error:
> > "
> >
> >
> >
> >
> > *Inconsistency in user input:Update task on the GPU was required,but the
> > following condition(s) were not satisfied:Domain decomposition without
> GPU
> > halo exchange is not supported. With separate PME rank(s), PME must use
> > direct communication.*"
> >
> >      Also, I'm using constraints = h-bonds in my .mdp file.
> >
> >      Am I doing something wrong?
> >
> > Thank you for your attention,
> > Leandro
> > -------
> > Leandro Oliveira Bortot
> > Postdoctoral Fellow
> > <http://linkedin.com> https://www.linkedin.com/in/leandro-obt/
> > Laboratory of Computational Biology
> > Brazilian Biosciences National Laboratory (LNBio)
> > Brazilian Center for Research in Energy and Materials (CNPEM)
> > Zip Code 13083-970, Campinas, São Paulo, Brazil.
> > --
> > Gromacs Users mailing list
> >
> > * Please search the archive at
> > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> > posting!
> >
> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >
> > * For (un)subscribe requests visit
> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> > send a mail to gmx-users-request at gromacs.org.
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.


More information about the gromacs.org_gmx-users mailing list