[gmx-users] Running GPU issue
Mark Abraham
mark.j.abraham at gmail.com
Thu Nov 15 21:10:15 CET 2018
Hi,
On Thu, Nov 15, 2018 at 1:54 PM Kovalskyy, Dmytro <Kovalskyy at uthscsa.edu>
wrote:
> The error you saw is clear evidence of a bug. There's a few things you
> might try to help narrow things down.
> * Is the error reproducible on each run of mdrun?
>
> Yes
>
>
>
> * 2018.3 was not designed or tested on CUDA 10, which was released rather
> later. If you have an earlier CUDA, please see if building with it
> alleviates the error
>
> No I have only CUDA 10.0 installed. Taking into account that I made that
> running (see replies to the next questions) please let me know whether
> should I go with CUDA 9.
>
>
>
> * The error message could be triggered by multiple parts of the code; what
> do you get with mdrun -nb gpu -pme cpu?
>
> This way I get MD running, no crash. All 72 cores (dual Xeon Gold 6140)
>
OK, that's a good clue that there may be a bug in the code for PME on GPU.
It's not yet clear whether it relates to CUDA 10, Quadro GPUs, or your
inputs, or something else. Can you please open a ticket at
https://redmine.gromacs.org and attach your .tpr and log files from the
cases that produce the error? Then we can try to reproduce and see what
aspect is the issue.
Thanks!
Mark
with my system I got
>
> $mdrun -deffnm md -v -nb gpu -pme cpu
>
> (ns/day) (hour/ns)
> Performance: 56.867 0.422
> $nvidia-smi
> | 0 Quadro P5000 On | 00000000:D5:00.0 On |
> Off |
> | 35% 56C P0 63W / 180W | 1400MiB / 16256MiB | 37%
> Default |
>
>
>
> * Do you get any more diagnostics from running with a build with cmake
> -DCMAKE_BUILD_TYPE=Debug?
>
> Interesting comes here. Gromacs with Debug = ON allows to stably run MD
> with no additional params.
> I mean
> $ gmx mdrun -deffnm md -v
> runs with no crash. However, only half of the Xeon cores are used
> (monitored with htop)
>
>
> (ns/day) (hour/ns)
> Performance: 69.577 0.345
> $nvidia-smi
> | 0 Quadro P5000 On | 00000000:D5:00.0 On |
> Off |
> | 38% 62C P0 68W / 180W | 1430MiB / 16256MiB | 56%
> Default |
>
>
>
> If I run
> $ gmx mdrun -deffnm md -v -nb gpu -pme cpu
> Then all CPU cores are busy but performancs is poor
> (ns/day) (hour/ns)
> Performance: 20.249 1.185
> $nvidia-smi
> | 0 Quadro P5000 On | 00000000:D5:00.0 On |
> Off |
> | 37% 57C P0 58W / 180W | 1408MiB / 16256MiB | 22%
> Default |
>
>
> Finally,
>
> If I run same MD with no GPU at all (i.e. -nb cpu) then I got performance
>
> (ns/day) (hour/ns)
> Performance: 47.536 0.505
>
>
> Does this help?
>
>
> Dmytro
>
>
>
> From: gromacs.org_gmx-users-bounces at maillist.sys.kth.se <
> gromacs.org_gmx-users-bounces at maillist.sys.kth.se> on behalf of Mark
> Abraham <mark.j.abraham at gmail.com>
> Sent: Wednesday, November 14, 2018 1:37 PM
> To: gmx-users at gromacs.org
> Cc: gromacs.org_gmx-users at maillist.sys.kth.se
> Subject: Re: [gmx-users] Running GPU issue
>
> Hi,
>
> I expect that that warning is fine (for now).
>
> The error you saw is clear evidence of a bug. There's a few things you
> might try to help narrow things down.
> * Is the error reproducible on each run of mdrun?
> * 2018.3 was not designed or tested on CUDA 10, which was released rather
> later. If you have an earlier CUDA, please see if building with it
> alleviates the error
> * The error message could be triggered by multiple parts of the code; what
> do you get with mdrun -nb gpu -pme cpu?
> * Do you get any more diagnostics from running with a build with cmake
> -DCMAKE_BUILD_TYPE=Debug?
>
> Mark
>
> On Wed, Nov 14, 2018 at 1:09 PM Kovalskyy, Dmytro <Kovalskyy at uthscsa.edu>
> wrote:
>
> > I forgot to add. While compiling gromacs I got following error at the
> very
> > beggining:
> >
> >
> > [ 3%] Built target gpu_utilstest_cuda
> > /usr/local/tmp/gromacs-2018.3/src/gromacs/gpu_utils/gpu_utils.cu: In
> > function ?int do_sanity_checks(int, cudaDeviceProp*)?:
> > /usr/local/tmp/gromacs-2018.3/src/gromacs/gpu_utils/gpu_utils.cu:258:28:
> > warning: ?cudaError_t cudaThreadSynchronize()? is deprecated
> > [-Wdeprecated-declarations]
> > if (cudaThreadSynchronize() != cudaSuccess)
> > ^
> > /usr/local/cuda/include/cuda_runtime_api.h:947:46: note: declared here
> > extern __CUDA_DEPRECATED __host__ cudaError_t CUDARTAPI
> > cudaThreadSynchronize(void);
> >
> >
> > But make has completed its job without falling down.
> >
> >
> >
> >
> > ________________________________________
> > From: gromacs.org_gmx-users-bounces at maillist.sys.kth.se <
> > gromacs.org_gmx-users-bounces at maillist.sys.kth.se> on behalf of Mark
> > Abraham <mark.j.abraham at gmail.com>
> > Sent: Tuesday, November 13, 2018 10:29 PM
> > To: gmx-users at gromacs.org
> > Cc: gromacs.org_gmx-users at maillist.sys.kth.se
> > Subject: Re: [gmx-users] Running GPU issue
> >
> > Hi,
> >
> > It can share.
> >
> > Mark
> >
> > On Mon, Nov 12, 2018 at 10:19 PM Kovalskyy, Dmytro <
> Kovalskyy at uthscsa.edu>
> > wrote:
> >
> > > Hi,
> > >
> > >
> > >
> > > To perform GPU with Gromacs does it require exclusive GPU card or
> > Gromacs
> > > can share the video card with X-server?
> > >
> > >
> > > Thank you
> > >
> > >
> > > Dmytro
> > > --
> > > Gromacs Users mailing list
> > >
> > > * Please search the archive at
> > > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> > > posting!
> > >
> > > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> > >
> > > * For (un)subscribe requests visit
> > > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> > > send a mail to gmx-users-request at gromacs.org.
> > >
> > --
> > Gromacs Users mailing list
> >
> > * Please search the archive at
> > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> > posting!
> >
> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >
> > * For (un)subscribe requests visit
> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> > send a mail to gmx-users-request at gromacs.org.
> > --
> > Gromacs Users mailing list
> >
> > * Please search the archive at
> > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> > posting!
> >
> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >
> > * For (un)subscribe requests visit
> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> > send a mail to gmx-users-request at gromacs.org.
> >
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
>
More information about the gromacs.org_gmx-users
mailing list