[gmx-users] GROMACS 2018.2 mdrun GPU Assertion failed: Condition: cudaSuccess == cudaPeekAtLastError()

Jia Hong jiahong96 at hotmail.com
Wed Aug 15 04:34:09 CEST 2018


Hi Szilárd,

Thanks to your tweaked version of the code, I eventually found out that the CUDA capable GPU's id wasn't what I thought it was.

I thought since nvidia-smi displayed 1080 as the second card (GPU number 1), it has id 1, but it actually has id 0. Therefore, setting CUDA_VISIBLE_DEVICES=0 gmx mdrun <ops> did the trick.

Thank you to everyone involved.

Cheers,
Harry
________________________________
From: gromacs.org_gmx-users-bounces at maillist.sys.kth.se <gromacs.org_gmx-users-bounces at maillist.sys.kth.se> on behalf of Szilárd Páll <pall.szilard at gmail.com>
Sent: Monday, August 13, 2018 10:52 PM
To: Discussion list for GROMACS users
Subject: Re: [gmx-users] GROMACS 2018.2 mdrun GPU Assertion failed: Condition: cudaSuccess == cudaPeekAtLastError()

Hi,

I can not reproduce such an error, not even by emulating your use case.

Can you please try to build this slightly tweaked version of the code:
https://gerrit.gromacs.org/changes/8172/revisions/fb358c835c312952571a78f32971cfacf42df966/archive?format=tgz
and let me know what the error is?

Also, please provide the full log and stdandard outputs
with/without CUDA_VISIBLE_DEVICES.

Cheers,
--
Szilárd


On Sat, Aug 11, 2018 at 4:24 AM Jia Hong <jiahong96 at hotmail.com> wrote:

> Hi Mark,
>
> I've set CUDA_VISIBLE_DEVICES to 1, the exact command was
>
> export CUDA_VISIBLE_DEVICES=1
>
> I chose 1 because that's the GPU id for the 1080 card. Then, in the same
> shell I ran the gmx command (both with and without -gpu_id 1 argument).
>
> As for the log file, I thought it was md_0_01_GPU.log, which I sent as a
> follow-up message. Nonetheless,
> here's the link to the file
> https://www.dropbox.com/s/xc8z09wvnl0314m/md_0_01_GPU.log?dl=0.
> [
> https://www.dropbox.com/static/images/spectrum-icons/generated/content/content-unknown-large.png
> ]<https://www.dropbox.com/s/xc8z09wvnl0314m/md_0_01_GPU.log?dl=0>
>
> md_0_01_GPU.log<
> https://www.dropbox.com/s/xc8z09wvnl0314m/md_0_01_GPU.log?dl=0>
> Shared with Dropbox
> www.dropbox.com<http://www.dropbox.com>
>
>
> Cheers,
> Harry
>
> ________________________________
> From: gromacs.org_gmx-users-bounces at maillist.sys.kth.se <
> gromacs.org_gmx-users-bounces at maillist.sys.kth.se> on behalf of Mark
> Abraham <mark.j.abraham at gmail.com>
> Sent: Saturday, August 11, 2018 7:43 AM
> To: gmx-users at gromacs.org
> Subject: Re: [gmx-users] GROMACS 2018.2 mdrun GPU Assertion failed:
> Condition: cudaSuccess == cudaPeekAtLastError()
>
> Hi,
>
> To what did you set CUDA_VISIBLE_DEVICES? Just saying it didn't work
> doesn't exclude that it couldn't have worked because it wasn't right :-)
>
> Also, could you please share a full log file via a sharing service and post
> a link. This fragment doesn't help us find out how the detection saw your
> system, unfortunately.
>
> Mark
>
> On Wed, Aug 8, 2018, 10:53 Jia Hong <jiahong96 at hotmail.com> wrote:
>
> > The log file:
> >
> >
> > Log file opened on Wed Aug  8 16:33:34 2018
> > Host: matthias-processing  pid: 28756  rank ID: 0  number of ranks:  1
> >                       :-) GROMACS - gmx mdrun, 2018.2 (-:
> >
> >                             GROMACS is written by:
> >      Emile Apol      Rossen Apostolov      Paul Bauer     Herman J.C.
> > Berendsen
> >     Par Bjelkmar    Aldert van Buuren   Rudi van Drunen     Anton
> Feenstra
> >   Gerrit Groenhof    Aleksei Iupinov   Christoph Junghans   Anca Hamuraru
> >  Vincent Hindriksen Dimitrios Karkoulis    Peter Kasson        Jiri Kraus
> >   Carsten Kutzner      Per Larsson      Justin A. Lemkul    Viveca
> Lindahl
> >   Magnus Lundborg   Pieter Meulenhoff    Erik Marklund      Teemu Murtola
> >     Szilard Pall       Sander Pronk      Roland Schulz     Alexey
> Shvetsov
> >    Michael Shirts     Alfons Sijbers     Peter Tieleman    Teemu
> Virolainen
> >  Christian Wennberg    Maarten Wolf
> >                            and the project leaders:
> >         Mark Abraham, Berk Hess, Erik Lindahl, and David van der Spoel
> >
> > Copyright (c) 1991-2000, University of Groningen, The Netherlands.
> > Copyright (c) 2001-2017, The GROMACS development team at
> > Uppsala University, Stockholm University and
> > the Royal Institute of Technology, Sweden.
> > check out http://www.gromacs.org for more information.
> >
> > GROMACS is free software; you can redistribute it and/or modify it
> > under the terms of the GNU Lesser General Public License
> > as published by the Free Software Foundation; either version 2.1
> > of the License, or (at your option) any later version.
> >
> > GROMACS:      gmx mdrun, version 2018.2
> > Executable:   /home/matthias/Documents/gromacs-2018.2/gpu_install/bin/gmx
> > Data prefix:  /home/matthias/Documents/gromacs-2018.2/gpu_install
> > Working dir:  /home/matthias/Documents/MD_simulations/D_EY_ext
> > Command line:
> >   gmx mdrun -deffnm md_0_01_GPU
> >
> > GROMACS version:    2018.2
> > Precision:          single
> > Memory model:       64 bit
> > MPI library:        thread_mpi
> > OpenMP support:     enabled (GMX_OPENMP_MAX_THREADS = 64)
> > GPU support:        CUDA
> > SIMD instructions:  AVX_256
> > FFT library:        fftw-3.3.8-sse2-avx-avx2-avx2_128-avx512
> > RDTSCP usage:       enabled
> > TNG support:        enabled
> > Hwloc support:      disabled
> > Tracing support:    disabled
> > Built on:           2018-08-03 08:06:55
> > Built by:           matthias at matthias-processing [CMAKE]
> > Build OS/arch:      Linux 4.15.0-29-generic x86_64
> > Build CPU vendor:   Intel
> > Build CPU brand:    Intel(R) Xeon(R) CPU E5-1650 0 @ 3.20GHz
> > Build CPU family:   6   Model: 45   Stepping: 7
> > Build CPU features: aes apic avx clfsh cmov cx8 cx16 htt intel lahf mmx
> > msr nonstop_tsc pcid pclmuldq pdcm pdpe1gb popcnt pse rdtscp sse2 sse3
> > sse4.1 sse4.2 ssse3 tdt x2apic
> > C compiler:         /usr/bin/cc GNU 5.4.0
> > C compiler flags:    -mavx     -O3 -DNDEBUG -funroll-all-loops
> > -fexcess-precision=fast
> > C++ compiler:       /usr/bin/c++ GNU 5.4.0
> > C++ compiler flags:  -mavx    -std=c++11   -O3 -DNDEBUG
> -funroll-all-loops
> > -fexcess-precision=fast
> > CUDA compiler:      /usr/local/cuda-9.0/bin/nvcc nvcc: NVIDIA (R) Cuda
> > compiler driver;Copyright (c) 2005-2017 NVIDIA Corporation;Built on
> > Fri_Sep__1_21:08:03_CDT_2017;Cuda compilation tools, release 9.0,
> V9.0.176
> > CUDA compiler
> >
> flags:-gencode;arch=compute_30,code=sm_30;-gencode;arch=compute_35,code=sm_35;-gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_52,code=sm_52;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_70,code=compute_70;-use_fast_math;-D_FORCE_INLINES;;
> >
> ;-mavx;-std=c++11;-O3;-DNDEBUG;-funroll-all-loops;-fexcess-precision=fast;
> > CUDA driver:        9.0
> > CUDA runtime:       9.0
> >
> >
> > ________________________________
> > From: gromacs.org_gmx-users-bounces at maillist.sys.kth.se <
> > gromacs.org_gmx-users-bounces at maillist.sys.kth.se> on behalf of Jia
> Hong <
> > jiahong96 at hotmail.com>
> > Sent: Wednesday, August 8, 2018 3:50 PM
> > To: gromacs.org_gmx-users at maillist.sys.kth.se
> > Subject: [gmx-users] GROMACS 2018.2 mdrun GPU Assertion failed:
> Condition:
> > cudaSuccess == cudaPeekAtLastError()
> >
> > Hi,
> >
> > I've recently compiled gromacs 2018.2 on ubuntu 16.04.5 LTS with CUDA 9.0
> > on an Nvidia 1080. I've got the error below on the mdrun command. I've
> > searched around (
> >
> https://mailman-1.sys.kth.se/pipermail/gromacs.org_gmx-users/2018-February/118631.html
> )
> > and I tried the suggested export CUDA_VISIBLE_DEVICES without any luck.
> >
> > I currently have 2 nvidia GPUs sitting inside the machine, the first
> > doesn't have CUDA support (old graphics card) while the second is the
> > aforementioned 1080. I've also verified that the 1080 has gpu id 1, and
> ran
> > the command:
> >
> >   gmx mdrun -deffnm md_0_01_GPU -gpu_id 1
> >
> > which gives the same error.
> > I've attached the log file too.
> >
> > (I don't know if this helps, but I manage to compile a CUDA test program
> > to verify that my CUDA installation isn't broken:
> >
> http://ramblingsofagamedevstudent.blogspot.com/2014/03/set-up-visual-studio-2012-for-cuda.html
> > )
> >
> > Cheers,
> > Harry
> >
> > The output:
> >
> >
> > :-) GROMACS - gmx mdrun, 2018.2 (-:
> >
> >                             GROMACS is written by:
> >      Emile Apol      Rossen Apostolov      Paul Bauer     Herman J.C.
> > Berendsen
> >     Par Bjelkmar    Aldert van Buuren   Rudi van Drunen     Anton
> Feenstra
> >   Gerrit Groenhof    Aleksei Iupinov   Christoph Junghans   Anca Hamuraru
> >  Vincent Hindriksen Dimitrios Karkoulis    Peter Kasson        Jiri Kraus
> >   Carsten Kutzner      Per Larsson      Justin A. Lemkul    Viveca
> Lindahl
> >   Magnus Lundborg   Pieter Meulenhoff    Erik Marklund      Teemu Murtola
> >     Szilard Pall       Sander Pronk      Roland Schulz     Alexey
> Shvetsov
> >    Michael Shirts     Alfons Sijbers     Peter Tieleman    Teemu
> Virolainen
> >  Christian Wennberg    Maarten Wolf
> >                            and the project leaders:
> >         Mark Abraham, Berk Hess, Erik Lindahl, and David van der Spoel
> >
> > Copyright (c) 1991-2000, University of Groningen, The Netherlands.
> > Copyright (c) 2001-2017, The GROMACS development team at
> > Uppsala University, Stockholm University and
> > the Royal Institute of Technology, Sweden.
> > check out http://www.gromacs.org for more information.
> >
> > GROMACS is free software; you can redistribute it and/or modify it
> > under the terms of the GNU Lesser General Public License
> > as published by the Free Software Foundation; either version 2.1
> > of the License, or (at your option) any later version.
> >
> > GROMACS:      gmx mdrun, version 2018.2
> > Executable:   /home/matthias/Documents/gromacs-2018.2/gpu_install/bin/gmx
> > Data prefix:  /home/matthias/Documents/gromacs-2018.2/gpu_install
> > Working dir:  /home/matthias/Documents/MD_simulations/D_EY_ext
> > Command line:
> >   gmx mdrun -deffnm md_0_01_GPU
> >
> >
> > Back Off! I just backed up md_0_01_GPU.log to ./#md_0_01_GPU.log.10#
> >
> > -------------------------------------------------------
> > Program:     gmx mdrun, version 2018.2
> > Source file: src/gromacs/gpu_utils/gpu_utils.cu (line 762)
> > Function:    void findGpus(gmx_gpu_info_t*)
> >
> > Assertion failed:
> > Condition: cudaSuccess == cudaPeekAtLastError()
> > We promise to return with clean CUDA state!
> >
> > For more information and tips for troubleshooting, please check the
> GROMACS
> > website at http://www.gromacs.org/Documentation/Errors
> >
> > --
> > Gromacs Users mailing list
> >
> > * Please search the archive at
> > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> > posting!
> >
> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >
> > * For (un)subscribe requests visit
> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> > send a mail to gmx-users-request at gromacs.org.
> >
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
>
--
Gromacs Users mailing list

* Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!

* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists

* For (un)subscribe requests visit
https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a mail to gmx-users-request at gromacs.org.


More information about the gromacs.org_gmx-users mailing list