[gmx-users] GROMACS 2018.2 mdrun GPU Assertion failed: Condition: cudaSuccess == cudaPeekAtLastError()
Szilárd Páll
pall.szilard at gmail.com
Mon Aug 13 16:52:16 CEST 2018
Hi,
I can not reproduce such an error, not even by emulating your use case.
Can you please try to build this slightly tweaked version of the code:
https://gerrit.gromacs.org/changes/8172/revisions/fb358c835c312952571a78f32971cfacf42df966/archive?format=tgz
and let me know what the error is?
Also, please provide the full log and stdandard outputs
with/without CUDA_VISIBLE_DEVICES.
Cheers,
--
Szilárd
On Sat, Aug 11, 2018 at 4:24 AM Jia Hong <jiahong96 at hotmail.com> wrote:
> Hi Mark,
>
> I've set CUDA_VISIBLE_DEVICES to 1, the exact command was
>
> export CUDA_VISIBLE_DEVICES=1
>
> I chose 1 because that's the GPU id for the 1080 card. Then, in the same
> shell I ran the gmx command (both with and without -gpu_id 1 argument).
>
> As for the log file, I thought it was md_0_01_GPU.log, which I sent as a
> follow-up message. Nonetheless,
> here's the link to the file
> https://www.dropbox.com/s/xc8z09wvnl0314m/md_0_01_GPU.log?dl=0.
> [
> https://www.dropbox.com/static/images/spectrum-icons/generated/content/content-unknown-large.png
> ]<https://www.dropbox.com/s/xc8z09wvnl0314m/md_0_01_GPU.log?dl=0>
>
> md_0_01_GPU.log<
> https://www.dropbox.com/s/xc8z09wvnl0314m/md_0_01_GPU.log?dl=0>
> Shared with Dropbox
> www.dropbox.com
>
>
> Cheers,
> Harry
>
> ________________________________
> From: gromacs.org_gmx-users-bounces at maillist.sys.kth.se <
> gromacs.org_gmx-users-bounces at maillist.sys.kth.se> on behalf of Mark
> Abraham <mark.j.abraham at gmail.com>
> Sent: Saturday, August 11, 2018 7:43 AM
> To: gmx-users at gromacs.org
> Subject: Re: [gmx-users] GROMACS 2018.2 mdrun GPU Assertion failed:
> Condition: cudaSuccess == cudaPeekAtLastError()
>
> Hi,
>
> To what did you set CUDA_VISIBLE_DEVICES? Just saying it didn't work
> doesn't exclude that it couldn't have worked because it wasn't right :-)
>
> Also, could you please share a full log file via a sharing service and post
> a link. This fragment doesn't help us find out how the detection saw your
> system, unfortunately.
>
> Mark
>
> On Wed, Aug 8, 2018, 10:53 Jia Hong <jiahong96 at hotmail.com> wrote:
>
> > The log file:
> >
> >
> > Log file opened on Wed Aug 8 16:33:34 2018
> > Host: matthias-processing pid: 28756 rank ID: 0 number of ranks: 1
> > :-) GROMACS - gmx mdrun, 2018.2 (-:
> >
> > GROMACS is written by:
> > Emile Apol Rossen Apostolov Paul Bauer Herman J.C.
> > Berendsen
> > Par Bjelkmar Aldert van Buuren Rudi van Drunen Anton
> Feenstra
> > Gerrit Groenhof Aleksei Iupinov Christoph Junghans Anca Hamuraru
> > Vincent Hindriksen Dimitrios Karkoulis Peter Kasson Jiri Kraus
> > Carsten Kutzner Per Larsson Justin A. Lemkul Viveca
> Lindahl
> > Magnus Lundborg Pieter Meulenhoff Erik Marklund Teemu Murtola
> > Szilard Pall Sander Pronk Roland Schulz Alexey
> Shvetsov
> > Michael Shirts Alfons Sijbers Peter Tieleman Teemu
> Virolainen
> > Christian Wennberg Maarten Wolf
> > and the project leaders:
> > Mark Abraham, Berk Hess, Erik Lindahl, and David van der Spoel
> >
> > Copyright (c) 1991-2000, University of Groningen, The Netherlands.
> > Copyright (c) 2001-2017, The GROMACS development team at
> > Uppsala University, Stockholm University and
> > the Royal Institute of Technology, Sweden.
> > check out http://www.gromacs.org for more information.
> >
> > GROMACS is free software; you can redistribute it and/or modify it
> > under the terms of the GNU Lesser General Public License
> > as published by the Free Software Foundation; either version 2.1
> > of the License, or (at your option) any later version.
> >
> > GROMACS: gmx mdrun, version 2018.2
> > Executable: /home/matthias/Documents/gromacs-2018.2/gpu_install/bin/gmx
> > Data prefix: /home/matthias/Documents/gromacs-2018.2/gpu_install
> > Working dir: /home/matthias/Documents/MD_simulations/D_EY_ext
> > Command line:
> > gmx mdrun -deffnm md_0_01_GPU
> >
> > GROMACS version: 2018.2
> > Precision: single
> > Memory model: 64 bit
> > MPI library: thread_mpi
> > OpenMP support: enabled (GMX_OPENMP_MAX_THREADS = 64)
> > GPU support: CUDA
> > SIMD instructions: AVX_256
> > FFT library: fftw-3.3.8-sse2-avx-avx2-avx2_128-avx512
> > RDTSCP usage: enabled
> > TNG support: enabled
> > Hwloc support: disabled
> > Tracing support: disabled
> > Built on: 2018-08-03 08:06:55
> > Built by: matthias at matthias-processing [CMAKE]
> > Build OS/arch: Linux 4.15.0-29-generic x86_64
> > Build CPU vendor: Intel
> > Build CPU brand: Intel(R) Xeon(R) CPU E5-1650 0 @ 3.20GHz
> > Build CPU family: 6 Model: 45 Stepping: 7
> > Build CPU features: aes apic avx clfsh cmov cx8 cx16 htt intel lahf mmx
> > msr nonstop_tsc pcid pclmuldq pdcm pdpe1gb popcnt pse rdtscp sse2 sse3
> > sse4.1 sse4.2 ssse3 tdt x2apic
> > C compiler: /usr/bin/cc GNU 5.4.0
> > C compiler flags: -mavx -O3 -DNDEBUG -funroll-all-loops
> > -fexcess-precision=fast
> > C++ compiler: /usr/bin/c++ GNU 5.4.0
> > C++ compiler flags: -mavx -std=c++11 -O3 -DNDEBUG
> -funroll-all-loops
> > -fexcess-precision=fast
> > CUDA compiler: /usr/local/cuda-9.0/bin/nvcc nvcc: NVIDIA (R) Cuda
> > compiler driver;Copyright (c) 2005-2017 NVIDIA Corporation;Built on
> > Fri_Sep__1_21:08:03_CDT_2017;Cuda compilation tools, release 9.0,
> V9.0.176
> > CUDA compiler
> >
> flags:-gencode;arch=compute_30,code=sm_30;-gencode;arch=compute_35,code=sm_35;-gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_52,code=sm_52;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_70,code=compute_70;-use_fast_math;-D_FORCE_INLINES;;
> >
> ;-mavx;-std=c++11;-O3;-DNDEBUG;-funroll-all-loops;-fexcess-precision=fast;
> > CUDA driver: 9.0
> > CUDA runtime: 9.0
> >
> >
> > ________________________________
> > From: gromacs.org_gmx-users-bounces at maillist.sys.kth.se <
> > gromacs.org_gmx-users-bounces at maillist.sys.kth.se> on behalf of Jia
> Hong <
> > jiahong96 at hotmail.com>
> > Sent: Wednesday, August 8, 2018 3:50 PM
> > To: gromacs.org_gmx-users at maillist.sys.kth.se
> > Subject: [gmx-users] GROMACS 2018.2 mdrun GPU Assertion failed:
> Condition:
> > cudaSuccess == cudaPeekAtLastError()
> >
> > Hi,
> >
> > I've recently compiled gromacs 2018.2 on ubuntu 16.04.5 LTS with CUDA 9.0
> > on an Nvidia 1080. I've got the error below on the mdrun command. I've
> > searched around (
> >
> https://mailman-1.sys.kth.se/pipermail/gromacs.org_gmx-users/2018-February/118631.html
> )
> > and I tried the suggested export CUDA_VISIBLE_DEVICES without any luck.
> >
> > I currently have 2 nvidia GPUs sitting inside the machine, the first
> > doesn't have CUDA support (old graphics card) while the second is the
> > aforementioned 1080. I've also verified that the 1080 has gpu id 1, and
> ran
> > the command:
> >
> > gmx mdrun -deffnm md_0_01_GPU -gpu_id 1
> >
> > which gives the same error.
> > I've attached the log file too.
> >
> > (I don't know if this helps, but I manage to compile a CUDA test program
> > to verify that my CUDA installation isn't broken:
> >
> http://ramblingsofagamedevstudent.blogspot.com/2014/03/set-up-visual-studio-2012-for-cuda.html
> > )
> >
> > Cheers,
> > Harry
> >
> > The output:
> >
> >
> > :-) GROMACS - gmx mdrun, 2018.2 (-:
> >
> > GROMACS is written by:
> > Emile Apol Rossen Apostolov Paul Bauer Herman J.C.
> > Berendsen
> > Par Bjelkmar Aldert van Buuren Rudi van Drunen Anton
> Feenstra
> > Gerrit Groenhof Aleksei Iupinov Christoph Junghans Anca Hamuraru
> > Vincent Hindriksen Dimitrios Karkoulis Peter Kasson Jiri Kraus
> > Carsten Kutzner Per Larsson Justin A. Lemkul Viveca
> Lindahl
> > Magnus Lundborg Pieter Meulenhoff Erik Marklund Teemu Murtola
> > Szilard Pall Sander Pronk Roland Schulz Alexey
> Shvetsov
> > Michael Shirts Alfons Sijbers Peter Tieleman Teemu
> Virolainen
> > Christian Wennberg Maarten Wolf
> > and the project leaders:
> > Mark Abraham, Berk Hess, Erik Lindahl, and David van der Spoel
> >
> > Copyright (c) 1991-2000, University of Groningen, The Netherlands.
> > Copyright (c) 2001-2017, The GROMACS development team at
> > Uppsala University, Stockholm University and
> > the Royal Institute of Technology, Sweden.
> > check out http://www.gromacs.org for more information.
> >
> > GROMACS is free software; you can redistribute it and/or modify it
> > under the terms of the GNU Lesser General Public License
> > as published by the Free Software Foundation; either version 2.1
> > of the License, or (at your option) any later version.
> >
> > GROMACS: gmx mdrun, version 2018.2
> > Executable: /home/matthias/Documents/gromacs-2018.2/gpu_install/bin/gmx
> > Data prefix: /home/matthias/Documents/gromacs-2018.2/gpu_install
> > Working dir: /home/matthias/Documents/MD_simulations/D_EY_ext
> > Command line:
> > gmx mdrun -deffnm md_0_01_GPU
> >
> >
> > Back Off! I just backed up md_0_01_GPU.log to ./#md_0_01_GPU.log.10#
> >
> > -------------------------------------------------------
> > Program: gmx mdrun, version 2018.2
> > Source file: src/gromacs/gpu_utils/gpu_utils.cu (line 762)
> > Function: void findGpus(gmx_gpu_info_t*)
> >
> > Assertion failed:
> > Condition: cudaSuccess == cudaPeekAtLastError()
> > We promise to return with clean CUDA state!
> >
> > For more information and tips for troubleshooting, please check the
> GROMACS
> > website at http://www.gromacs.org/Documentation/Errors
> >
> > --
> > Gromacs Users mailing list
> >
> > * Please search the archive at
> > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> > posting!
> >
> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >
> > * For (un)subscribe requests visit
> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> > send a mail to gmx-users-request at gromacs.org.
> >
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
>
More information about the gromacs.org_gmx-users
mailing list