[gmx-users] GROMACS 2018.2 mdrun GPU Assertion failed: Condition: cudaSuccess == cudaPeekAtLastError()

Jia Hong jiahong96 at hotmail.com
Sat Aug 11 04:23:55 CEST 2018


Hi Mark,

I've set CUDA_VISIBLE_DEVICES to 1, the exact command was

export CUDA_VISIBLE_DEVICES=1

I chose 1 because that's the GPU id for the 1080 card. Then, in the same shell I ran the gmx command (both with and without -gpu_id 1 argument).

As for the log file, I thought it was md_0_01_GPU.log, which I sent as a follow-up message. Nonetheless,
here's the link to the file https://www.dropbox.com/s/xc8z09wvnl0314m/md_0_01_GPU.log?dl=0.
[https://www.dropbox.com/static/images/spectrum-icons/generated/content/content-unknown-large.png]<https://www.dropbox.com/s/xc8z09wvnl0314m/md_0_01_GPU.log?dl=0>

md_0_01_GPU.log<https://www.dropbox.com/s/xc8z09wvnl0314m/md_0_01_GPU.log?dl=0>
Shared with Dropbox
www.dropbox.com


Cheers,
Harry

________________________________
From: gromacs.org_gmx-users-bounces at maillist.sys.kth.se <gromacs.org_gmx-users-bounces at maillist.sys.kth.se> on behalf of Mark Abraham <mark.j.abraham at gmail.com>
Sent: Saturday, August 11, 2018 7:43 AM
To: gmx-users at gromacs.org
Subject: Re: [gmx-users] GROMACS 2018.2 mdrun GPU Assertion failed: Condition: cudaSuccess == cudaPeekAtLastError()

Hi,

To what did you set CUDA_VISIBLE_DEVICES? Just saying it didn't work
doesn't exclude that it couldn't have worked because it wasn't right :-)

Also, could you please share a full log file via a sharing service and post
a link. This fragment doesn't help us find out how the detection saw your
system, unfortunately.

Mark

On Wed, Aug 8, 2018, 10:53 Jia Hong <jiahong96 at hotmail.com> wrote:

> The log file:
>
>
> Log file opened on Wed Aug  8 16:33:34 2018
> Host: matthias-processing  pid: 28756  rank ID: 0  number of ranks:  1
>                       :-) GROMACS - gmx mdrun, 2018.2 (-:
>
>                             GROMACS is written by:
>      Emile Apol      Rossen Apostolov      Paul Bauer     Herman J.C.
> Berendsen
>     Par Bjelkmar    Aldert van Buuren   Rudi van Drunen     Anton Feenstra
>   Gerrit Groenhof    Aleksei Iupinov   Christoph Junghans   Anca Hamuraru
>  Vincent Hindriksen Dimitrios Karkoulis    Peter Kasson        Jiri Kraus
>   Carsten Kutzner      Per Larsson      Justin A. Lemkul    Viveca Lindahl
>   Magnus Lundborg   Pieter Meulenhoff    Erik Marklund      Teemu Murtola
>     Szilard Pall       Sander Pronk      Roland Schulz     Alexey Shvetsov
>    Michael Shirts     Alfons Sijbers     Peter Tieleman    Teemu Virolainen
>  Christian Wennberg    Maarten Wolf
>                            and the project leaders:
>         Mark Abraham, Berk Hess, Erik Lindahl, and David van der Spoel
>
> Copyright (c) 1991-2000, University of Groningen, The Netherlands.
> Copyright (c) 2001-2017, The GROMACS development team at
> Uppsala University, Stockholm University and
> the Royal Institute of Technology, Sweden.
> check out http://www.gromacs.org for more information.
>
> GROMACS is free software; you can redistribute it and/or modify it
> under the terms of the GNU Lesser General Public License
> as published by the Free Software Foundation; either version 2.1
> of the License, or (at your option) any later version.
>
> GROMACS:      gmx mdrun, version 2018.2
> Executable:   /home/matthias/Documents/gromacs-2018.2/gpu_install/bin/gmx
> Data prefix:  /home/matthias/Documents/gromacs-2018.2/gpu_install
> Working dir:  /home/matthias/Documents/MD_simulations/D_EY_ext
> Command line:
>   gmx mdrun -deffnm md_0_01_GPU
>
> GROMACS version:    2018.2
> Precision:          single
> Memory model:       64 bit
> MPI library:        thread_mpi
> OpenMP support:     enabled (GMX_OPENMP_MAX_THREADS = 64)
> GPU support:        CUDA
> SIMD instructions:  AVX_256
> FFT library:        fftw-3.3.8-sse2-avx-avx2-avx2_128-avx512
> RDTSCP usage:       enabled
> TNG support:        enabled
> Hwloc support:      disabled
> Tracing support:    disabled
> Built on:           2018-08-03 08:06:55
> Built by:           matthias at matthias-processing [CMAKE]
> Build OS/arch:      Linux 4.15.0-29-generic x86_64
> Build CPU vendor:   Intel
> Build CPU brand:    Intel(R) Xeon(R) CPU E5-1650 0 @ 3.20GHz
> Build CPU family:   6   Model: 45   Stepping: 7
> Build CPU features: aes apic avx clfsh cmov cx8 cx16 htt intel lahf mmx
> msr nonstop_tsc pcid pclmuldq pdcm pdpe1gb popcnt pse rdtscp sse2 sse3
> sse4.1 sse4.2 ssse3 tdt x2apic
> C compiler:         /usr/bin/cc GNU 5.4.0
> C compiler flags:    -mavx     -O3 -DNDEBUG -funroll-all-loops
> -fexcess-precision=fast
> C++ compiler:       /usr/bin/c++ GNU 5.4.0
> C++ compiler flags:  -mavx    -std=c++11   -O3 -DNDEBUG -funroll-all-loops
> -fexcess-precision=fast
> CUDA compiler:      /usr/local/cuda-9.0/bin/nvcc nvcc: NVIDIA (R) Cuda
> compiler driver;Copyright (c) 2005-2017 NVIDIA Corporation;Built on
> Fri_Sep__1_21:08:03_CDT_2017;Cuda compilation tools, release 9.0, V9.0.176
> CUDA compiler
> flags:-gencode;arch=compute_30,code=sm_30;-gencode;arch=compute_35,code=sm_35;-gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_52,code=sm_52;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_70,code=compute_70;-use_fast_math;-D_FORCE_INLINES;;
> ;-mavx;-std=c++11;-O3;-DNDEBUG;-funroll-all-loops;-fexcess-precision=fast;
> CUDA driver:        9.0
> CUDA runtime:       9.0
>
>
> ________________________________
> From: gromacs.org_gmx-users-bounces at maillist.sys.kth.se <
> gromacs.org_gmx-users-bounces at maillist.sys.kth.se> on behalf of Jia Hong <
> jiahong96 at hotmail.com>
> Sent: Wednesday, August 8, 2018 3:50 PM
> To: gromacs.org_gmx-users at maillist.sys.kth.se
> Subject: [gmx-users] GROMACS 2018.2 mdrun GPU Assertion failed: Condition:
> cudaSuccess == cudaPeekAtLastError()
>
> Hi,
>
> I've recently compiled gromacs 2018.2 on ubuntu 16.04.5 LTS with CUDA 9.0
> on an Nvidia 1080. I've got the error below on the mdrun command. I've
> searched around (
> https://mailman-1.sys.kth.se/pipermail/gromacs.org_gmx-users/2018-February/118631.html)
> and I tried the suggested export CUDA_VISIBLE_DEVICES without any luck.
>
> I currently have 2 nvidia GPUs sitting inside the machine, the first
> doesn't have CUDA support (old graphics card) while the second is the
> aforementioned 1080. I've also verified that the 1080 has gpu id 1, and ran
> the command:
>
>   gmx mdrun -deffnm md_0_01_GPU -gpu_id 1
>
> which gives the same error.
> I've attached the log file too.
>
> (I don't know if this helps, but I manage to compile a CUDA test program
> to verify that my CUDA installation isn't broken:
> http://ramblingsofagamedevstudent.blogspot.com/2014/03/set-up-visual-studio-2012-for-cuda.html
> )
>
> Cheers,
> Harry
>
> The output:
>
>
> :-) GROMACS - gmx mdrun, 2018.2 (-:
>
>                             GROMACS is written by:
>      Emile Apol      Rossen Apostolov      Paul Bauer     Herman J.C.
> Berendsen
>     Par Bjelkmar    Aldert van Buuren   Rudi van Drunen     Anton Feenstra
>   Gerrit Groenhof    Aleksei Iupinov   Christoph Junghans   Anca Hamuraru
>  Vincent Hindriksen Dimitrios Karkoulis    Peter Kasson        Jiri Kraus
>   Carsten Kutzner      Per Larsson      Justin A. Lemkul    Viveca Lindahl
>   Magnus Lundborg   Pieter Meulenhoff    Erik Marklund      Teemu Murtola
>     Szilard Pall       Sander Pronk      Roland Schulz     Alexey Shvetsov
>    Michael Shirts     Alfons Sijbers     Peter Tieleman    Teemu Virolainen
>  Christian Wennberg    Maarten Wolf
>                            and the project leaders:
>         Mark Abraham, Berk Hess, Erik Lindahl, and David van der Spoel
>
> Copyright (c) 1991-2000, University of Groningen, The Netherlands.
> Copyright (c) 2001-2017, The GROMACS development team at
> Uppsala University, Stockholm University and
> the Royal Institute of Technology, Sweden.
> check out http://www.gromacs.org for more information.
>
> GROMACS is free software; you can redistribute it and/or modify it
> under the terms of the GNU Lesser General Public License
> as published by the Free Software Foundation; either version 2.1
> of the License, or (at your option) any later version.
>
> GROMACS:      gmx mdrun, version 2018.2
> Executable:   /home/matthias/Documents/gromacs-2018.2/gpu_install/bin/gmx
> Data prefix:  /home/matthias/Documents/gromacs-2018.2/gpu_install
> Working dir:  /home/matthias/Documents/MD_simulations/D_EY_ext
> Command line:
>   gmx mdrun -deffnm md_0_01_GPU
>
>
> Back Off! I just backed up md_0_01_GPU.log to ./#md_0_01_GPU.log.10#
>
> -------------------------------------------------------
> Program:     gmx mdrun, version 2018.2
> Source file: src/gromacs/gpu_utils/gpu_utils.cu (line 762)
> Function:    void findGpus(gmx_gpu_info_t*)
>
> Assertion failed:
> Condition: cudaSuccess == cudaPeekAtLastError()
> We promise to return with clean CUDA state!
>
> For more information and tips for troubleshooting, please check the GROMACS
> website at http://www.gromacs.org/Documentation/Errors
>
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
>
--
Gromacs Users mailing list

* Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!

* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists

* For (un)subscribe requests visit
https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a mail to gmx-users-request at gromacs.org.


More information about the gromacs.org_gmx-users mailing list