[gmx-users] WG: Issue with CUDA and gromacs

Wed Feb 27 16:55:37 CET 2019

Dear Szilárd

I wrote before because of the error message when keeping occurring for mdrun simulation:

Assertion failed:
Condition: stat == cudaSuccess
Asynchronous H2D copy failed

As I mentioned all Versions to install (Gormacs, Cuda, nvcc, gcc) are the newest once now.

If I run mdrun without further settings it will lead to the error message. If I run it and choose the thread amount directly the mdrun is performing well. But only for –nt numbers between 1 – 22. Higher ones again lead to the before mentioned error message.

In order to investigate in more detail, I tried different versions for –nt, –ntmpi – ntomp also combined with –npme: 
-	The best performance in the sense of ns/day is with –nt 22 respectively –ntomp 22 alone. But then only 22 threads are involved. Which is fine if I run more than one mdrun simultaneously, as I can distribute the other 66 threads. The GPU usage is then around 65%.
-	A similar good performance is reached with mdrun  -ntmpi 4 -ntomp 18 -npme 1 -pme gpu -nb gpu. But then 44 threads are involved. The GPU usage is then around 50%.

I read the information on http://manual.gromacs.org/documentation/5.1/user-guide/mdrun-performance.html which was very helpful, bur some thinsg are not still not clear now to me:
I was wondering if there is any other enhancement of the performance? Or what is the reason, that –nt maximum is at 22 threads? Could this be connected to the sockets (see details below)? 

Additionally, just I was following the discussion:
I also had the same problem as Lalehan. So I reinstalled with -DGMX_OPENMP_MAX_THREADS=128. This changed the error message:

you are using 72 openmp threads, which is larger than GMX_OPENMP_MAX_THREADS (64). Decrease the number of OpenMP threads or rebuild GROMACS with a larger value for GMX_OPENMP_MAX_THREADS.

To:

Assertion failed:
Condition: stat == cudaSuccess
Asynchronous H2D copy failed

Many thanks for your suggestion before and I would appreciate if you might have a hint for these as well.

Best regards,
Steffi

The workstation details are:
Running on 1 node with total 44 cores, 88 logical cores, 1 compatible GPU
Hardware detected:

  CPU info:
    Vendor: Intel
    Brand:  Intel(R) Xeon(R) Gold 6152 CPU @ 2.10GHz
    Family: 6   Model: 85   Stepping: 4
    Features: aes apic avx avx2 avx512f avx512cd avx512bw avx512vl clfsh cmov cx8 cx16 f16c fma hle htt intel lahf mmx msr nonstop_tsc pcid pclmuldq pdcm pdpe1gb popcnt pse rdrnd rdtscp rtm sse2 sse3 sse4.1 sse4.2 ssse3 tdt x2apic

    Number of AVX-512 FMA units: 2
  Hardware topology: Basic
    Sockets, cores, and logical processors:
      Socket  0: [   0  44] [   1  45] [   2  46] [   3  47] [   4  48] [   5  49] [   6  50] [   7  51] [   8  52] [   9  53] [  10  54] [  11  55] [  12  56] [  13  57] [  14  58] [  15  59] [  16  60] [  17  61] [  18  62] [  19  63] [  20  64] [  21  65]
      Socket  1: [  22  66] [  23  67] [  24  68] [  25  69] [  26  70] [  27  71] [  28  72] [  29  73] [  30  74] [  31  75] [  32  76] [  33  77] [  34  78] [  35  79] [  36  80] [  37  81] [  38  82] [  39  83] [  40  84] [  41  85] [  42  86] [  43  87]
  GPU info:
    Number of GPUs detected: 1
    #0: NVIDIA Quadro P6000, compute cap.: 6.1, ECC:  no, stat: compatible

-----Ursprüngliche Nachricht-----
Von: gromacs.org_gmx-users-bounces at maillist.sys.kth.se [mailto:gromacs.org_gmx-users-bounces at maillist.sys.kth.se] Im Auftrag von Szilárd Páll
Gesendet: Donnerstag, 31. Januar 2019 17:15
An: Discussion list for GROMACS users
Betreff: Re: [gmx-users] WG: Issue with CUDA and gromacs

On Thu, Jan 31, 2019 at 2:14 PM Szilárd Páll <pall.szilard at gmail.com> wrote:
>
> On Wed, Jan 30, 2019 at 5:15 PM Tafelmeier, Stefanie
> <Stefanie.Tafelmeier at zae-bayern.de> wrote:
> >
> > Dear all,
> >
> > We are facing an issue with the CUDA toolkit.
> > We tried several combinations of gromacs versions and CUDA Toolkits. No Toolkit older than 9.2 was possible to try as there are no driver for nvidia available for a Quadro P6000.
> > Gromacs
>
> Install the latest 410.xx drivers and it will work; the NVIDIA driver
> download website (https://www.nvidia.com/Download/index.aspx)
> recommends 410.93.
>
> Here's a system with CUDA 10-compatible driver running o a system with
> a P6000: https://termbin.com/ofzo

Sorry, I misread that as "CUDA >=9.2 was not possible".

Note that the driver is backward compatible, so you can use a new
driver with older CUDA versions.

Also note that the oldest driver NVIDIA claims to have P6000 support
is 390.59 which is, as far as I know, one gen older than the 396 that
the CUDA 9.2 toolkit came with. This is however, not something I'd
recommend pursuing, use a new driver from the official site with any
CUDA version that GROMACS supports and it should be fine.

>
> > CUDA
> >
> > Error message
> >
> > 2019
> >
> > 10.0
> >
> > gmx mdrun:
> > Assertion failed:
> > Condition: stat == cudaSuccess
> > Asynchronous H2D copy failed
> >
> > 2019
> >
> > 9.2
> >
> > gmx mdrun:
> > Assertion failed:
> > Condition: stat == cudaSuccess
> > Asynchronous H2D copy failed
> >
> > 2018.5
> >
> > 9.2
> >
> > gmx mdrun: Fatal error:
> > HtoD cudaMemcpyAsync failed: invalid argument
>
> Can we get some more details on these, please? complete log files
> would be a good start.
>
> > 5.1.5
> >
> > 9.2
> >
> > Installation make: nvcc fatal   : Unsupported gpu architecture 'compute_20'*
> >
> > 2016.2
> >
> > 9.2
> >
> > Installation make: nvcc fatal   : Unsupported gpu architecture 'compute_20'*
> >
> >
> > *We also tried to set the target CUDA architectures as described in the installation guide (manual.gromacs.org/documentation/2019/install-guide/index.html). Unfortunately it didn't work.
>
> What does it mean that it didn't work? Can you share the command you
> used and what exactly did not work?
>
> For the P6000 which is a "compute capability 6.1" device (for anyone
> who needs to look it up, go here:
> https://developer.nvidia.com/cuda-gpus), you should set
> cmake ../ -DGMX_CUDA_TARGET_SM="61"
>
> --
> Szilárd
>
> > Performing simulations on CPU only always works, yet of cause are more slowly than they could be with additionally using the GPU.
> > The issue #2761 (https://redmine.gromacs.org/issues/2762) seems similar to our problem.
> > Even though this issue is still open, we wanted to ask if you can give us any information about how to solve this problem?
> >
> > Many thanks in advance.
> > Best regards,
> > Stefanie Tafelmeier
> >
> >
> > Further details if necessary:
> > The workstation:
> > 2 x Xeon Gold 6152 @ 3,7Ghz (22 K, 44Th, AVX512)
> > Nvidia Quadro P6000 with 3840 Cuda-Cores
> >
> > The simulations system:
> > Long chain alkanes (previously used with gromacs 5.1.5 and CUDA 7.5 - worked perfectly)
> >
> >
> >
> >
> > ZAE Bayern
> > Stefanie Tafelmeier
> > Bereich Energiespeicherung/Division Energy Storage
> > Thermische Energiespeicher/Thermal Energy Storage
> > Walther-Meißner-Str. 6
> > 85748 Garching
> >
> > Tel.: +49 89 329442-75
> > Fax: +49 89 329442-12
> > Stefanie.tafelmeier at zae-bayern.de<mailto:Stefanie.tafelmeier at zae-bayern.de>
> > http://www.zae-bayern.de<http://www.zae-bayern.de/>
> >
> >
> > ZAE Bayern - Bayerisches Zentrum für Angewandte Energieforschung e. V.
> > Vorstand/Board:
> > Prof. Dr. Hartmut Spliethoff (Vorsitzender/Chairman),
> > Prof. Dr. Vladimir Dyakonov
> > Sitz/Registered Office: Würzburg
> > Registergericht/Register Court: Amtsgericht Würzburg
> > Registernummer/Register Number: VR 1386
> >
> > Sämtliche Willenserklärungen, z. B. Angebote, Aufträge, Anträge und Verträge, sind für das ZAE Bayern nur in schriftlicher und ordnungsgemäß unterschriebener Form rechtsverbindlich. Diese E-Mail ist ausschließlich zur Nutzung durch den/die vorgenannten Empfänger bestimmt. Jegliche unbefugte Offenbarung, Nutzung oder Verbreitung, sei es insgesamt oder teilweise, ist untersagt. Sollten Sie diese E-Mail irrtümlich erhalten haben, benachrichtigen Sie bitte unverzüglich den Absender und löschen Sie diese E-Mail.
> >
> > Any declarations of intent, such as quotations, orders, applications and contracts, are legally binding for ZAE Bayern only if expressed in a written and duly signed form. This e-mail is intended solely for use by the recipient(s) named above. Any unauthorised disclosure, use or dissemination, whether in whole or in part, is prohibited. If you have received this e-mail in error, please notify the sender immediately and delete this e-mail.
> >
> >
> > --
> > Gromacs Users mailing list
> >
> > * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!
> >
> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >
> > * For (un)subscribe requests visit
> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a mail to gmx-users-request at gromacs.org.
-- 
Gromacs Users mailing list

* Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!

* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists

* For (un)subscribe requests visit
https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a mail to gmx-users-request at gromacs.org.