[gmx-users] WG: Issue with CUDA and gromacs

[Szilárd Páll]

On Thu, Jan 31, 2019 at 2:14 PM Szilárd Páll <pall.szilard at gmail.com> wrote:
>
> On Wed, Jan 30, 2019 at 5:15 PM Tafelmeier, Stefanie
> <Stefanie.Tafelmeier at zae-bayern.de> wrote:
> >
> > Dear all,
> >
> > We are facing an issue with the CUDA toolkit.
> > We tried several combinations of gromacs versions and CUDA Toolkits. No Toolkit older than 9.2 was possible to try as there are no driver for nvidia available for a Quadro P6000.
> > Gromacs
>
> Install the latest 410.xx drivers and it will work; the NVIDIA driver
> download website (https://www.nvidia.com/Download/index.aspx)
> recommends 410.93.
>
> Here's a system with CUDA 10-compatible driver running o a system with
> a P6000: https://termbin.com/ofzo

Sorry, I misread that as "CUDA >=9.2 was not possible".

Note that the driver is backward compatible, so you can use a new
driver with older CUDA versions.

Also note that the oldest driver NVIDIA claims to have P6000 support
is 390.59 which is, as far as I know, one gen older than the 396 that
the CUDA 9.2 toolkit came with. This is however, not something I'd
recommend pursuing, use a new driver from the official site with any
CUDA version that GROMACS supports and it should be fine.

>
> > CUDA
> >
> > Error message
> >
> > 2019
> >
> > 10.0
> >
> > gmx mdrun:
> > Assertion failed:
> > Condition: stat == cudaSuccess
> > Asynchronous H2D copy failed
> >
> > 2019
> >
> > 9.2
> >
> > gmx mdrun:
> > Assertion failed:
> > Condition: stat == cudaSuccess
> > Asynchronous H2D copy failed
> >
> > 2018.5
> >
> > 9.2
> >
> > gmx mdrun: Fatal error:
> > HtoD cudaMemcpyAsync failed: invalid argument
>
> Can we get some more details on these, please? complete log files
> would be a good start.
>
> > 5.1.5
> >
> > 9.2
> >
> > Installation make: nvcc fatal   : Unsupported gpu architecture 'compute_20'*
> >
> > 2016.2
> >
> > 9.2
> >
> > Installation make: nvcc fatal   : Unsupported gpu architecture 'compute_20'*
> >
> >
> > *We also tried to set the target CUDA architectures as described in the installation guide (manual.gromacs.org/documentation/2019/install-guide/index.html). Unfortunately it didn't work.
>
> What does it mean that it didn't work? Can you share the command you
> used and what exactly did not work?
>
> For the P6000 which is a "compute capability 6.1" device (for anyone
> who needs to look it up, go here:
> https://developer.nvidia.com/cuda-gpus), you should set
> cmake ../ -DGMX_CUDA_TARGET_SM="61"
>
> --
> Szilárd
>
> > Performing simulations on CPU only always works, yet of cause are more slowly than they could be with additionally using the GPU.
> > The issue #2761 (https://redmine.gromacs.org/issues/2762) seems similar to our problem.
> > Even though this issue is still open, we wanted to ask if you can give us any information about how to solve this problem?
> >
> > Many thanks in advance.
> > Best regards,
> > Stefanie Tafelmeier
> >
> >
> > Further details if necessary:
> > The workstation:
> > 2 x Xeon Gold 6152 @ 3,7Ghz (22 K, 44Th, AVX512)
> > Nvidia Quadro P6000 with 3840 Cuda-Cores
> >
> > The simulations system:
> > Long chain alkanes (previously used with gromacs 5.1.5 and CUDA 7.5 - worked perfectly)
> >
> >
> >
> >
> > ZAE Bayern
> > Stefanie Tafelmeier
> > Bereich Energiespeicherung/Division Energy Storage
> > Thermische Energiespeicher/Thermal Energy Storage
> > Walther-Meißner-Str. 6
> > 85748 Garching
> >
> > Tel.: +49 89 329442-75
> > Fax: +49 89 329442-12
> > Stefanie.tafelmeier at zae-bayern.de<mailto:Stefanie.tafelmeier at zae-bayern.de>
> > http://www.zae-bayern.de<http://www.zae-bayern.de/>
> >
> >
> > ZAE Bayern - Bayerisches Zentrum für Angewandte Energieforschung e. V.
> > Vorstand/Board:
> > Prof. Dr. Hartmut Spliethoff (Vorsitzender/Chairman),
> > Prof. Dr. Vladimir Dyakonov
> > Sitz/Registered Office: Würzburg
> > Registergericht/Register Court: Amtsgericht Würzburg
> > Registernummer/Register Number: VR 1386
> >
> > Sämtliche Willenserklärungen, z. B. Angebote, Aufträge, Anträge und Verträge, sind für das ZAE Bayern nur in schriftlicher und ordnungsgemäß unterschriebener Form rechtsverbindlich. Diese E-Mail ist ausschließlich zur Nutzung durch den/die vorgenannten Empfänger bestimmt. Jegliche unbefugte Offenbarung, Nutzung oder Verbreitung, sei es insgesamt oder teilweise, ist untersagt. Sollten Sie diese E-Mail irrtümlich erhalten haben, benachrichtigen Sie bitte unverzüglich den Absender und löschen Sie diese E-Mail.
> >
> > Any declarations of intent, such as quotations, orders, applications and contracts, are legally binding for ZAE Bayern only if expressed in a written and duly signed form. This e-mail is intended solely for use by the recipient(s) named above. Any unauthorised disclosure, use or dissemination, whether in whole or in part, is prohibited. If you have received this e-mail in error, please notify the sender immediately and delete this e-mail.
> >
> >
> > --
> > Gromacs Users mailing list
> >
> > * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!
> >
> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >
> > * For (un)subscribe requests visit
> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a mail to gmx-users-request at gromacs.org.