[gmx-users] GPU Not Being Utilized during mdrun

Vito Spadavecchio spadavecchio at gmail.com
Thu Jul 28 20:04:11 CEST 2016


Just as an update: I upgraded my cuda7.5 installation to cuda8, (keeping
the 367.27 driver), and am now getting ~150ns/day on the same system. This
is still ~1/2 of what I would expect given the benchmarks I've seen for the
1080gtx, but I will take a ~10x increase in throughput for now!

On Thu, Jul 28, 2016 at 9:10 AM, Szilárd Páll <pall.szilard at gmail.com>
wrote:

> Note that the CPU-GPU load balancing scaled the cut-off to 1.5 nm
> increasing the GPU load by about 3x (see the table on line 550 of the
> log file you posted).
> I find it somewhat suspicious that your CPU seems so slow, but given
> that you have a GTX 1080, that may just be normal.
> --
> Szilárd
>
>
> On Sun, Jul 17, 2016 at 10:57 PM, Vito Spadavecchio
> <spadavecchio at gmail.com> wrote:
> > Just wanted to ping this and see if anyone had any ideas :)
> >
> > On Fri, Jul 15, 2016 at 10:30 AM, Vito Spadavecchio <
> spadavecchio at gmail.com>
> > wrote:
> >
> >> My apologies! I meant to include the logfile as well. You can view it
> >> here:
> >>
> >> https://www.dropbox.com/s/8ddm226gpgs5kpr/md_0_2.log?dl=0
> >>
> >> Thanks!
> >>
> >> On Fri, Jul 15, 2016 at 2:45 AM, Justin Lemkul <jalemkul at vt.edu> wrote:
> >>
> >>>
> >>>
> >>> On 7/14/16 10:35 PM, Vito Spadavecchio wrote:
> >>>
> >>>> Szilard
> >>>>
> >>>> You are correct; I misspoke. It appears that the GPU is doing
> >>>> *something*,
> >>>> but I it would seem it is severely under performing (near the point
> where
> >>>> its basically doing nothing).
> >>>>
> >>>> the entire log, not just parts, in particular we want to see the
> >>>>
> >>>>> header, perf table
> >>>>>
> >>>>>
> >>>> xyz at Turing:~/Desktop/xyz_MD/APO_xyz$ gmx mdrun -deffnm md_0_2 -nb gpu
> >>>>                    :-) GROMACS - gmx mdrun, VERSION 5.1.2 (-:
> >>>>
> >>>>                             GROMACS is written by:
> >>>>      Emile Apol      Rossen Apostolov  Herman J.C. Berendsen    Par
> >>>> Bjelkmar
> >>>>  Aldert van Buuren   Rudi van Drunen     Anton Feenstra   Sebastian
> >>>> Fritsch
> >>>>   Gerrit Groenhof   Christoph Junghans   Anca Hamuraru    Vincent
> >>>> Hindriksen
> >>>>  Dimitrios Karkoulis    Peter Kasson        Jiri Kraus      Carsten
> >>>> Kutzner
> >>>>
> >>>>     Per Larsson      Justin A. Lemkul   Magnus Lundborg   Pieter
> >>>> Meulenhoff
> >>>>    Erik Marklund      Teemu Murtola       Szilard Pall       Sander
> Pronk
> >>>>    Roland Schulz     Alexey Shvetsov     Michael Shirts     Alfons
> >>>> Sijbers
> >>>>    Peter Tieleman    Teemu Virolainen  Christian Wennberg    Maarten
> Wolf
> >>>>                            and the project leaders:
> >>>>         Mark Abraham, Berk Hess, Erik Lindahl, and David van der Spoel
> >>>>
> >>>> Copyright (c) 1991-2000, University of Groningen, The Netherlands.
> >>>> Copyright (c) 2001-2015, The GROMACS development team at
> >>>> Uppsala University, Stockholm University and
> >>>> the Royal Institute of Technology, Sweden.
> >>>> check out http://www.gromacs.org for more information.
> >>>>
> >>>> GROMACS is free software; you can redistribute it and/or modify it
> >>>> under the terms of the GNU Lesser General Public License
> >>>> as published by the Free Software Foundation; either version 2.1
> >>>> of the License, or (at your option) any later version.
> >>>>
> >>>> GROMACS:      gmx mdrun, VERSION 5.1.2
> >>>> Executable:   /usr/local/gromacs/bin/gmx
> >>>> Data prefix:  /usr/local/gromacs
> >>>> Command line:
> >>>>   gmx mdrun -deffnm md_0_2 -nb gpu
> >>>>
> >>>>
> >>>> Back Off! I just backed up md_0_2.log to ./#md_0_2.log.1#
> >>>>
> >>>> Running on 1 node with total 4 cores, 8 logical cores, 1 compatible
> GPU
> >>>> Hardware detected:
> >>>>   CPU info:
> >>>>     Vendor: GenuineIntel
> >>>>     Brand:  Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
> >>>>     SIMD instructions most likely to fit this hardware: AVX2_256
> >>>>     SIMD instructions selected at GROMACS compile time: AVX2_256
> >>>>   GPU info:
> >>>>     Number of GPUs detected: 1
> >>>>     #0: NVIDIA GeForce GTX 1080, compute cap.: 6.1, ECC:  no, stat:
> >>>> compatible
> >>>>
> >>>> Reading file md_0_2.tpr, VERSION 5.1.2 (single precision)
> >>>> Changing nstlist from 10 to 40, rlist from 1 to 1.099
> >>>>
> >>>> Using 1 MPI thread
> >>>> Using 8 OpenMP threads
> >>>>
> >>>> 1 compatible GPU is present, with ID 0
> >>>> 1 GPU auto-selected for this run.
> >>>> Mapping of GPU ID to the 1 PP rank in this node: 0
> >>>>
> >>>>
> >>>> Back Off! I just backed up md_0_2.trr to ./#md_0_2.trr.1#
> >>>>
> >>>> Back Off! I just backed up md_0_2.xtc to ./#md_0_2.xtc.1#
> >>>>
> >>>> Back Off! I just backed up md_0_2.edr to ./#md_0_2.edr.1#
> >>>> starting mdrun 'Protein in water'
> >>>> 500000 steps,   1000.0 ps.
> >>>>
> >>>>
> >>>> - the output of nvidia-smi dmon executing *during* mdrun's execution
> >>>>
> >>>>
> >>>> heres a snapshot:
> >>>>
> >>>> # gpu   pwr  temp    sm   mem   enc   dec  mclk  pclk
> >>>>
> >>>>> # Idx     W     C     %     %     %     %   MHz   MHz
> >>>>>  0   117    51    56     3     0     0  4513  1847
> >>>>>     0   115    52    54     3     0     0  4513  1847
> >>>>>     0   117    52    47     3     0     0  4513  1847
> >>>>>     0   101    52    52     3     0     0  4513  1848
> >>>>>     0   119    53    54     3     0     0  4513  1847
> >>>>>     0   115    53    54     3     0     0  4513  1848
> >>>>>     0   106    54    54     3     0     0  4513  1848
> >>>>>     0   118    54    54     3     0     0  4513  1849
> >>>>>     0    94    54    52     3     0     0  4513  1848
> >>>>>     0   110    55    52     3     0     0  4513  1849
> >>>>>     0   116    55    53     3     0     0  4513  1854
> >>>>>     0   126    55    51     3     0     0  4513  1854
> >>>>>     0   119    56    51     3     0     0  4513  1855
> >>>>>     0   116    56    52     3     0     0  4513  1855
> >>>>>     0   116    56    53     3     0     0  4513  1855
> >>>>>     0   120    57    53     3     0     0  4513  1855
> >>>>>     0   113    57    54     3     0     0  4513  1855
> >>>>>
> >>>>>
> >>>> and here's the tail of the performance in .log
> >>>>
> >>>> Reading file md_0_2.tpr, VERSION 5.1.2 (single precision)
> >>>>
> >>>>> Changing nstlist from 10 to 40, rlist from 1 to 1.099
> >>>>> Using 1 MPI thread
> >>>>> Using 8 OpenMP threads
> >>>>> 1 compatible GPU is present, with ID 0
> >>>>> 1 GPU auto-selected for this run.
> >>>>> Mapping of GPU ID to the 1 PP rank in this node: 0
> >>>>>
> >>>>> Back Off! I just backed up md_0_2.trr to ./#md_0_2.trr.1#
> >>>>> Back Off! I just backed up md_0_2.xtc to ./#md_0_2.xtc.1#
> >>>>> Back Off! I just backed up md_0_2.edr to ./#md_0_2.edr.1#
> >>>>> starting mdrun 'Protein in water'
> >>>>> 500000 steps,   1000.0 ps.
> >>>>> Writing final coordinates.
> >>>>> Back Off! I just backed up md_0_2.gro to ./#md_0_2.gro.1#
> >>>>>                Core t (s)   Wall t (s)        (%)
> >>>>>        Time:    25285.344     3409.337      741.6
> >>>>>                          56:49
> >>>>>                  (ns/day)    (hour/ns)
> >>>>> Performance:       25.342        0.947
> >>>>> gcq#11: "She's Not Bad, She's Just Genetically Mean" (Captain
> Beefheart)
> >>>>>
> >>>>>
> >>>>> As we can see, there is *something* running on the GPU (process id:
> >>>> 30784),
> >>>> but its never more than 135MiB of memory being used and only uses a
> very
> >>>> small portion of the gpus power.
> >>>>
> >>>> Any thoughts are greatly appreciated!
> >>>>
> >>>>
> >>> You cut out the part of the .log file that shows how mdrun actually
> spent
> >>> its time.  This is perhaps the most valuable information.  This is why
> >>> Szilard asked for you to share a complete .log file (upload it
> somewhere,
> >>> post a URL) so we can see all relevant details.
> >>>
> >>> -Justin
> >>>
> >>>
> >>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Thu, Jul 14, 2016 at 4:54 PM, Szilárd Páll <pall.szilard at gmail.com
> >
> >>>> wrote:
> >>>>
> >>>> Hi,
> >>>>>
> >>>>> If mdrun claims it's using the GPU and you see no errors (and results
> >>>>> look reasonable), the GPU is likely being used. That nvidia-smi is
> not
> >>>>> showing it is admittedly strange. However, this truncated log output
> >>>>> does not confirm much, e.g. it does not show the performance table
> and
> >>>>> the final performance.
> >>>>>
> >>>>> Could you please run again and look at (and share if approapriate):
> >>>>> - the entire log, not just parts, in particular we want to see the
> >>>>> header, perf table
> >>>>> - the output of nvidia-smi dmon executing *during* mdrun's execution
> >>>>> (this monitors the device continuously)
> >>>>>
> >>>>> Cheers,
> >>>>>
> >>>>> --
> >>>>> Szilárd
> >>>>>
> >>>>>
> >>>>> On Thu, Jul 14, 2016 at 8:28 PM, Vito Spadavecchio
> >>>>> <spadavecchio at gmail.com> wrote:
> >>>>>
> >>>>>> Hello
> >>>>>>
> >>>>>> After building GROMACS with GPU support and running some simple MD,
> I
> >>>>>> am
> >>>>>> only getting ~20ns/day (which is what I would expect to get out of
> my
> >>>>>> i7-6700k cpu only. My machine currently has a 1080 GTX in it, which
> >>>>>>
> >>>>> should
> >>>>>
> >>>>>> even on moderately large systems, be getting ~100ns/day)
> >>>>>>
> >>>>>>  I've tried launching a mdrun with with the following command:
> >>>>>>
> >>>>>>   gmx mdrun -deffnm md_0_1 -gpu_id 0 -nb gpu
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> With the following in my md.mdp file:
> >>>>>>
> >>>>>> ; Neighborsearching
> >>>>>>
> >>>>>> cutoff-scheme   = Verlet
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> The top of the log file for the mdrun notes:
> >>>>>>
> >>>>>> Running on 1 node with total 4 cores, 8 logical cores, 1 compatible
> GPU
> >>>>>>
> >>>>>>> Hardware detected:
> >>>>>>>   CPU info:
> >>>>>>>     Vendor: GenuineIntel
> >>>>>>>     Brand:  Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
> >>>>>>>     SIMD instructions most likely to fit this hardware: AVX2_256
> >>>>>>>     SIMD instructions selected at GROMACS compile time: AVX2_256
> >>>>>>>   GPU info:
> >>>>>>>     Number of GPUs detected: 1
> >>>>>>>     #0: NVIDIA GeForce GTX 1080, compute cap.: 6.1, ECC:  no, stat:
> >>>>>>> compatible
> >>>>>>> Reading file md_0_1.tpr, VERSION 5.1.2 (single precision)
> >>>>>>> Changing nstlist from 10 to 40, rlist from 1 to 1.099
> >>>>>>> Using 1 MPI thread
> >>>>>>> Using 8 OpenMP threads
> >>>>>>> 1 GPU user-selected for this run.
> >>>>>>> Mapping of GPU ID to the 1 PP rank in this node: 0
> >>>>>>>
> >>>>>>>
> >>>>>> The output of *nvidia-smi *on my machine is:
> >>>>>>
> >>>>>> hu Jul 14 11:25:04 2016
> >>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>
> +-----------------------------------------------------------------------------+
> >>>>>
> >>>>>> | NVIDIA-SMI 367.27                 Driver Version: 367.27
> >>>>>>>    |
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>
> |-------------------------------+----------------------+----------------------+
> >>>>>
> >>>>>> | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile
> >>>>>>>
> >>>>>> Uncorr.
> >>>>>
> >>>>>> ECC |
> >>>>>>> | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util
> >>>>>>>
> >>>>>> Compute
> >>>>>
> >>>>>> M. |
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>
> |===============================+======================+======================|
> >>>>>
> >>>>>> |   0  GeForce GTX 1080    Off  | 0000:01:00.0      On |
> >>>>>>>  N/A |
> >>>>>>> | 30%   41C    P8     7W / 180W |    730MiB /  8113MiB |      3%
> >>>>>>>  Default |
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>
> +-------------------------------+----------------------+----------------------+
> >>>>>
> >>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>
> +-----------------------------------------------------------------------------+
> >>>>>
> >>>>>> | Processes:
>  GPU
> >>>>>>> Memory |
> >>>>>>> |  GPU       PID  Type  Process name
> >>>>>>>  Usage
> >>>>>>>    |
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>
> |=============================================================================|
> >>>>>
> >>>>>> |    0      1035    G   /usr/lib/xorg/Xorg
> >>>>>>> 424MiB |
> >>>>>>> |    0      2132    G   compiz
> >>>>>>> 103MiB |
> >>>>>>> |    0      5296    G   ...s-passed-by-fd
> --v8-snapshot-passed-by-fd
> >>>>>>>  59MiB |
> >>>>>>> |    0     14079    G   ...s-passed-by-fd
> --v8-snapshot-passed-by-fd
> >>>>>>>  31MiB |
> >>>>>>> |    0     14234    G   ...ing<BrotliEncoding
> --disable-features=Doc
> >>>>>>> 108MiB |
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>
> +-----------------------------------------------------------------------------+
> >>>>>
> >>>>>>
> >>>>>>
> >>>>>> and output of *nvcc --version* is:
> >>>>>>
> >>>>>> nvcc: NVIDIA (R) Cuda compiler driver
> >>>>>>
> >>>>>>> Copyright (c) 2005-2015 NVIDIA Corporation
> >>>>>>> Built on Tue_Aug_11_14:27:32_CDT_2015
> >>>>>>> Cuda compilation tools, release 7.5, V7.5.17
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Does anyone have any ideas what the problem might be?
> >>>>>>
> >>>>>> Thanks!
> >>>>>> --
> >>>>>> Gromacs Users mailing list
> >>>>>>
> >>>>>> * Please search the archive at
> >>>>>>
> >>>>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> >>>>> posting!
> >>>>>
> >>>>>>
> >>>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >>>>>>
> >>>>>> * For (un)subscribe requests visit
> >>>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users
> or
> >>>>>>
> >>>>> send a mail to gmx-users-request at gromacs.org.
> >>>>> --
> >>>>> Gromacs Users mailing list
> >>>>>
> >>>>> * Please search the archive at
> >>>>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> >>>>> posting!
> >>>>>
> >>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >>>>>
> >>>>> * For (un)subscribe requests visit
> >>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users
> or
> >>>>> send a mail to gmx-users-request at gromacs.org.
> >>>>>
> >>>>>
> >>> --
> >>> ==================================================
> >>>
> >>> Justin A. Lemkul, Ph.D.
> >>> Ruth L. Kirschstein NRSA Postdoctoral Fellow
> >>>
> >>> Department of Pharmaceutical Sciences
> >>> School of Pharmacy
> >>> Health Sciences Facility II, Room 629
> >>> University of Maryland, Baltimore
> >>> 20 Penn St.
> >>> Baltimore, MD 21201
> >>>
> >>> jalemkul at outerbanks.umaryland.edu | (410) 706-7441
> >>> http://mackerell.umaryland.edu/~jalemkul
> >>>
> >>> ==================================================
> >>>
> >>> --
> >>> Gromacs Users mailing list
> >>>
> >>> * Please search the archive at
> >>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> >>> posting!
> >>>
> >>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >>>
> >>> * For (un)subscribe requests visit
> >>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> >>> send a mail to gmx-users-request at gromacs.org.
> >>>
> >>
> >>
> > --
> > Gromacs Users mailing list
> >
> > * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
> >
> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >
> > * For (un)subscribe requests visit
> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
>


More information about the gromacs.org_gmx-users mailing list