[gmx-users] GPU Not Being Utilized during mdrun

Mark Abraham mark.j.abraham at gmail.com
Mon Jul 18 00:15:55 CEST 2016


Hi,

As you can see towards the end of the log file, mdrun observes that the
GPU/CPU force time is 0.794. That means that despite the auto-tuning, mdrun
found it was best that the GPU lie idle for ~20% of the time spent
computing forces. It's also idle for all the other parts, which is about
30% of the total run time in your case. So your GPU is indeed idle about
half the time. Unfortunately, mdrun needs roughly balanced expenditure on
CPU vs GPU hardware, so there's not much to be done about this situation.
The 2016 release will likely improve the CPU side by more than the GPU
side, at least. We are working on various approaches to improve this, but
they're not ready yet. The advice at
http://onlinelibrary.wiley.com/enhanced/exportCitation/doi/10.1002/jcc.24030 is
still good.

Mark

On Sun, Jul 17, 2016 at 10:57 PM Vito Spadavecchio <spadavecchio at gmail.com>
wrote:

> Just wanted to ping this and see if anyone had any ideas :)
>
> On Fri, Jul 15, 2016 at 10:30 AM, Vito Spadavecchio <
> spadavecchio at gmail.com>
> wrote:
>
> > My apologies! I meant to include the logfile as well. You can view it
> > here:
> >
> > https://www.dropbox.com/s/8ddm226gpgs5kpr/md_0_2.log?dl=0
> >
> > Thanks!
> >
> > On Fri, Jul 15, 2016 at 2:45 AM, Justin Lemkul <jalemkul at vt.edu> wrote:
> >
> >>
> >>
> >> On 7/14/16 10:35 PM, Vito Spadavecchio wrote:
> >>
> >>> Szilard
> >>>
> >>> You are correct; I misspoke. It appears that the GPU is doing
> >>> *something*,
> >>> but I it would seem it is severely under performing (near the point
> where
> >>> its basically doing nothing).
> >>>
> >>> the entire log, not just parts, in particular we want to see the
> >>>
> >>>> header, perf table
> >>>>
> >>>>
> >>> xyz at Turing:~/Desktop/xyz_MD/APO_xyz$ gmx mdrun -deffnm md_0_2 -nb gpu
> >>>                    :-) GROMACS - gmx mdrun, VERSION 5.1.2 (-:
> >>>
> >>>                             GROMACS is written by:
> >>>      Emile Apol      Rossen Apostolov  Herman J.C. Berendsen    Par
> >>> Bjelkmar
> >>>  Aldert van Buuren   Rudi van Drunen     Anton Feenstra   Sebastian
> >>> Fritsch
> >>>   Gerrit Groenhof   Christoph Junghans   Anca Hamuraru    Vincent
> >>> Hindriksen
> >>>  Dimitrios Karkoulis    Peter Kasson        Jiri Kraus      Carsten
> >>> Kutzner
> >>>
> >>>     Per Larsson      Justin A. Lemkul   Magnus Lundborg   Pieter
> >>> Meulenhoff
> >>>    Erik Marklund      Teemu Murtola       Szilard Pall       Sander
> Pronk
> >>>    Roland Schulz     Alexey Shvetsov     Michael Shirts     Alfons
> >>> Sijbers
> >>>    Peter Tieleman    Teemu Virolainen  Christian Wennberg    Maarten
> Wolf
> >>>                            and the project leaders:
> >>>         Mark Abraham, Berk Hess, Erik Lindahl, and David van der Spoel
> >>>
> >>> Copyright (c) 1991-2000, University of Groningen, The Netherlands.
> >>> Copyright (c) 2001-2015, The GROMACS development team at
> >>> Uppsala University, Stockholm University and
> >>> the Royal Institute of Technology, Sweden.
> >>> check out http://www.gromacs.org for more information.
> >>>
> >>> GROMACS is free software; you can redistribute it and/or modify it
> >>> under the terms of the GNU Lesser General Public License
> >>> as published by the Free Software Foundation; either version 2.1
> >>> of the License, or (at your option) any later version.
> >>>
> >>> GROMACS:      gmx mdrun, VERSION 5.1.2
> >>> Executable:   /usr/local/gromacs/bin/gmx
> >>> Data prefix:  /usr/local/gromacs
> >>> Command line:
> >>>   gmx mdrun -deffnm md_0_2 -nb gpu
> >>>
> >>>
> >>> Back Off! I just backed up md_0_2.log to ./#md_0_2.log.1#
> >>>
> >>> Running on 1 node with total 4 cores, 8 logical cores, 1 compatible GPU
> >>> Hardware detected:
> >>>   CPU info:
> >>>     Vendor: GenuineIntel
> >>>     Brand:  Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
> >>>     SIMD instructions most likely to fit this hardware: AVX2_256
> >>>     SIMD instructions selected at GROMACS compile time: AVX2_256
> >>>   GPU info:
> >>>     Number of GPUs detected: 1
> >>>     #0: NVIDIA GeForce GTX 1080, compute cap.: 6.1, ECC:  no, stat:
> >>> compatible
> >>>
> >>> Reading file md_0_2.tpr, VERSION 5.1.2 (single precision)
> >>> Changing nstlist from 10 to 40, rlist from 1 to 1.099
> >>>
> >>> Using 1 MPI thread
> >>> Using 8 OpenMP threads
> >>>
> >>> 1 compatible GPU is present, with ID 0
> >>> 1 GPU auto-selected for this run.
> >>> Mapping of GPU ID to the 1 PP rank in this node: 0
> >>>
> >>>
> >>> Back Off! I just backed up md_0_2.trr to ./#md_0_2.trr.1#
> >>>
> >>> Back Off! I just backed up md_0_2.xtc to ./#md_0_2.xtc.1#
> >>>
> >>> Back Off! I just backed up md_0_2.edr to ./#md_0_2.edr.1#
> >>> starting mdrun 'Protein in water'
> >>> 500000 steps,   1000.0 ps.
> >>>
> >>>
> >>> - the output of nvidia-smi dmon executing *during* mdrun's execution
> >>>
> >>>
> >>> heres a snapshot:
> >>>
> >>> # gpu   pwr  temp    sm   mem   enc   dec  mclk  pclk
> >>>
> >>>> # Idx     W     C     %     %     %     %   MHz   MHz
> >>>>  0   117    51    56     3     0     0  4513  1847
> >>>>     0   115    52    54     3     0     0  4513  1847
> >>>>     0   117    52    47     3     0     0  4513  1847
> >>>>     0   101    52    52     3     0     0  4513  1848
> >>>>     0   119    53    54     3     0     0  4513  1847
> >>>>     0   115    53    54     3     0     0  4513  1848
> >>>>     0   106    54    54     3     0     0  4513  1848
> >>>>     0   118    54    54     3     0     0  4513  1849
> >>>>     0    94    54    52     3     0     0  4513  1848
> >>>>     0   110    55    52     3     0     0  4513  1849
> >>>>     0   116    55    53     3     0     0  4513  1854
> >>>>     0   126    55    51     3     0     0  4513  1854
> >>>>     0   119    56    51     3     0     0  4513  1855
> >>>>     0   116    56    52     3     0     0  4513  1855
> >>>>     0   116    56    53     3     0     0  4513  1855
> >>>>     0   120    57    53     3     0     0  4513  1855
> >>>>     0   113    57    54     3     0     0  4513  1855
> >>>>
> >>>>
> >>> and here's the tail of the performance in .log
> >>>
> >>> Reading file md_0_2.tpr, VERSION 5.1.2 (single precision)
> >>>
> >>>> Changing nstlist from 10 to 40, rlist from 1 to 1.099
> >>>> Using 1 MPI thread
> >>>> Using 8 OpenMP threads
> >>>> 1 compatible GPU is present, with ID 0
> >>>> 1 GPU auto-selected for this run.
> >>>> Mapping of GPU ID to the 1 PP rank in this node: 0
> >>>>
> >>>> Back Off! I just backed up md_0_2.trr to ./#md_0_2.trr.1#
> >>>> Back Off! I just backed up md_0_2.xtc to ./#md_0_2.xtc.1#
> >>>> Back Off! I just backed up md_0_2.edr to ./#md_0_2.edr.1#
> >>>> starting mdrun 'Protein in water'
> >>>> 500000 steps,   1000.0 ps.
> >>>> Writing final coordinates.
> >>>> Back Off! I just backed up md_0_2.gro to ./#md_0_2.gro.1#
> >>>>                Core t (s)   Wall t (s)        (%)
> >>>>        Time:    25285.344     3409.337      741.6
> >>>>                          56:49
> >>>>                  (ns/day)    (hour/ns)
> >>>> Performance:       25.342        0.947
> >>>> gcq#11: "She's Not Bad, She's Just Genetically Mean" (Captain
> Beefheart)
> >>>>
> >>>>
> >>>> As we can see, there is *something* running on the GPU (process id:
> >>> 30784),
> >>> but its never more than 135MiB of memory being used and only uses a
> very
> >>> small portion of the gpus power.
> >>>
> >>> Any thoughts are greatly appreciated!
> >>>
> >>>
> >> You cut out the part of the .log file that shows how mdrun actually
> spent
> >> its time.  This is perhaps the most valuable information.  This is why
> >> Szilard asked for you to share a complete .log file (upload it
> somewhere,
> >> post a URL) so we can see all relevant details.
> >>
> >> -Justin
> >>
> >>
> >>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Thu, Jul 14, 2016 at 4:54 PM, Szilárd Páll <pall.szilard at gmail.com>
> >>> wrote:
> >>>
> >>> Hi,
> >>>>
> >>>> If mdrun claims it's using the GPU and you see no errors (and results
> >>>> look reasonable), the GPU is likely being used. That nvidia-smi is not
> >>>> showing it is admittedly strange. However, this truncated log output
> >>>> does not confirm much, e.g. it does not show the performance table and
> >>>> the final performance.
> >>>>
> >>>> Could you please run again and look at (and share if approapriate):
> >>>> - the entire log, not just parts, in particular we want to see the
> >>>> header, perf table
> >>>> - the output of nvidia-smi dmon executing *during* mdrun's execution
> >>>> (this monitors the device continuously)
> >>>>
> >>>> Cheers,
> >>>>
> >>>> --
> >>>> Szilárd
> >>>>
> >>>>
> >>>> On Thu, Jul 14, 2016 at 8:28 PM, Vito Spadavecchio
> >>>> <spadavecchio at gmail.com> wrote:
> >>>>
> >>>>> Hello
> >>>>>
> >>>>> After building GROMACS with GPU support and running some simple MD, I
> >>>>> am
> >>>>> only getting ~20ns/day (which is what I would expect to get out of my
> >>>>> i7-6700k cpu only. My machine currently has a 1080 GTX in it, which
> >>>>>
> >>>> should
> >>>>
> >>>>> even on moderately large systems, be getting ~100ns/day)
> >>>>>
> >>>>>  I've tried launching a mdrun with with the following command:
> >>>>>
> >>>>>   gmx mdrun -deffnm md_0_1 -gpu_id 0 -nb gpu
> >>>>>
> >>>>>
> >>>>>
> >>>>> With the following in my md.mdp file:
> >>>>>
> >>>>> ; Neighborsearching
> >>>>>
> >>>>> cutoff-scheme   = Verlet
> >>>>>
> >>>>>
> >>>>>
> >>>>> The top of the log file for the mdrun notes:
> >>>>>
> >>>>> Running on 1 node with total 4 cores, 8 logical cores, 1 compatible
> GPU
> >>>>>
> >>>>>> Hardware detected:
> >>>>>>   CPU info:
> >>>>>>     Vendor: GenuineIntel
> >>>>>>     Brand:  Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
> >>>>>>     SIMD instructions most likely to fit this hardware: AVX2_256
> >>>>>>     SIMD instructions selected at GROMACS compile time: AVX2_256
> >>>>>>   GPU info:
> >>>>>>     Number of GPUs detected: 1
> >>>>>>     #0: NVIDIA GeForce GTX 1080, compute cap.: 6.1, ECC:  no, stat:
> >>>>>> compatible
> >>>>>> Reading file md_0_1.tpr, VERSION 5.1.2 (single precision)
> >>>>>> Changing nstlist from 10 to 40, rlist from 1 to 1.099
> >>>>>> Using 1 MPI thread
> >>>>>> Using 8 OpenMP threads
> >>>>>> 1 GPU user-selected for this run.
> >>>>>> Mapping of GPU ID to the 1 PP rank in this node: 0
> >>>>>>
> >>>>>>
> >>>>> The output of *nvidia-smi *on my machine is:
> >>>>>
> >>>>> hu Jul 14 11:25:04 2016
> >>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>
> +-----------------------------------------------------------------------------+
> >>>>
> >>>>> | NVIDIA-SMI 367.27                 Driver Version: 367.27
> >>>>>>    |
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>
> |-------------------------------+----------------------+----------------------+
> >>>>
> >>>>> | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile
> >>>>>>
> >>>>> Uncorr.
> >>>>
> >>>>> ECC |
> >>>>>> | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util
> >>>>>>
> >>>>> Compute
> >>>>
> >>>>> M. |
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>
> |===============================+======================+======================|
> >>>>
> >>>>> |   0  GeForce GTX 1080    Off  | 0000:01:00.0      On |
> >>>>>>  N/A |
> >>>>>> | 30%   41C    P8     7W / 180W |    730MiB /  8113MiB |      3%
> >>>>>>  Default |
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>
> +-------------------------------+----------------------+----------------------+
> >>>>
> >>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>
> +-----------------------------------------------------------------------------+
> >>>>
> >>>>> | Processes:
>  GPU
> >>>>>> Memory |
> >>>>>> |  GPU       PID  Type  Process name
> >>>>>>  Usage
> >>>>>>    |
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>
> |=============================================================================|
> >>>>
> >>>>> |    0      1035    G   /usr/lib/xorg/Xorg
> >>>>>> 424MiB |
> >>>>>> |    0      2132    G   compiz
> >>>>>> 103MiB |
> >>>>>> |    0      5296    G   ...s-passed-by-fd --v8-snapshot-passed-by-fd
> >>>>>>  59MiB |
> >>>>>> |    0     14079    G   ...s-passed-by-fd --v8-snapshot-passed-by-fd
> >>>>>>  31MiB |
> >>>>>> |    0     14234    G   ...ing<BrotliEncoding --disable-features=Doc
> >>>>>> 108MiB |
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>
> +-----------------------------------------------------------------------------+
> >>>>
> >>>>>
> >>>>>
> >>>>> and output of *nvcc --version* is:
> >>>>>
> >>>>> nvcc: NVIDIA (R) Cuda compiler driver
> >>>>>
> >>>>>> Copyright (c) 2005-2015 NVIDIA Corporation
> >>>>>> Built on Tue_Aug_11_14:27:32_CDT_2015
> >>>>>> Cuda compilation tools, release 7.5, V7.5.17
> >>>>>>
> >>>>>
> >>>>>
> >>>>> Does anyone have any ideas what the problem might be?
> >>>>>
> >>>>> Thanks!
> >>>>> --
> >>>>> Gromacs Users mailing list
> >>>>>
> >>>>> * Please search the archive at
> >>>>>
> >>>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> >>>> posting!
> >>>>
> >>>>>
> >>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >>>>>
> >>>>> * For (un)subscribe requests visit
> >>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users
> or
> >>>>>
> >>>> send a mail to gmx-users-request at gromacs.org.
> >>>> --
> >>>> Gromacs Users mailing list
> >>>>
> >>>> * Please search the archive at
> >>>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> >>>> posting!
> >>>>
> >>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >>>>
> >>>> * For (un)subscribe requests visit
> >>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> >>>> send a mail to gmx-users-request at gromacs.org.
> >>>>
> >>>>
> >> --
> >> ==================================================
> >>
> >> Justin A. Lemkul, Ph.D.
> >> Ruth L. Kirschstein NRSA Postdoctoral Fellow
> >>
> >> Department of Pharmaceutical Sciences
> >> School of Pharmacy
> >> Health Sciences Facility II, Room 629
> >> University of Maryland, Baltimore
> >> 20 Penn St.
> >> Baltimore, MD 21201
> >>
> >> jalemkul at outerbanks.umaryland.edu | (410) 706-7441
> >> http://mackerell.umaryland.edu/~jalemkul
> >>
> >> ==================================================
> >>
> >> --
> >> Gromacs Users mailing list
> >>
> >> * Please search the archive at
> >> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> >> posting!
> >>
> >> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >>
> >> * For (un)subscribe requests visit
> >> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> >> send a mail to gmx-users-request at gromacs.org.
> >>
> >
> >
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.


More information about the gromacs.org_gmx-users mailing list