[gmx-users] GPU Not Being Utilized during mdrun
Vito Spadavecchio
spadavecchio at gmail.com
Thu Jul 28 20:04:11 CEST 2016
Just as an update: I upgraded my cuda7.5 installation to cuda8, (keeping
the 367.27 driver), and am now getting ~150ns/day on the same system. This
is still ~1/2 of what I would expect given the benchmarks I've seen for the
1080gtx, but I will take a ~10x increase in throughput for now!
On Thu, Jul 28, 2016 at 9:10 AM, Szilárd Páll <pall.szilard at gmail.com>
wrote:
> Note that the CPU-GPU load balancing scaled the cut-off to 1.5 nm
> increasing the GPU load by about 3x (see the table on line 550 of the
> log file you posted).
> I find it somewhat suspicious that your CPU seems so slow, but given
> that you have a GTX 1080, that may just be normal.
> --
> Szilárd
>
>
> On Sun, Jul 17, 2016 at 10:57 PM, Vito Spadavecchio
> <spadavecchio at gmail.com> wrote:
> > Just wanted to ping this and see if anyone had any ideas :)
> >
> > On Fri, Jul 15, 2016 at 10:30 AM, Vito Spadavecchio <
> spadavecchio at gmail.com>
> > wrote:
> >
> >> My apologies! I meant to include the logfile as well. You can view it
> >> here:
> >>
> >> https://www.dropbox.com/s/8ddm226gpgs5kpr/md_0_2.log?dl=0
> >>
> >> Thanks!
> >>
> >> On Fri, Jul 15, 2016 at 2:45 AM, Justin Lemkul <jalemkul at vt.edu> wrote:
> >>
> >>>
> >>>
> >>> On 7/14/16 10:35 PM, Vito Spadavecchio wrote:
> >>>
> >>>> Szilard
> >>>>
> >>>> You are correct; I misspoke. It appears that the GPU is doing
> >>>> *something*,
> >>>> but I it would seem it is severely under performing (near the point
> where
> >>>> its basically doing nothing).
> >>>>
> >>>> the entire log, not just parts, in particular we want to see the
> >>>>
> >>>>> header, perf table
> >>>>>
> >>>>>
> >>>> xyz at Turing:~/Desktop/xyz_MD/APO_xyz$ gmx mdrun -deffnm md_0_2 -nb gpu
> >>>> :-) GROMACS - gmx mdrun, VERSION 5.1.2 (-:
> >>>>
> >>>> GROMACS is written by:
> >>>> Emile Apol Rossen Apostolov Herman J.C. Berendsen Par
> >>>> Bjelkmar
> >>>> Aldert van Buuren Rudi van Drunen Anton Feenstra Sebastian
> >>>> Fritsch
> >>>> Gerrit Groenhof Christoph Junghans Anca Hamuraru Vincent
> >>>> Hindriksen
> >>>> Dimitrios Karkoulis Peter Kasson Jiri Kraus Carsten
> >>>> Kutzner
> >>>>
> >>>> Per Larsson Justin A. Lemkul Magnus Lundborg Pieter
> >>>> Meulenhoff
> >>>> Erik Marklund Teemu Murtola Szilard Pall Sander
> Pronk
> >>>> Roland Schulz Alexey Shvetsov Michael Shirts Alfons
> >>>> Sijbers
> >>>> Peter Tieleman Teemu Virolainen Christian Wennberg Maarten
> Wolf
> >>>> and the project leaders:
> >>>> Mark Abraham, Berk Hess, Erik Lindahl, and David van der Spoel
> >>>>
> >>>> Copyright (c) 1991-2000, University of Groningen, The Netherlands.
> >>>> Copyright (c) 2001-2015, The GROMACS development team at
> >>>> Uppsala University, Stockholm University and
> >>>> the Royal Institute of Technology, Sweden.
> >>>> check out http://www.gromacs.org for more information.
> >>>>
> >>>> GROMACS is free software; you can redistribute it and/or modify it
> >>>> under the terms of the GNU Lesser General Public License
> >>>> as published by the Free Software Foundation; either version 2.1
> >>>> of the License, or (at your option) any later version.
> >>>>
> >>>> GROMACS: gmx mdrun, VERSION 5.1.2
> >>>> Executable: /usr/local/gromacs/bin/gmx
> >>>> Data prefix: /usr/local/gromacs
> >>>> Command line:
> >>>> gmx mdrun -deffnm md_0_2 -nb gpu
> >>>>
> >>>>
> >>>> Back Off! I just backed up md_0_2.log to ./#md_0_2.log.1#
> >>>>
> >>>> Running on 1 node with total 4 cores, 8 logical cores, 1 compatible
> GPU
> >>>> Hardware detected:
> >>>> CPU info:
> >>>> Vendor: GenuineIntel
> >>>> Brand: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
> >>>> SIMD instructions most likely to fit this hardware: AVX2_256
> >>>> SIMD instructions selected at GROMACS compile time: AVX2_256
> >>>> GPU info:
> >>>> Number of GPUs detected: 1
> >>>> #0: NVIDIA GeForce GTX 1080, compute cap.: 6.1, ECC: no, stat:
> >>>> compatible
> >>>>
> >>>> Reading file md_0_2.tpr, VERSION 5.1.2 (single precision)
> >>>> Changing nstlist from 10 to 40, rlist from 1 to 1.099
> >>>>
> >>>> Using 1 MPI thread
> >>>> Using 8 OpenMP threads
> >>>>
> >>>> 1 compatible GPU is present, with ID 0
> >>>> 1 GPU auto-selected for this run.
> >>>> Mapping of GPU ID to the 1 PP rank in this node: 0
> >>>>
> >>>>
> >>>> Back Off! I just backed up md_0_2.trr to ./#md_0_2.trr.1#
> >>>>
> >>>> Back Off! I just backed up md_0_2.xtc to ./#md_0_2.xtc.1#
> >>>>
> >>>> Back Off! I just backed up md_0_2.edr to ./#md_0_2.edr.1#
> >>>> starting mdrun 'Protein in water'
> >>>> 500000 steps, 1000.0 ps.
> >>>>
> >>>>
> >>>> - the output of nvidia-smi dmon executing *during* mdrun's execution
> >>>>
> >>>>
> >>>> heres a snapshot:
> >>>>
> >>>> # gpu pwr temp sm mem enc dec mclk pclk
> >>>>
> >>>>> # Idx W C % % % % MHz MHz
> >>>>> 0 117 51 56 3 0 0 4513 1847
> >>>>> 0 115 52 54 3 0 0 4513 1847
> >>>>> 0 117 52 47 3 0 0 4513 1847
> >>>>> 0 101 52 52 3 0 0 4513 1848
> >>>>> 0 119 53 54 3 0 0 4513 1847
> >>>>> 0 115 53 54 3 0 0 4513 1848
> >>>>> 0 106 54 54 3 0 0 4513 1848
> >>>>> 0 118 54 54 3 0 0 4513 1849
> >>>>> 0 94 54 52 3 0 0 4513 1848
> >>>>> 0 110 55 52 3 0 0 4513 1849
> >>>>> 0 116 55 53 3 0 0 4513 1854
> >>>>> 0 126 55 51 3 0 0 4513 1854
> >>>>> 0 119 56 51 3 0 0 4513 1855
> >>>>> 0 116 56 52 3 0 0 4513 1855
> >>>>> 0 116 56 53 3 0 0 4513 1855
> >>>>> 0 120 57 53 3 0 0 4513 1855
> >>>>> 0 113 57 54 3 0 0 4513 1855
> >>>>>
> >>>>>
> >>>> and here's the tail of the performance in .log
> >>>>
> >>>> Reading file md_0_2.tpr, VERSION 5.1.2 (single precision)
> >>>>
> >>>>> Changing nstlist from 10 to 40, rlist from 1 to 1.099
> >>>>> Using 1 MPI thread
> >>>>> Using 8 OpenMP threads
> >>>>> 1 compatible GPU is present, with ID 0
> >>>>> 1 GPU auto-selected for this run.
> >>>>> Mapping of GPU ID to the 1 PP rank in this node: 0
> >>>>>
> >>>>> Back Off! I just backed up md_0_2.trr to ./#md_0_2.trr.1#
> >>>>> Back Off! I just backed up md_0_2.xtc to ./#md_0_2.xtc.1#
> >>>>> Back Off! I just backed up md_0_2.edr to ./#md_0_2.edr.1#
> >>>>> starting mdrun 'Protein in water'
> >>>>> 500000 steps, 1000.0 ps.
> >>>>> Writing final coordinates.
> >>>>> Back Off! I just backed up md_0_2.gro to ./#md_0_2.gro.1#
> >>>>> Core t (s) Wall t (s) (%)
> >>>>> Time: 25285.344 3409.337 741.6
> >>>>> 56:49
> >>>>> (ns/day) (hour/ns)
> >>>>> Performance: 25.342 0.947
> >>>>> gcq#11: "She's Not Bad, She's Just Genetically Mean" (Captain
> Beefheart)
> >>>>>
> >>>>>
> >>>>> As we can see, there is *something* running on the GPU (process id:
> >>>> 30784),
> >>>> but its never more than 135MiB of memory being used and only uses a
> very
> >>>> small portion of the gpus power.
> >>>>
> >>>> Any thoughts are greatly appreciated!
> >>>>
> >>>>
> >>> You cut out the part of the .log file that shows how mdrun actually
> spent
> >>> its time. This is perhaps the most valuable information. This is why
> >>> Szilard asked for you to share a complete .log file (upload it
> somewhere,
> >>> post a URL) so we can see all relevant details.
> >>>
> >>> -Justin
> >>>
> >>>
> >>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Thu, Jul 14, 2016 at 4:54 PM, Szilárd Páll <pall.szilard at gmail.com
> >
> >>>> wrote:
> >>>>
> >>>> Hi,
> >>>>>
> >>>>> If mdrun claims it's using the GPU and you see no errors (and results
> >>>>> look reasonable), the GPU is likely being used. That nvidia-smi is
> not
> >>>>> showing it is admittedly strange. However, this truncated log output
> >>>>> does not confirm much, e.g. it does not show the performance table
> and
> >>>>> the final performance.
> >>>>>
> >>>>> Could you please run again and look at (and share if approapriate):
> >>>>> - the entire log, not just parts, in particular we want to see the
> >>>>> header, perf table
> >>>>> - the output of nvidia-smi dmon executing *during* mdrun's execution
> >>>>> (this monitors the device continuously)
> >>>>>
> >>>>> Cheers,
> >>>>>
> >>>>> --
> >>>>> Szilárd
> >>>>>
> >>>>>
> >>>>> On Thu, Jul 14, 2016 at 8:28 PM, Vito Spadavecchio
> >>>>> <spadavecchio at gmail.com> wrote:
> >>>>>
> >>>>>> Hello
> >>>>>>
> >>>>>> After building GROMACS with GPU support and running some simple MD,
> I
> >>>>>> am
> >>>>>> only getting ~20ns/day (which is what I would expect to get out of
> my
> >>>>>> i7-6700k cpu only. My machine currently has a 1080 GTX in it, which
> >>>>>>
> >>>>> should
> >>>>>
> >>>>>> even on moderately large systems, be getting ~100ns/day)
> >>>>>>
> >>>>>> I've tried launching a mdrun with with the following command:
> >>>>>>
> >>>>>> gmx mdrun -deffnm md_0_1 -gpu_id 0 -nb gpu
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> With the following in my md.mdp file:
> >>>>>>
> >>>>>> ; Neighborsearching
> >>>>>>
> >>>>>> cutoff-scheme = Verlet
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> The top of the log file for the mdrun notes:
> >>>>>>
> >>>>>> Running on 1 node with total 4 cores, 8 logical cores, 1 compatible
> GPU
> >>>>>>
> >>>>>>> Hardware detected:
> >>>>>>> CPU info:
> >>>>>>> Vendor: GenuineIntel
> >>>>>>> Brand: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
> >>>>>>> SIMD instructions most likely to fit this hardware: AVX2_256
> >>>>>>> SIMD instructions selected at GROMACS compile time: AVX2_256
> >>>>>>> GPU info:
> >>>>>>> Number of GPUs detected: 1
> >>>>>>> #0: NVIDIA GeForce GTX 1080, compute cap.: 6.1, ECC: no, stat:
> >>>>>>> compatible
> >>>>>>> Reading file md_0_1.tpr, VERSION 5.1.2 (single precision)
> >>>>>>> Changing nstlist from 10 to 40, rlist from 1 to 1.099
> >>>>>>> Using 1 MPI thread
> >>>>>>> Using 8 OpenMP threads
> >>>>>>> 1 GPU user-selected for this run.
> >>>>>>> Mapping of GPU ID to the 1 PP rank in this node: 0
> >>>>>>>
> >>>>>>>
> >>>>>> The output of *nvidia-smi *on my machine is:
> >>>>>>
> >>>>>> hu Jul 14 11:25:04 2016
> >>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>
> +-----------------------------------------------------------------------------+
> >>>>>
> >>>>>> | NVIDIA-SMI 367.27 Driver Version: 367.27
> >>>>>>> |
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>
> |-------------------------------+----------------------+----------------------+
> >>>>>
> >>>>>> | GPU Name Persistence-M| Bus-Id Disp.A | Volatile
> >>>>>>>
> >>>>>> Uncorr.
> >>>>>
> >>>>>> ECC |
> >>>>>>> | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util
> >>>>>>>
> >>>>>> Compute
> >>>>>
> >>>>>> M. |
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>
> |===============================+======================+======================|
> >>>>>
> >>>>>> | 0 GeForce GTX 1080 Off | 0000:01:00.0 On |
> >>>>>>> N/A |
> >>>>>>> | 30% 41C P8 7W / 180W | 730MiB / 8113MiB | 3%
> >>>>>>> Default |
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>
> +-------------------------------+----------------------+----------------------+
> >>>>>
> >>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>
> +-----------------------------------------------------------------------------+
> >>>>>
> >>>>>> | Processes:
> GPU
> >>>>>>> Memory |
> >>>>>>> | GPU PID Type Process name
> >>>>>>> Usage
> >>>>>>> |
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>
> |=============================================================================|
> >>>>>
> >>>>>> | 0 1035 G /usr/lib/xorg/Xorg
> >>>>>>> 424MiB |
> >>>>>>> | 0 2132 G compiz
> >>>>>>> 103MiB |
> >>>>>>> | 0 5296 G ...s-passed-by-fd
> --v8-snapshot-passed-by-fd
> >>>>>>> 59MiB |
> >>>>>>> | 0 14079 G ...s-passed-by-fd
> --v8-snapshot-passed-by-fd
> >>>>>>> 31MiB |
> >>>>>>> | 0 14234 G ...ing<BrotliEncoding
> --disable-features=Doc
> >>>>>>> 108MiB |
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>
> +-----------------------------------------------------------------------------+
> >>>>>
> >>>>>>
> >>>>>>
> >>>>>> and output of *nvcc --version* is:
> >>>>>>
> >>>>>> nvcc: NVIDIA (R) Cuda compiler driver
> >>>>>>
> >>>>>>> Copyright (c) 2005-2015 NVIDIA Corporation
> >>>>>>> Built on Tue_Aug_11_14:27:32_CDT_2015
> >>>>>>> Cuda compilation tools, release 7.5, V7.5.17
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Does anyone have any ideas what the problem might be?
> >>>>>>
> >>>>>> Thanks!
> >>>>>> --
> >>>>>> Gromacs Users mailing list
> >>>>>>
> >>>>>> * Please search the archive at
> >>>>>>
> >>>>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> >>>>> posting!
> >>>>>
> >>>>>>
> >>>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >>>>>>
> >>>>>> * For (un)subscribe requests visit
> >>>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users
> or
> >>>>>>
> >>>>> send a mail to gmx-users-request at gromacs.org.
> >>>>> --
> >>>>> Gromacs Users mailing list
> >>>>>
> >>>>> * Please search the archive at
> >>>>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> >>>>> posting!
> >>>>>
> >>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >>>>>
> >>>>> * For (un)subscribe requests visit
> >>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users
> or
> >>>>> send a mail to gmx-users-request at gromacs.org.
> >>>>>
> >>>>>
> >>> --
> >>> ==================================================
> >>>
> >>> Justin A. Lemkul, Ph.D.
> >>> Ruth L. Kirschstein NRSA Postdoctoral Fellow
> >>>
> >>> Department of Pharmaceutical Sciences
> >>> School of Pharmacy
> >>> Health Sciences Facility II, Room 629
> >>> University of Maryland, Baltimore
> >>> 20 Penn St.
> >>> Baltimore, MD 21201
> >>>
> >>> jalemkul at outerbanks.umaryland.edu | (410) 706-7441
> >>> http://mackerell.umaryland.edu/~jalemkul
> >>>
> >>> ==================================================
> >>>
> >>> --
> >>> Gromacs Users mailing list
> >>>
> >>> * Please search the archive at
> >>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> >>> posting!
> >>>
> >>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >>>
> >>> * For (un)subscribe requests visit
> >>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> >>> send a mail to gmx-users-request at gromacs.org.
> >>>
> >>
> >>
> > --
> > Gromacs Users mailing list
> >
> > * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
> >
> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >
> > * For (un)subscribe requests visit
> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
>
More information about the gromacs.org_gmx-users
mailing list