[gmx-users] GPU Not Being Utilized during mdrun
Szilárd Páll
pall.szilard at gmail.com
Thu Jul 28 23:54:23 CEST 2016
Can you post a log, I have a hard time believing this was a driver causing it.
BTW, GROMACS uses the CPU heavily. If your rcoulomb gets tuned by >10%
or so, you'll see much more gain by using a faster GPU (or you can
crank up the CPU clock if you feel like adventurous and have a decent
fan).
--
Szilárd
On Thu, Jul 28, 2016 at 8:04 PM, Vito Spadavecchio
<spadavecchio at gmail.com> wrote:
> Just as an update: I upgraded my cuda7.5 installation to cuda8, (keeping
> the 367.27 driver), and am now getting ~150ns/day on the same system. This
> is still ~1/2 of what I would expect given the benchmarks I've seen for the
> 1080gtx, but I will take a ~10x increase in throughput for now!
>
> On Thu, Jul 28, 2016 at 9:10 AM, Szilárd Páll <pall.szilard at gmail.com>
> wrote:
>
>> Note that the CPU-GPU load balancing scaled the cut-off to 1.5 nm
>> increasing the GPU load by about 3x (see the table on line 550 of the
>> log file you posted).
>> I find it somewhat suspicious that your CPU seems so slow, but given
>> that you have a GTX 1080, that may just be normal.
>> --
>> Szilárd
>>
>>
>> On Sun, Jul 17, 2016 at 10:57 PM, Vito Spadavecchio
>> <spadavecchio at gmail.com> wrote:
>> > Just wanted to ping this and see if anyone had any ideas :)
>> >
>> > On Fri, Jul 15, 2016 at 10:30 AM, Vito Spadavecchio <
>> spadavecchio at gmail.com>
>> > wrote:
>> >
>> >> My apologies! I meant to include the logfile as well. You can view it
>> >> here:
>> >>
>> >> https://www.dropbox.com/s/8ddm226gpgs5kpr/md_0_2.log?dl=0
>> >>
>> >> Thanks!
>> >>
>> >> On Fri, Jul 15, 2016 at 2:45 AM, Justin Lemkul <jalemkul at vt.edu> wrote:
>> >>
>> >>>
>> >>>
>> >>> On 7/14/16 10:35 PM, Vito Spadavecchio wrote:
>> >>>
>> >>>> Szilard
>> >>>>
>> >>>> You are correct; I misspoke. It appears that the GPU is doing
>> >>>> *something*,
>> >>>> but I it would seem it is severely under performing (near the point
>> where
>> >>>> its basically doing nothing).
>> >>>>
>> >>>> the entire log, not just parts, in particular we want to see the
>> >>>>
>> >>>>> header, perf table
>> >>>>>
>> >>>>>
>> >>>> xyz at Turing:~/Desktop/xyz_MD/APO_xyz$ gmx mdrun -deffnm md_0_2 -nb gpu
>> >>>> :-) GROMACS - gmx mdrun, VERSION 5.1.2 (-:
>> >>>>
>> >>>> GROMACS is written by:
>> >>>> Emile Apol Rossen Apostolov Herman J.C. Berendsen Par
>> >>>> Bjelkmar
>> >>>> Aldert van Buuren Rudi van Drunen Anton Feenstra Sebastian
>> >>>> Fritsch
>> >>>> Gerrit Groenhof Christoph Junghans Anca Hamuraru Vincent
>> >>>> Hindriksen
>> >>>> Dimitrios Karkoulis Peter Kasson Jiri Kraus Carsten
>> >>>> Kutzner
>> >>>>
>> >>>> Per Larsson Justin A. Lemkul Magnus Lundborg Pieter
>> >>>> Meulenhoff
>> >>>> Erik Marklund Teemu Murtola Szilard Pall Sander
>> Pronk
>> >>>> Roland Schulz Alexey Shvetsov Michael Shirts Alfons
>> >>>> Sijbers
>> >>>> Peter Tieleman Teemu Virolainen Christian Wennberg Maarten
>> Wolf
>> >>>> and the project leaders:
>> >>>> Mark Abraham, Berk Hess, Erik Lindahl, and David van der Spoel
>> >>>>
>> >>>> Copyright (c) 1991-2000, University of Groningen, The Netherlands.
>> >>>> Copyright (c) 2001-2015, The GROMACS development team at
>> >>>> Uppsala University, Stockholm University and
>> >>>> the Royal Institute of Technology, Sweden.
>> >>>> check out http://www.gromacs.org for more information.
>> >>>>
>> >>>> GROMACS is free software; you can redistribute it and/or modify it
>> >>>> under the terms of the GNU Lesser General Public License
>> >>>> as published by the Free Software Foundation; either version 2.1
>> >>>> of the License, or (at your option) any later version.
>> >>>>
>> >>>> GROMACS: gmx mdrun, VERSION 5.1.2
>> >>>> Executable: /usr/local/gromacs/bin/gmx
>> >>>> Data prefix: /usr/local/gromacs
>> >>>> Command line:
>> >>>> gmx mdrun -deffnm md_0_2 -nb gpu
>> >>>>
>> >>>>
>> >>>> Back Off! I just backed up md_0_2.log to ./#md_0_2.log.1#
>> >>>>
>> >>>> Running on 1 node with total 4 cores, 8 logical cores, 1 compatible
>> GPU
>> >>>> Hardware detected:
>> >>>> CPU info:
>> >>>> Vendor: GenuineIntel
>> >>>> Brand: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
>> >>>> SIMD instructions most likely to fit this hardware: AVX2_256
>> >>>> SIMD instructions selected at GROMACS compile time: AVX2_256
>> >>>> GPU info:
>> >>>> Number of GPUs detected: 1
>> >>>> #0: NVIDIA GeForce GTX 1080, compute cap.: 6.1, ECC: no, stat:
>> >>>> compatible
>> >>>>
>> >>>> Reading file md_0_2.tpr, VERSION 5.1.2 (single precision)
>> >>>> Changing nstlist from 10 to 40, rlist from 1 to 1.099
>> >>>>
>> >>>> Using 1 MPI thread
>> >>>> Using 8 OpenMP threads
>> >>>>
>> >>>> 1 compatible GPU is present, with ID 0
>> >>>> 1 GPU auto-selected for this run.
>> >>>> Mapping of GPU ID to the 1 PP rank in this node: 0
>> >>>>
>> >>>>
>> >>>> Back Off! I just backed up md_0_2.trr to ./#md_0_2.trr.1#
>> >>>>
>> >>>> Back Off! I just backed up md_0_2.xtc to ./#md_0_2.xtc.1#
>> >>>>
>> >>>> Back Off! I just backed up md_0_2.edr to ./#md_0_2.edr.1#
>> >>>> starting mdrun 'Protein in water'
>> >>>> 500000 steps, 1000.0 ps.
>> >>>>
>> >>>>
>> >>>> - the output of nvidia-smi dmon executing *during* mdrun's execution
>> >>>>
>> >>>>
>> >>>> heres a snapshot:
>> >>>>
>> >>>> # gpu pwr temp sm mem enc dec mclk pclk
>> >>>>
>> >>>>> # Idx W C % % % % MHz MHz
>> >>>>> 0 117 51 56 3 0 0 4513 1847
>> >>>>> 0 115 52 54 3 0 0 4513 1847
>> >>>>> 0 117 52 47 3 0 0 4513 1847
>> >>>>> 0 101 52 52 3 0 0 4513 1848
>> >>>>> 0 119 53 54 3 0 0 4513 1847
>> >>>>> 0 115 53 54 3 0 0 4513 1848
>> >>>>> 0 106 54 54 3 0 0 4513 1848
>> >>>>> 0 118 54 54 3 0 0 4513 1849
>> >>>>> 0 94 54 52 3 0 0 4513 1848
>> >>>>> 0 110 55 52 3 0 0 4513 1849
>> >>>>> 0 116 55 53 3 0 0 4513 1854
>> >>>>> 0 126 55 51 3 0 0 4513 1854
>> >>>>> 0 119 56 51 3 0 0 4513 1855
>> >>>>> 0 116 56 52 3 0 0 4513 1855
>> >>>>> 0 116 56 53 3 0 0 4513 1855
>> >>>>> 0 120 57 53 3 0 0 4513 1855
>> >>>>> 0 113 57 54 3 0 0 4513 1855
>> >>>>>
>> >>>>>
>> >>>> and here's the tail of the performance in .log
>> >>>>
>> >>>> Reading file md_0_2.tpr, VERSION 5.1.2 (single precision)
>> >>>>
>> >>>>> Changing nstlist from 10 to 40, rlist from 1 to 1.099
>> >>>>> Using 1 MPI thread
>> >>>>> Using 8 OpenMP threads
>> >>>>> 1 compatible GPU is present, with ID 0
>> >>>>> 1 GPU auto-selected for this run.
>> >>>>> Mapping of GPU ID to the 1 PP rank in this node: 0
>> >>>>>
>> >>>>> Back Off! I just backed up md_0_2.trr to ./#md_0_2.trr.1#
>> >>>>> Back Off! I just backed up md_0_2.xtc to ./#md_0_2.xtc.1#
>> >>>>> Back Off! I just backed up md_0_2.edr to ./#md_0_2.edr.1#
>> >>>>> starting mdrun 'Protein in water'
>> >>>>> 500000 steps, 1000.0 ps.
>> >>>>> Writing final coordinates.
>> >>>>> Back Off! I just backed up md_0_2.gro to ./#md_0_2.gro.1#
>> >>>>> Core t (s) Wall t (s) (%)
>> >>>>> Time: 25285.344 3409.337 741.6
>> >>>>> 56:49
>> >>>>> (ns/day) (hour/ns)
>> >>>>> Performance: 25.342 0.947
>> >>>>> gcq#11: "She's Not Bad, She's Just Genetically Mean" (Captain
>> Beefheart)
>> >>>>>
>> >>>>>
>> >>>>> As we can see, there is *something* running on the GPU (process id:
>> >>>> 30784),
>> >>>> but its never more than 135MiB of memory being used and only uses a
>> very
>> >>>> small portion of the gpus power.
>> >>>>
>> >>>> Any thoughts are greatly appreciated!
>> >>>>
>> >>>>
>> >>> You cut out the part of the .log file that shows how mdrun actually
>> spent
>> >>> its time. This is perhaps the most valuable information. This is why
>> >>> Szilard asked for you to share a complete .log file (upload it
>> somewhere,
>> >>> post a URL) so we can see all relevant details.
>> >>>
>> >>> -Justin
>> >>>
>> >>>
>> >>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Thu, Jul 14, 2016 at 4:54 PM, Szilárd Páll <pall.szilard at gmail.com
>> >
>> >>>> wrote:
>> >>>>
>> >>>> Hi,
>> >>>>>
>> >>>>> If mdrun claims it's using the GPU and you see no errors (and results
>> >>>>> look reasonable), the GPU is likely being used. That nvidia-smi is
>> not
>> >>>>> showing it is admittedly strange. However, this truncated log output
>> >>>>> does not confirm much, e.g. it does not show the performance table
>> and
>> >>>>> the final performance.
>> >>>>>
>> >>>>> Could you please run again and look at (and share if approapriate):
>> >>>>> - the entire log, not just parts, in particular we want to see the
>> >>>>> header, perf table
>> >>>>> - the output of nvidia-smi dmon executing *during* mdrun's execution
>> >>>>> (this monitors the device continuously)
>> >>>>>
>> >>>>> Cheers,
>> >>>>>
>> >>>>> --
>> >>>>> Szilárd
>> >>>>>
>> >>>>>
>> >>>>> On Thu, Jul 14, 2016 at 8:28 PM, Vito Spadavecchio
>> >>>>> <spadavecchio at gmail.com> wrote:
>> >>>>>
>> >>>>>> Hello
>> >>>>>>
>> >>>>>> After building GROMACS with GPU support and running some simple MD,
>> I
>> >>>>>> am
>> >>>>>> only getting ~20ns/day (which is what I would expect to get out of
>> my
>> >>>>>> i7-6700k cpu only. My machine currently has a 1080 GTX in it, which
>> >>>>>>
>> >>>>> should
>> >>>>>
>> >>>>>> even on moderately large systems, be getting ~100ns/day)
>> >>>>>>
>> >>>>>> I've tried launching a mdrun with with the following command:
>> >>>>>>
>> >>>>>> gmx mdrun -deffnm md_0_1 -gpu_id 0 -nb gpu
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> With the following in my md.mdp file:
>> >>>>>>
>> >>>>>> ; Neighborsearching
>> >>>>>>
>> >>>>>> cutoff-scheme = Verlet
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> The top of the log file for the mdrun notes:
>> >>>>>>
>> >>>>>> Running on 1 node with total 4 cores, 8 logical cores, 1 compatible
>> GPU
>> >>>>>>
>> >>>>>>> Hardware detected:
>> >>>>>>> CPU info:
>> >>>>>>> Vendor: GenuineIntel
>> >>>>>>> Brand: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
>> >>>>>>> SIMD instructions most likely to fit this hardware: AVX2_256
>> >>>>>>> SIMD instructions selected at GROMACS compile time: AVX2_256
>> >>>>>>> GPU info:
>> >>>>>>> Number of GPUs detected: 1
>> >>>>>>> #0: NVIDIA GeForce GTX 1080, compute cap.: 6.1, ECC: no, stat:
>> >>>>>>> compatible
>> >>>>>>> Reading file md_0_1.tpr, VERSION 5.1.2 (single precision)
>> >>>>>>> Changing nstlist from 10 to 40, rlist from 1 to 1.099
>> >>>>>>> Using 1 MPI thread
>> >>>>>>> Using 8 OpenMP threads
>> >>>>>>> 1 GPU user-selected for this run.
>> >>>>>>> Mapping of GPU ID to the 1 PP rank in this node: 0
>> >>>>>>>
>> >>>>>>>
>> >>>>>> The output of *nvidia-smi *on my machine is:
>> >>>>>>
>> >>>>>> hu Jul 14 11:25:04 2016
>> >>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>
>> +-----------------------------------------------------------------------------+
>> >>>>>
>> >>>>>> | NVIDIA-SMI 367.27 Driver Version: 367.27
>> >>>>>>> |
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>
>> |-------------------------------+----------------------+----------------------+
>> >>>>>
>> >>>>>> | GPU Name Persistence-M| Bus-Id Disp.A | Volatile
>> >>>>>>>
>> >>>>>> Uncorr.
>> >>>>>
>> >>>>>> ECC |
>> >>>>>>> | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util
>> >>>>>>>
>> >>>>>> Compute
>> >>>>>
>> >>>>>> M. |
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>
>> |===============================+======================+======================|
>> >>>>>
>> >>>>>> | 0 GeForce GTX 1080 Off | 0000:01:00.0 On |
>> >>>>>>> N/A |
>> >>>>>>> | 30% 41C P8 7W / 180W | 730MiB / 8113MiB | 3%
>> >>>>>>> Default |
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>
>> +-------------------------------+----------------------+----------------------+
>> >>>>>
>> >>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>
>> +-----------------------------------------------------------------------------+
>> >>>>>
>> >>>>>> | Processes:
>> GPU
>> >>>>>>> Memory |
>> >>>>>>> | GPU PID Type Process name
>> >>>>>>> Usage
>> >>>>>>> |
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>
>> |=============================================================================|
>> >>>>>
>> >>>>>> | 0 1035 G /usr/lib/xorg/Xorg
>> >>>>>>> 424MiB |
>> >>>>>>> | 0 2132 G compiz
>> >>>>>>> 103MiB |
>> >>>>>>> | 0 5296 G ...s-passed-by-fd
>> --v8-snapshot-passed-by-fd
>> >>>>>>> 59MiB |
>> >>>>>>> | 0 14079 G ...s-passed-by-fd
>> --v8-snapshot-passed-by-fd
>> >>>>>>> 31MiB |
>> >>>>>>> | 0 14234 G ...ing<BrotliEncoding
>> --disable-features=Doc
>> >>>>>>> 108MiB |
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>
>> +-----------------------------------------------------------------------------+
>> >>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> and output of *nvcc --version* is:
>> >>>>>>
>> >>>>>> nvcc: NVIDIA (R) Cuda compiler driver
>> >>>>>>
>> >>>>>>> Copyright (c) 2005-2015 NVIDIA Corporation
>> >>>>>>> Built on Tue_Aug_11_14:27:32_CDT_2015
>> >>>>>>> Cuda compilation tools, release 7.5, V7.5.17
>> >>>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> Does anyone have any ideas what the problem might be?
>> >>>>>>
>> >>>>>> Thanks!
>> >>>>>> --
>> >>>>>> Gromacs Users mailing list
>> >>>>>>
>> >>>>>> * Please search the archive at
>> >>>>>>
>> >>>>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
>> >>>>> posting!
>> >>>>>
>> >>>>>>
>> >>>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>> >>>>>>
>> >>>>>> * For (un)subscribe requests visit
>> >>>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users
>> or
>> >>>>>>
>> >>>>> send a mail to gmx-users-request at gromacs.org.
>> >>>>> --
>> >>>>> Gromacs Users mailing list
>> >>>>>
>> >>>>> * Please search the archive at
>> >>>>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
>> >>>>> posting!
>> >>>>>
>> >>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>> >>>>>
>> >>>>> * For (un)subscribe requests visit
>> >>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users
>> or
>> >>>>> send a mail to gmx-users-request at gromacs.org.
>> >>>>>
>> >>>>>
>> >>> --
>> >>> ==================================================
>> >>>
>> >>> Justin A. Lemkul, Ph.D.
>> >>> Ruth L. Kirschstein NRSA Postdoctoral Fellow
>> >>>
>> >>> Department of Pharmaceutical Sciences
>> >>> School of Pharmacy
>> >>> Health Sciences Facility II, Room 629
>> >>> University of Maryland, Baltimore
>> >>> 20 Penn St.
>> >>> Baltimore, MD 21201
>> >>>
>> >>> jalemkul at outerbanks.umaryland.edu | (410) 706-7441
>> >>> http://mackerell.umaryland.edu/~jalemkul
>> >>>
>> >>> ==================================================
>> >>>
>> >>> --
>> >>> Gromacs Users mailing list
>> >>>
>> >>> * Please search the archive at
>> >>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
>> >>> posting!
>> >>>
>> >>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>> >>>
>> >>> * For (un)subscribe requests visit
>> >>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>> >>> send a mail to gmx-users-request at gromacs.org.
>> >>>
>> >>
>> >>
>> > --
>> > Gromacs Users mailing list
>> >
>> > * Please search the archive at
>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
>> posting!
>> >
>> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>> >
>> > * For (un)subscribe requests visit
>> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>> send a mail to gmx-users-request at gromacs.org.
>> --
>> Gromacs Users mailing list
>>
>> * Please search the archive at
>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
>> posting!
>>
>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>
>> * For (un)subscribe requests visit
>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>> send a mail to gmx-users-request at gromacs.org.
>>
> --
> Gromacs Users mailing list
>
> * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a mail to gmx-users-request at gromacs.org.
More information about the gromacs.org_gmx-users
mailing list