[gmx-users] GPU Not Being Utilized during mdrun
Szilárd Páll
pall.szilard at gmail.com
Thu Jul 28 18:11:03 CEST 2016
Note that the CPU-GPU load balancing scaled the cut-off to 1.5 nm
increasing the GPU load by about 3x (see the table on line 550 of the
log file you posted).
I find it somewhat suspicious that your CPU seems so slow, but given
that you have a GTX 1080, that may just be normal.
--
Szilárd
On Sun, Jul 17, 2016 at 10:57 PM, Vito Spadavecchio
<spadavecchio at gmail.com> wrote:
> Just wanted to ping this and see if anyone had any ideas :)
>
> On Fri, Jul 15, 2016 at 10:30 AM, Vito Spadavecchio <spadavecchio at gmail.com>
> wrote:
>
>> My apologies! I meant to include the logfile as well. You can view it
>> here:
>>
>> https://www.dropbox.com/s/8ddm226gpgs5kpr/md_0_2.log?dl=0
>>
>> Thanks!
>>
>> On Fri, Jul 15, 2016 at 2:45 AM, Justin Lemkul <jalemkul at vt.edu> wrote:
>>
>>>
>>>
>>> On 7/14/16 10:35 PM, Vito Spadavecchio wrote:
>>>
>>>> Szilard
>>>>
>>>> You are correct; I misspoke. It appears that the GPU is doing
>>>> *something*,
>>>> but I it would seem it is severely under performing (near the point where
>>>> its basically doing nothing).
>>>>
>>>> the entire log, not just parts, in particular we want to see the
>>>>
>>>>> header, perf table
>>>>>
>>>>>
>>>> xyz at Turing:~/Desktop/xyz_MD/APO_xyz$ gmx mdrun -deffnm md_0_2 -nb gpu
>>>> :-) GROMACS - gmx mdrun, VERSION 5.1.2 (-:
>>>>
>>>> GROMACS is written by:
>>>> Emile Apol Rossen Apostolov Herman J.C. Berendsen Par
>>>> Bjelkmar
>>>> Aldert van Buuren Rudi van Drunen Anton Feenstra Sebastian
>>>> Fritsch
>>>> Gerrit Groenhof Christoph Junghans Anca Hamuraru Vincent
>>>> Hindriksen
>>>> Dimitrios Karkoulis Peter Kasson Jiri Kraus Carsten
>>>> Kutzner
>>>>
>>>> Per Larsson Justin A. Lemkul Magnus Lundborg Pieter
>>>> Meulenhoff
>>>> Erik Marklund Teemu Murtola Szilard Pall Sander Pronk
>>>> Roland Schulz Alexey Shvetsov Michael Shirts Alfons
>>>> Sijbers
>>>> Peter Tieleman Teemu Virolainen Christian Wennberg Maarten Wolf
>>>> and the project leaders:
>>>> Mark Abraham, Berk Hess, Erik Lindahl, and David van der Spoel
>>>>
>>>> Copyright (c) 1991-2000, University of Groningen, The Netherlands.
>>>> Copyright (c) 2001-2015, The GROMACS development team at
>>>> Uppsala University, Stockholm University and
>>>> the Royal Institute of Technology, Sweden.
>>>> check out http://www.gromacs.org for more information.
>>>>
>>>> GROMACS is free software; you can redistribute it and/or modify it
>>>> under the terms of the GNU Lesser General Public License
>>>> as published by the Free Software Foundation; either version 2.1
>>>> of the License, or (at your option) any later version.
>>>>
>>>> GROMACS: gmx mdrun, VERSION 5.1.2
>>>> Executable: /usr/local/gromacs/bin/gmx
>>>> Data prefix: /usr/local/gromacs
>>>> Command line:
>>>> gmx mdrun -deffnm md_0_2 -nb gpu
>>>>
>>>>
>>>> Back Off! I just backed up md_0_2.log to ./#md_0_2.log.1#
>>>>
>>>> Running on 1 node with total 4 cores, 8 logical cores, 1 compatible GPU
>>>> Hardware detected:
>>>> CPU info:
>>>> Vendor: GenuineIntel
>>>> Brand: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
>>>> SIMD instructions most likely to fit this hardware: AVX2_256
>>>> SIMD instructions selected at GROMACS compile time: AVX2_256
>>>> GPU info:
>>>> Number of GPUs detected: 1
>>>> #0: NVIDIA GeForce GTX 1080, compute cap.: 6.1, ECC: no, stat:
>>>> compatible
>>>>
>>>> Reading file md_0_2.tpr, VERSION 5.1.2 (single precision)
>>>> Changing nstlist from 10 to 40, rlist from 1 to 1.099
>>>>
>>>> Using 1 MPI thread
>>>> Using 8 OpenMP threads
>>>>
>>>> 1 compatible GPU is present, with ID 0
>>>> 1 GPU auto-selected for this run.
>>>> Mapping of GPU ID to the 1 PP rank in this node: 0
>>>>
>>>>
>>>> Back Off! I just backed up md_0_2.trr to ./#md_0_2.trr.1#
>>>>
>>>> Back Off! I just backed up md_0_2.xtc to ./#md_0_2.xtc.1#
>>>>
>>>> Back Off! I just backed up md_0_2.edr to ./#md_0_2.edr.1#
>>>> starting mdrun 'Protein in water'
>>>> 500000 steps, 1000.0 ps.
>>>>
>>>>
>>>> - the output of nvidia-smi dmon executing *during* mdrun's execution
>>>>
>>>>
>>>> heres a snapshot:
>>>>
>>>> # gpu pwr temp sm mem enc dec mclk pclk
>>>>
>>>>> # Idx W C % % % % MHz MHz
>>>>> 0 117 51 56 3 0 0 4513 1847
>>>>> 0 115 52 54 3 0 0 4513 1847
>>>>> 0 117 52 47 3 0 0 4513 1847
>>>>> 0 101 52 52 3 0 0 4513 1848
>>>>> 0 119 53 54 3 0 0 4513 1847
>>>>> 0 115 53 54 3 0 0 4513 1848
>>>>> 0 106 54 54 3 0 0 4513 1848
>>>>> 0 118 54 54 3 0 0 4513 1849
>>>>> 0 94 54 52 3 0 0 4513 1848
>>>>> 0 110 55 52 3 0 0 4513 1849
>>>>> 0 116 55 53 3 0 0 4513 1854
>>>>> 0 126 55 51 3 0 0 4513 1854
>>>>> 0 119 56 51 3 0 0 4513 1855
>>>>> 0 116 56 52 3 0 0 4513 1855
>>>>> 0 116 56 53 3 0 0 4513 1855
>>>>> 0 120 57 53 3 0 0 4513 1855
>>>>> 0 113 57 54 3 0 0 4513 1855
>>>>>
>>>>>
>>>> and here's the tail of the performance in .log
>>>>
>>>> Reading file md_0_2.tpr, VERSION 5.1.2 (single precision)
>>>>
>>>>> Changing nstlist from 10 to 40, rlist from 1 to 1.099
>>>>> Using 1 MPI thread
>>>>> Using 8 OpenMP threads
>>>>> 1 compatible GPU is present, with ID 0
>>>>> 1 GPU auto-selected for this run.
>>>>> Mapping of GPU ID to the 1 PP rank in this node: 0
>>>>>
>>>>> Back Off! I just backed up md_0_2.trr to ./#md_0_2.trr.1#
>>>>> Back Off! I just backed up md_0_2.xtc to ./#md_0_2.xtc.1#
>>>>> Back Off! I just backed up md_0_2.edr to ./#md_0_2.edr.1#
>>>>> starting mdrun 'Protein in water'
>>>>> 500000 steps, 1000.0 ps.
>>>>> Writing final coordinates.
>>>>> Back Off! I just backed up md_0_2.gro to ./#md_0_2.gro.1#
>>>>> Core t (s) Wall t (s) (%)
>>>>> Time: 25285.344 3409.337 741.6
>>>>> 56:49
>>>>> (ns/day) (hour/ns)
>>>>> Performance: 25.342 0.947
>>>>> gcq#11: "She's Not Bad, She's Just Genetically Mean" (Captain Beefheart)
>>>>>
>>>>>
>>>>> As we can see, there is *something* running on the GPU (process id:
>>>> 30784),
>>>> but its never more than 135MiB of memory being used and only uses a very
>>>> small portion of the gpus power.
>>>>
>>>> Any thoughts are greatly appreciated!
>>>>
>>>>
>>> You cut out the part of the .log file that shows how mdrun actually spent
>>> its time. This is perhaps the most valuable information. This is why
>>> Szilard asked for you to share a complete .log file (upload it somewhere,
>>> post a URL) so we can see all relevant details.
>>>
>>> -Justin
>>>
>>>
>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Jul 14, 2016 at 4:54 PM, Szilárd Páll <pall.szilard at gmail.com>
>>>> wrote:
>>>>
>>>> Hi,
>>>>>
>>>>> If mdrun claims it's using the GPU and you see no errors (and results
>>>>> look reasonable), the GPU is likely being used. That nvidia-smi is not
>>>>> showing it is admittedly strange. However, this truncated log output
>>>>> does not confirm much, e.g. it does not show the performance table and
>>>>> the final performance.
>>>>>
>>>>> Could you please run again and look at (and share if approapriate):
>>>>> - the entire log, not just parts, in particular we want to see the
>>>>> header, perf table
>>>>> - the output of nvidia-smi dmon executing *during* mdrun's execution
>>>>> (this monitors the device continuously)
>>>>>
>>>>> Cheers,
>>>>>
>>>>> --
>>>>> Szilárd
>>>>>
>>>>>
>>>>> On Thu, Jul 14, 2016 at 8:28 PM, Vito Spadavecchio
>>>>> <spadavecchio at gmail.com> wrote:
>>>>>
>>>>>> Hello
>>>>>>
>>>>>> After building GROMACS with GPU support and running some simple MD, I
>>>>>> am
>>>>>> only getting ~20ns/day (which is what I would expect to get out of my
>>>>>> i7-6700k cpu only. My machine currently has a 1080 GTX in it, which
>>>>>>
>>>>> should
>>>>>
>>>>>> even on moderately large systems, be getting ~100ns/day)
>>>>>>
>>>>>> I've tried launching a mdrun with with the following command:
>>>>>>
>>>>>> gmx mdrun -deffnm md_0_1 -gpu_id 0 -nb gpu
>>>>>>
>>>>>>
>>>>>>
>>>>>> With the following in my md.mdp file:
>>>>>>
>>>>>> ; Neighborsearching
>>>>>>
>>>>>> cutoff-scheme = Verlet
>>>>>>
>>>>>>
>>>>>>
>>>>>> The top of the log file for the mdrun notes:
>>>>>>
>>>>>> Running on 1 node with total 4 cores, 8 logical cores, 1 compatible GPU
>>>>>>
>>>>>>> Hardware detected:
>>>>>>> CPU info:
>>>>>>> Vendor: GenuineIntel
>>>>>>> Brand: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
>>>>>>> SIMD instructions most likely to fit this hardware: AVX2_256
>>>>>>> SIMD instructions selected at GROMACS compile time: AVX2_256
>>>>>>> GPU info:
>>>>>>> Number of GPUs detected: 1
>>>>>>> #0: NVIDIA GeForce GTX 1080, compute cap.: 6.1, ECC: no, stat:
>>>>>>> compatible
>>>>>>> Reading file md_0_1.tpr, VERSION 5.1.2 (single precision)
>>>>>>> Changing nstlist from 10 to 40, rlist from 1 to 1.099
>>>>>>> Using 1 MPI thread
>>>>>>> Using 8 OpenMP threads
>>>>>>> 1 GPU user-selected for this run.
>>>>>>> Mapping of GPU ID to the 1 PP rank in this node: 0
>>>>>>>
>>>>>>>
>>>>>> The output of *nvidia-smi *on my machine is:
>>>>>>
>>>>>> hu Jul 14 11:25:04 2016
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>> +-----------------------------------------------------------------------------+
>>>>>
>>>>>> | NVIDIA-SMI 367.27 Driver Version: 367.27
>>>>>>> |
>>>>>>>
>>>>>>>
>>>>>>>
>>>>> |-------------------------------+----------------------+----------------------+
>>>>>
>>>>>> | GPU Name Persistence-M| Bus-Id Disp.A | Volatile
>>>>>>>
>>>>>> Uncorr.
>>>>>
>>>>>> ECC |
>>>>>>> | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util
>>>>>>>
>>>>>> Compute
>>>>>
>>>>>> M. |
>>>>>>>
>>>>>>>
>>>>>>>
>>>>> |===============================+======================+======================|
>>>>>
>>>>>> | 0 GeForce GTX 1080 Off | 0000:01:00.0 On |
>>>>>>> N/A |
>>>>>>> | 30% 41C P8 7W / 180W | 730MiB / 8113MiB | 3%
>>>>>>> Default |
>>>>>>>
>>>>>>>
>>>>>>>
>>>>> +-------------------------------+----------------------+----------------------+
>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>> +-----------------------------------------------------------------------------+
>>>>>
>>>>>> | Processes: GPU
>>>>>>> Memory |
>>>>>>> | GPU PID Type Process name
>>>>>>> Usage
>>>>>>> |
>>>>>>>
>>>>>>>
>>>>>>>
>>>>> |=============================================================================|
>>>>>
>>>>>> | 0 1035 G /usr/lib/xorg/Xorg
>>>>>>> 424MiB |
>>>>>>> | 0 2132 G compiz
>>>>>>> 103MiB |
>>>>>>> | 0 5296 G ...s-passed-by-fd --v8-snapshot-passed-by-fd
>>>>>>> 59MiB |
>>>>>>> | 0 14079 G ...s-passed-by-fd --v8-snapshot-passed-by-fd
>>>>>>> 31MiB |
>>>>>>> | 0 14234 G ...ing<BrotliEncoding --disable-features=Doc
>>>>>>> 108MiB |
>>>>>>>
>>>>>>>
>>>>>>>
>>>>> +-----------------------------------------------------------------------------+
>>>>>
>>>>>>
>>>>>>
>>>>>> and output of *nvcc --version* is:
>>>>>>
>>>>>> nvcc: NVIDIA (R) Cuda compiler driver
>>>>>>
>>>>>>> Copyright (c) 2005-2015 NVIDIA Corporation
>>>>>>> Built on Tue_Aug_11_14:27:32_CDT_2015
>>>>>>> Cuda compilation tools, release 7.5, V7.5.17
>>>>>>>
>>>>>>
>>>>>>
>>>>>> Does anyone have any ideas what the problem might be?
>>>>>>
>>>>>> Thanks!
>>>>>> --
>>>>>> Gromacs Users mailing list
>>>>>>
>>>>>> * Please search the archive at
>>>>>>
>>>>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
>>>>> posting!
>>>>>
>>>>>>
>>>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>>>>>
>>>>>> * For (un)subscribe requests visit
>>>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>>>>>>
>>>>> send a mail to gmx-users-request at gromacs.org.
>>>>> --
>>>>> Gromacs Users mailing list
>>>>>
>>>>> * Please search the archive at
>>>>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
>>>>> posting!
>>>>>
>>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>>>>
>>>>> * For (un)subscribe requests visit
>>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>>>>> send a mail to gmx-users-request at gromacs.org.
>>>>>
>>>>>
>>> --
>>> ==================================================
>>>
>>> Justin A. Lemkul, Ph.D.
>>> Ruth L. Kirschstein NRSA Postdoctoral Fellow
>>>
>>> Department of Pharmaceutical Sciences
>>> School of Pharmacy
>>> Health Sciences Facility II, Room 629
>>> University of Maryland, Baltimore
>>> 20 Penn St.
>>> Baltimore, MD 21201
>>>
>>> jalemkul at outerbanks.umaryland.edu | (410) 706-7441
>>> http://mackerell.umaryland.edu/~jalemkul
>>>
>>> ==================================================
>>>
>>> --
>>> Gromacs Users mailing list
>>>
>>> * Please search the archive at
>>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
>>> posting!
>>>
>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>>
>>> * For (un)subscribe requests visit
>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>>> send a mail to gmx-users-request at gromacs.org.
>>>
>>
>>
> --
> Gromacs Users mailing list
>
> * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a mail to gmx-users-request at gromacs.org.
More information about the gromacs.org_gmx-users
mailing list