[gmx-users] GPU Not Being Utilized during mdrun
Vito Spadavecchio
spadavecchio at gmail.com
Fri Jul 15 19:31:04 CEST 2016
My apologies! I meant to include the logfile as well. You can view it here:
https://www.dropbox.com/s/8ddm226gpgs5kpr/md_0_2.log?dl=0
Thanks!
On Fri, Jul 15, 2016 at 2:45 AM, Justin Lemkul <jalemkul at vt.edu> wrote:
>
>
> On 7/14/16 10:35 PM, Vito Spadavecchio wrote:
>
>> Szilard
>>
>> You are correct; I misspoke. It appears that the GPU is doing *something*,
>> but I it would seem it is severely under performing (near the point where
>> its basically doing nothing).
>>
>> the entire log, not just parts, in particular we want to see the
>>
>>> header, perf table
>>>
>>>
>> xyz at Turing:~/Desktop/xyz_MD/APO_xyz$ gmx mdrun -deffnm md_0_2 -nb gpu
>> :-) GROMACS - gmx mdrun, VERSION 5.1.2 (-:
>>
>> GROMACS is written by:
>> Emile Apol Rossen Apostolov Herman J.C. Berendsen Par
>> Bjelkmar
>> Aldert van Buuren Rudi van Drunen Anton Feenstra Sebastian
>> Fritsch
>> Gerrit Groenhof Christoph Junghans Anca Hamuraru Vincent
>> Hindriksen
>> Dimitrios Karkoulis Peter Kasson Jiri Kraus Carsten
>> Kutzner
>>
>> Per Larsson Justin A. Lemkul Magnus Lundborg Pieter
>> Meulenhoff
>> Erik Marklund Teemu Murtola Szilard Pall Sander Pronk
>> Roland Schulz Alexey Shvetsov Michael Shirts Alfons Sijbers
>> Peter Tieleman Teemu Virolainen Christian Wennberg Maarten Wolf
>> and the project leaders:
>> Mark Abraham, Berk Hess, Erik Lindahl, and David van der Spoel
>>
>> Copyright (c) 1991-2000, University of Groningen, The Netherlands.
>> Copyright (c) 2001-2015, The GROMACS development team at
>> Uppsala University, Stockholm University and
>> the Royal Institute of Technology, Sweden.
>> check out http://www.gromacs.org for more information.
>>
>> GROMACS is free software; you can redistribute it and/or modify it
>> under the terms of the GNU Lesser General Public License
>> as published by the Free Software Foundation; either version 2.1
>> of the License, or (at your option) any later version.
>>
>> GROMACS: gmx mdrun, VERSION 5.1.2
>> Executable: /usr/local/gromacs/bin/gmx
>> Data prefix: /usr/local/gromacs
>> Command line:
>> gmx mdrun -deffnm md_0_2 -nb gpu
>>
>>
>> Back Off! I just backed up md_0_2.log to ./#md_0_2.log.1#
>>
>> Running on 1 node with total 4 cores, 8 logical cores, 1 compatible GPU
>> Hardware detected:
>> CPU info:
>> Vendor: GenuineIntel
>> Brand: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
>> SIMD instructions most likely to fit this hardware: AVX2_256
>> SIMD instructions selected at GROMACS compile time: AVX2_256
>> GPU info:
>> Number of GPUs detected: 1
>> #0: NVIDIA GeForce GTX 1080, compute cap.: 6.1, ECC: no, stat:
>> compatible
>>
>> Reading file md_0_2.tpr, VERSION 5.1.2 (single precision)
>> Changing nstlist from 10 to 40, rlist from 1 to 1.099
>>
>> Using 1 MPI thread
>> Using 8 OpenMP threads
>>
>> 1 compatible GPU is present, with ID 0
>> 1 GPU auto-selected for this run.
>> Mapping of GPU ID to the 1 PP rank in this node: 0
>>
>>
>> Back Off! I just backed up md_0_2.trr to ./#md_0_2.trr.1#
>>
>> Back Off! I just backed up md_0_2.xtc to ./#md_0_2.xtc.1#
>>
>> Back Off! I just backed up md_0_2.edr to ./#md_0_2.edr.1#
>> starting mdrun 'Protein in water'
>> 500000 steps, 1000.0 ps.
>>
>>
>> - the output of nvidia-smi dmon executing *during* mdrun's execution
>>
>>
>> heres a snapshot:
>>
>> # gpu pwr temp sm mem enc dec mclk pclk
>>
>>> # Idx W C % % % % MHz MHz
>>> 0 117 51 56 3 0 0 4513 1847
>>> 0 115 52 54 3 0 0 4513 1847
>>> 0 117 52 47 3 0 0 4513 1847
>>> 0 101 52 52 3 0 0 4513 1848
>>> 0 119 53 54 3 0 0 4513 1847
>>> 0 115 53 54 3 0 0 4513 1848
>>> 0 106 54 54 3 0 0 4513 1848
>>> 0 118 54 54 3 0 0 4513 1849
>>> 0 94 54 52 3 0 0 4513 1848
>>> 0 110 55 52 3 0 0 4513 1849
>>> 0 116 55 53 3 0 0 4513 1854
>>> 0 126 55 51 3 0 0 4513 1854
>>> 0 119 56 51 3 0 0 4513 1855
>>> 0 116 56 52 3 0 0 4513 1855
>>> 0 116 56 53 3 0 0 4513 1855
>>> 0 120 57 53 3 0 0 4513 1855
>>> 0 113 57 54 3 0 0 4513 1855
>>>
>>>
>> and here's the tail of the performance in .log
>>
>> Reading file md_0_2.tpr, VERSION 5.1.2 (single precision)
>>
>>> Changing nstlist from 10 to 40, rlist from 1 to 1.099
>>> Using 1 MPI thread
>>> Using 8 OpenMP threads
>>> 1 compatible GPU is present, with ID 0
>>> 1 GPU auto-selected for this run.
>>> Mapping of GPU ID to the 1 PP rank in this node: 0
>>>
>>> Back Off! I just backed up md_0_2.trr to ./#md_0_2.trr.1#
>>> Back Off! I just backed up md_0_2.xtc to ./#md_0_2.xtc.1#
>>> Back Off! I just backed up md_0_2.edr to ./#md_0_2.edr.1#
>>> starting mdrun 'Protein in water'
>>> 500000 steps, 1000.0 ps.
>>> Writing final coordinates.
>>> Back Off! I just backed up md_0_2.gro to ./#md_0_2.gro.1#
>>> Core t (s) Wall t (s) (%)
>>> Time: 25285.344 3409.337 741.6
>>> 56:49
>>> (ns/day) (hour/ns)
>>> Performance: 25.342 0.947
>>> gcq#11: "She's Not Bad, She's Just Genetically Mean" (Captain Beefheart)
>>>
>>>
>>> As we can see, there is *something* running on the GPU (process id:
>> 30784),
>> but its never more than 135MiB of memory being used and only uses a very
>> small portion of the gpus power.
>>
>> Any thoughts are greatly appreciated!
>>
>>
> You cut out the part of the .log file that shows how mdrun actually spent
> its time. This is perhaps the most valuable information. This is why
> Szilard asked for you to share a complete .log file (upload it somewhere,
> post a URL) so we can see all relevant details.
>
> -Justin
>
>
>
>>
>>
>>
>>
>>
>> On Thu, Jul 14, 2016 at 4:54 PM, Szilárd Páll <pall.szilard at gmail.com>
>> wrote:
>>
>> Hi,
>>>
>>> If mdrun claims it's using the GPU and you see no errors (and results
>>> look reasonable), the GPU is likely being used. That nvidia-smi is not
>>> showing it is admittedly strange. However, this truncated log output
>>> does not confirm much, e.g. it does not show the performance table and
>>> the final performance.
>>>
>>> Could you please run again and look at (and share if approapriate):
>>> - the entire log, not just parts, in particular we want to see the
>>> header, perf table
>>> - the output of nvidia-smi dmon executing *during* mdrun's execution
>>> (this monitors the device continuously)
>>>
>>> Cheers,
>>>
>>> --
>>> Szilárd
>>>
>>>
>>> On Thu, Jul 14, 2016 at 8:28 PM, Vito Spadavecchio
>>> <spadavecchio at gmail.com> wrote:
>>>
>>>> Hello
>>>>
>>>> After building GROMACS with GPU support and running some simple MD, I am
>>>> only getting ~20ns/day (which is what I would expect to get out of my
>>>> i7-6700k cpu only. My machine currently has a 1080 GTX in it, which
>>>>
>>> should
>>>
>>>> even on moderately large systems, be getting ~100ns/day)
>>>>
>>>> I've tried launching a mdrun with with the following command:
>>>>
>>>> gmx mdrun -deffnm md_0_1 -gpu_id 0 -nb gpu
>>>>
>>>>
>>>>
>>>> With the following in my md.mdp file:
>>>>
>>>> ; Neighborsearching
>>>>
>>>> cutoff-scheme = Verlet
>>>>
>>>>
>>>>
>>>> The top of the log file for the mdrun notes:
>>>>
>>>> Running on 1 node with total 4 cores, 8 logical cores, 1 compatible GPU
>>>>
>>>>> Hardware detected:
>>>>> CPU info:
>>>>> Vendor: GenuineIntel
>>>>> Brand: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
>>>>> SIMD instructions most likely to fit this hardware: AVX2_256
>>>>> SIMD instructions selected at GROMACS compile time: AVX2_256
>>>>> GPU info:
>>>>> Number of GPUs detected: 1
>>>>> #0: NVIDIA GeForce GTX 1080, compute cap.: 6.1, ECC: no, stat:
>>>>> compatible
>>>>> Reading file md_0_1.tpr, VERSION 5.1.2 (single precision)
>>>>> Changing nstlist from 10 to 40, rlist from 1 to 1.099
>>>>> Using 1 MPI thread
>>>>> Using 8 OpenMP threads
>>>>> 1 GPU user-selected for this run.
>>>>> Mapping of GPU ID to the 1 PP rank in this node: 0
>>>>>
>>>>>
>>>> The output of *nvidia-smi *on my machine is:
>>>>
>>>> hu Jul 14 11:25:04 2016
>>>>
>>>>>
>>>>>
>>>>>
>>> +-----------------------------------------------------------------------------+
>>>
>>>> | NVIDIA-SMI 367.27 Driver Version: 367.27
>>>>> |
>>>>>
>>>>>
>>>>>
>>> |-------------------------------+----------------------+----------------------+
>>>
>>>> | GPU Name Persistence-M| Bus-Id Disp.A | Volatile
>>>>>
>>>> Uncorr.
>>>
>>>> ECC |
>>>>> | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util
>>>>>
>>>> Compute
>>>
>>>> M. |
>>>>>
>>>>>
>>>>>
>>> |===============================+======================+======================|
>>>
>>>> | 0 GeForce GTX 1080 Off | 0000:01:00.0 On |
>>>>> N/A |
>>>>> | 30% 41C P8 7W / 180W | 730MiB / 8113MiB | 3%
>>>>> Default |
>>>>>
>>>>>
>>>>>
>>> +-------------------------------+----------------------+----------------------+
>>>
>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>> +-----------------------------------------------------------------------------+
>>>
>>>> | Processes: GPU
>>>>> Memory |
>>>>> | GPU PID Type Process name
>>>>> Usage
>>>>> |
>>>>>
>>>>>
>>>>>
>>> |=============================================================================|
>>>
>>>> | 0 1035 G /usr/lib/xorg/Xorg
>>>>> 424MiB |
>>>>> | 0 2132 G compiz
>>>>> 103MiB |
>>>>> | 0 5296 G ...s-passed-by-fd --v8-snapshot-passed-by-fd
>>>>> 59MiB |
>>>>> | 0 14079 G ...s-passed-by-fd --v8-snapshot-passed-by-fd
>>>>> 31MiB |
>>>>> | 0 14234 G ...ing<BrotliEncoding --disable-features=Doc
>>>>> 108MiB |
>>>>>
>>>>>
>>>>>
>>> +-----------------------------------------------------------------------------+
>>>
>>>>
>>>>
>>>> and output of *nvcc --version* is:
>>>>
>>>> nvcc: NVIDIA (R) Cuda compiler driver
>>>>
>>>>> Copyright (c) 2005-2015 NVIDIA Corporation
>>>>> Built on Tue_Aug_11_14:27:32_CDT_2015
>>>>> Cuda compilation tools, release 7.5, V7.5.17
>>>>>
>>>>
>>>>
>>>> Does anyone have any ideas what the problem might be?
>>>>
>>>> Thanks!
>>>> --
>>>> Gromacs Users mailing list
>>>>
>>>> * Please search the archive at
>>>>
>>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
>>> posting!
>>>
>>>>
>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>>>
>>>> * For (un)subscribe requests visit
>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>>>>
>>> send a mail to gmx-users-request at gromacs.org.
>>> --
>>> Gromacs Users mailing list
>>>
>>> * Please search the archive at
>>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
>>> posting!
>>>
>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>>
>>> * For (un)subscribe requests visit
>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>>> send a mail to gmx-users-request at gromacs.org.
>>>
>>>
> --
> ==================================================
>
> Justin A. Lemkul, Ph.D.
> Ruth L. Kirschstein NRSA Postdoctoral Fellow
>
> Department of Pharmaceutical Sciences
> School of Pharmacy
> Health Sciences Facility II, Room 629
> University of Maryland, Baltimore
> 20 Penn St.
> Baltimore, MD 21201
>
> jalemkul at outerbanks.umaryland.edu | (410) 706-7441
> http://mackerell.umaryland.edu/~jalemkul
>
> ==================================================
>
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
>
More information about the gromacs.org_gmx-users
mailing list