[gmx-users] GPU Not Being Utilized during mdrun
Vito Spadavecchio
spadavecchio at gmail.com
Sun Jul 17 22:57:18 CEST 2016
Just wanted to ping this and see if anyone had any ideas :)
On Fri, Jul 15, 2016 at 10:30 AM, Vito Spadavecchio <spadavecchio at gmail.com>
wrote:
> My apologies! I meant to include the logfile as well. You can view it
> here:
>
> https://www.dropbox.com/s/8ddm226gpgs5kpr/md_0_2.log?dl=0
>
> Thanks!
>
> On Fri, Jul 15, 2016 at 2:45 AM, Justin Lemkul <jalemkul at vt.edu> wrote:
>
>>
>>
>> On 7/14/16 10:35 PM, Vito Spadavecchio wrote:
>>
>>> Szilard
>>>
>>> You are correct; I misspoke. It appears that the GPU is doing
>>> *something*,
>>> but I it would seem it is severely under performing (near the point where
>>> its basically doing nothing).
>>>
>>> the entire log, not just parts, in particular we want to see the
>>>
>>>> header, perf table
>>>>
>>>>
>>> xyz at Turing:~/Desktop/xyz_MD/APO_xyz$ gmx mdrun -deffnm md_0_2 -nb gpu
>>> :-) GROMACS - gmx mdrun, VERSION 5.1.2 (-:
>>>
>>> GROMACS is written by:
>>> Emile Apol Rossen Apostolov Herman J.C. Berendsen Par
>>> Bjelkmar
>>> Aldert van Buuren Rudi van Drunen Anton Feenstra Sebastian
>>> Fritsch
>>> Gerrit Groenhof Christoph Junghans Anca Hamuraru Vincent
>>> Hindriksen
>>> Dimitrios Karkoulis Peter Kasson Jiri Kraus Carsten
>>> Kutzner
>>>
>>> Per Larsson Justin A. Lemkul Magnus Lundborg Pieter
>>> Meulenhoff
>>> Erik Marklund Teemu Murtola Szilard Pall Sander Pronk
>>> Roland Schulz Alexey Shvetsov Michael Shirts Alfons
>>> Sijbers
>>> Peter Tieleman Teemu Virolainen Christian Wennberg Maarten Wolf
>>> and the project leaders:
>>> Mark Abraham, Berk Hess, Erik Lindahl, and David van der Spoel
>>>
>>> Copyright (c) 1991-2000, University of Groningen, The Netherlands.
>>> Copyright (c) 2001-2015, The GROMACS development team at
>>> Uppsala University, Stockholm University and
>>> the Royal Institute of Technology, Sweden.
>>> check out http://www.gromacs.org for more information.
>>>
>>> GROMACS is free software; you can redistribute it and/or modify it
>>> under the terms of the GNU Lesser General Public License
>>> as published by the Free Software Foundation; either version 2.1
>>> of the License, or (at your option) any later version.
>>>
>>> GROMACS: gmx mdrun, VERSION 5.1.2
>>> Executable: /usr/local/gromacs/bin/gmx
>>> Data prefix: /usr/local/gromacs
>>> Command line:
>>> gmx mdrun -deffnm md_0_2 -nb gpu
>>>
>>>
>>> Back Off! I just backed up md_0_2.log to ./#md_0_2.log.1#
>>>
>>> Running on 1 node with total 4 cores, 8 logical cores, 1 compatible GPU
>>> Hardware detected:
>>> CPU info:
>>> Vendor: GenuineIntel
>>> Brand: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
>>> SIMD instructions most likely to fit this hardware: AVX2_256
>>> SIMD instructions selected at GROMACS compile time: AVX2_256
>>> GPU info:
>>> Number of GPUs detected: 1
>>> #0: NVIDIA GeForce GTX 1080, compute cap.: 6.1, ECC: no, stat:
>>> compatible
>>>
>>> Reading file md_0_2.tpr, VERSION 5.1.2 (single precision)
>>> Changing nstlist from 10 to 40, rlist from 1 to 1.099
>>>
>>> Using 1 MPI thread
>>> Using 8 OpenMP threads
>>>
>>> 1 compatible GPU is present, with ID 0
>>> 1 GPU auto-selected for this run.
>>> Mapping of GPU ID to the 1 PP rank in this node: 0
>>>
>>>
>>> Back Off! I just backed up md_0_2.trr to ./#md_0_2.trr.1#
>>>
>>> Back Off! I just backed up md_0_2.xtc to ./#md_0_2.xtc.1#
>>>
>>> Back Off! I just backed up md_0_2.edr to ./#md_0_2.edr.1#
>>> starting mdrun 'Protein in water'
>>> 500000 steps, 1000.0 ps.
>>>
>>>
>>> - the output of nvidia-smi dmon executing *during* mdrun's execution
>>>
>>>
>>> heres a snapshot:
>>>
>>> # gpu pwr temp sm mem enc dec mclk pclk
>>>
>>>> # Idx W C % % % % MHz MHz
>>>> 0 117 51 56 3 0 0 4513 1847
>>>> 0 115 52 54 3 0 0 4513 1847
>>>> 0 117 52 47 3 0 0 4513 1847
>>>> 0 101 52 52 3 0 0 4513 1848
>>>> 0 119 53 54 3 0 0 4513 1847
>>>> 0 115 53 54 3 0 0 4513 1848
>>>> 0 106 54 54 3 0 0 4513 1848
>>>> 0 118 54 54 3 0 0 4513 1849
>>>> 0 94 54 52 3 0 0 4513 1848
>>>> 0 110 55 52 3 0 0 4513 1849
>>>> 0 116 55 53 3 0 0 4513 1854
>>>> 0 126 55 51 3 0 0 4513 1854
>>>> 0 119 56 51 3 0 0 4513 1855
>>>> 0 116 56 52 3 0 0 4513 1855
>>>> 0 116 56 53 3 0 0 4513 1855
>>>> 0 120 57 53 3 0 0 4513 1855
>>>> 0 113 57 54 3 0 0 4513 1855
>>>>
>>>>
>>> and here's the tail of the performance in .log
>>>
>>> Reading file md_0_2.tpr, VERSION 5.1.2 (single precision)
>>>
>>>> Changing nstlist from 10 to 40, rlist from 1 to 1.099
>>>> Using 1 MPI thread
>>>> Using 8 OpenMP threads
>>>> 1 compatible GPU is present, with ID 0
>>>> 1 GPU auto-selected for this run.
>>>> Mapping of GPU ID to the 1 PP rank in this node: 0
>>>>
>>>> Back Off! I just backed up md_0_2.trr to ./#md_0_2.trr.1#
>>>> Back Off! I just backed up md_0_2.xtc to ./#md_0_2.xtc.1#
>>>> Back Off! I just backed up md_0_2.edr to ./#md_0_2.edr.1#
>>>> starting mdrun 'Protein in water'
>>>> 500000 steps, 1000.0 ps.
>>>> Writing final coordinates.
>>>> Back Off! I just backed up md_0_2.gro to ./#md_0_2.gro.1#
>>>> Core t (s) Wall t (s) (%)
>>>> Time: 25285.344 3409.337 741.6
>>>> 56:49
>>>> (ns/day) (hour/ns)
>>>> Performance: 25.342 0.947
>>>> gcq#11: "She's Not Bad, She's Just Genetically Mean" (Captain Beefheart)
>>>>
>>>>
>>>> As we can see, there is *something* running on the GPU (process id:
>>> 30784),
>>> but its never more than 135MiB of memory being used and only uses a very
>>> small portion of the gpus power.
>>>
>>> Any thoughts are greatly appreciated!
>>>
>>>
>> You cut out the part of the .log file that shows how mdrun actually spent
>> its time. This is perhaps the most valuable information. This is why
>> Szilard asked for you to share a complete .log file (upload it somewhere,
>> post a URL) so we can see all relevant details.
>>
>> -Justin
>>
>>
>>
>>>
>>>
>>>
>>>
>>>
>>> On Thu, Jul 14, 2016 at 4:54 PM, Szilárd Páll <pall.szilard at gmail.com>
>>> wrote:
>>>
>>> Hi,
>>>>
>>>> If mdrun claims it's using the GPU and you see no errors (and results
>>>> look reasonable), the GPU is likely being used. That nvidia-smi is not
>>>> showing it is admittedly strange. However, this truncated log output
>>>> does not confirm much, e.g. it does not show the performance table and
>>>> the final performance.
>>>>
>>>> Could you please run again and look at (and share if approapriate):
>>>> - the entire log, not just parts, in particular we want to see the
>>>> header, perf table
>>>> - the output of nvidia-smi dmon executing *during* mdrun's execution
>>>> (this monitors the device continuously)
>>>>
>>>> Cheers,
>>>>
>>>> --
>>>> Szilárd
>>>>
>>>>
>>>> On Thu, Jul 14, 2016 at 8:28 PM, Vito Spadavecchio
>>>> <spadavecchio at gmail.com> wrote:
>>>>
>>>>> Hello
>>>>>
>>>>> After building GROMACS with GPU support and running some simple MD, I
>>>>> am
>>>>> only getting ~20ns/day (which is what I would expect to get out of my
>>>>> i7-6700k cpu only. My machine currently has a 1080 GTX in it, which
>>>>>
>>>> should
>>>>
>>>>> even on moderately large systems, be getting ~100ns/day)
>>>>>
>>>>> I've tried launching a mdrun with with the following command:
>>>>>
>>>>> gmx mdrun -deffnm md_0_1 -gpu_id 0 -nb gpu
>>>>>
>>>>>
>>>>>
>>>>> With the following in my md.mdp file:
>>>>>
>>>>> ; Neighborsearching
>>>>>
>>>>> cutoff-scheme = Verlet
>>>>>
>>>>>
>>>>>
>>>>> The top of the log file for the mdrun notes:
>>>>>
>>>>> Running on 1 node with total 4 cores, 8 logical cores, 1 compatible GPU
>>>>>
>>>>>> Hardware detected:
>>>>>> CPU info:
>>>>>> Vendor: GenuineIntel
>>>>>> Brand: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
>>>>>> SIMD instructions most likely to fit this hardware: AVX2_256
>>>>>> SIMD instructions selected at GROMACS compile time: AVX2_256
>>>>>> GPU info:
>>>>>> Number of GPUs detected: 1
>>>>>> #0: NVIDIA GeForce GTX 1080, compute cap.: 6.1, ECC: no, stat:
>>>>>> compatible
>>>>>> Reading file md_0_1.tpr, VERSION 5.1.2 (single precision)
>>>>>> Changing nstlist from 10 to 40, rlist from 1 to 1.099
>>>>>> Using 1 MPI thread
>>>>>> Using 8 OpenMP threads
>>>>>> 1 GPU user-selected for this run.
>>>>>> Mapping of GPU ID to the 1 PP rank in this node: 0
>>>>>>
>>>>>>
>>>>> The output of *nvidia-smi *on my machine is:
>>>>>
>>>>> hu Jul 14 11:25:04 2016
>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>> +-----------------------------------------------------------------------------+
>>>>
>>>>> | NVIDIA-SMI 367.27 Driver Version: 367.27
>>>>>> |
>>>>>>
>>>>>>
>>>>>>
>>>> |-------------------------------+----------------------+----------------------+
>>>>
>>>>> | GPU Name Persistence-M| Bus-Id Disp.A | Volatile
>>>>>>
>>>>> Uncorr.
>>>>
>>>>> ECC |
>>>>>> | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util
>>>>>>
>>>>> Compute
>>>>
>>>>> M. |
>>>>>>
>>>>>>
>>>>>>
>>>> |===============================+======================+======================|
>>>>
>>>>> | 0 GeForce GTX 1080 Off | 0000:01:00.0 On |
>>>>>> N/A |
>>>>>> | 30% 41C P8 7W / 180W | 730MiB / 8113MiB | 3%
>>>>>> Default |
>>>>>>
>>>>>>
>>>>>>
>>>> +-------------------------------+----------------------+----------------------+
>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>> +-----------------------------------------------------------------------------+
>>>>
>>>>> | Processes: GPU
>>>>>> Memory |
>>>>>> | GPU PID Type Process name
>>>>>> Usage
>>>>>> |
>>>>>>
>>>>>>
>>>>>>
>>>> |=============================================================================|
>>>>
>>>>> | 0 1035 G /usr/lib/xorg/Xorg
>>>>>> 424MiB |
>>>>>> | 0 2132 G compiz
>>>>>> 103MiB |
>>>>>> | 0 5296 G ...s-passed-by-fd --v8-snapshot-passed-by-fd
>>>>>> 59MiB |
>>>>>> | 0 14079 G ...s-passed-by-fd --v8-snapshot-passed-by-fd
>>>>>> 31MiB |
>>>>>> | 0 14234 G ...ing<BrotliEncoding --disable-features=Doc
>>>>>> 108MiB |
>>>>>>
>>>>>>
>>>>>>
>>>> +-----------------------------------------------------------------------------+
>>>>
>>>>>
>>>>>
>>>>> and output of *nvcc --version* is:
>>>>>
>>>>> nvcc: NVIDIA (R) Cuda compiler driver
>>>>>
>>>>>> Copyright (c) 2005-2015 NVIDIA Corporation
>>>>>> Built on Tue_Aug_11_14:27:32_CDT_2015
>>>>>> Cuda compilation tools, release 7.5, V7.5.17
>>>>>>
>>>>>
>>>>>
>>>>> Does anyone have any ideas what the problem might be?
>>>>>
>>>>> Thanks!
>>>>> --
>>>>> Gromacs Users mailing list
>>>>>
>>>>> * Please search the archive at
>>>>>
>>>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
>>>> posting!
>>>>
>>>>>
>>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>>>>
>>>>> * For (un)subscribe requests visit
>>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>>>>>
>>>> send a mail to gmx-users-request at gromacs.org.
>>>> --
>>>> Gromacs Users mailing list
>>>>
>>>> * Please search the archive at
>>>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
>>>> posting!
>>>>
>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>>>
>>>> * For (un)subscribe requests visit
>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>>>> send a mail to gmx-users-request at gromacs.org.
>>>>
>>>>
>> --
>> ==================================================
>>
>> Justin A. Lemkul, Ph.D.
>> Ruth L. Kirschstein NRSA Postdoctoral Fellow
>>
>> Department of Pharmaceutical Sciences
>> School of Pharmacy
>> Health Sciences Facility II, Room 629
>> University of Maryland, Baltimore
>> 20 Penn St.
>> Baltimore, MD 21201
>>
>> jalemkul at outerbanks.umaryland.edu | (410) 706-7441
>> http://mackerell.umaryland.edu/~jalemkul
>>
>> ==================================================
>>
>> --
>> Gromacs Users mailing list
>>
>> * Please search the archive at
>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
>> posting!
>>
>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>
>> * For (un)subscribe requests visit
>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>> send a mail to gmx-users-request at gromacs.org.
>>
>
>
More information about the gromacs.org_gmx-users
mailing list