[gmx-users] GPU Not Being Utilized during mdrun

Vito Spadavecchio spadavecchio at gmail.com
Sun Jul 17 22:57:18 CEST 2016


Just wanted to ping this and see if anyone had any ideas :)

On Fri, Jul 15, 2016 at 10:30 AM, Vito Spadavecchio <spadavecchio at gmail.com>
wrote:

> My apologies! I meant to include the logfile as well. You can view it
> here:
>
> https://www.dropbox.com/s/8ddm226gpgs5kpr/md_0_2.log?dl=0
>
> Thanks!
>
> On Fri, Jul 15, 2016 at 2:45 AM, Justin Lemkul <jalemkul at vt.edu> wrote:
>
>>
>>
>> On 7/14/16 10:35 PM, Vito Spadavecchio wrote:
>>
>>> Szilard
>>>
>>> You are correct; I misspoke. It appears that the GPU is doing
>>> *something*,
>>> but I it would seem it is severely under performing (near the point where
>>> its basically doing nothing).
>>>
>>> the entire log, not just parts, in particular we want to see the
>>>
>>>> header, perf table
>>>>
>>>>
>>> xyz at Turing:~/Desktop/xyz_MD/APO_xyz$ gmx mdrun -deffnm md_0_2 -nb gpu
>>>                    :-) GROMACS - gmx mdrun, VERSION 5.1.2 (-:
>>>
>>>                             GROMACS is written by:
>>>      Emile Apol      Rossen Apostolov  Herman J.C. Berendsen    Par
>>> Bjelkmar
>>>  Aldert van Buuren   Rudi van Drunen     Anton Feenstra   Sebastian
>>> Fritsch
>>>   Gerrit Groenhof   Christoph Junghans   Anca Hamuraru    Vincent
>>> Hindriksen
>>>  Dimitrios Karkoulis    Peter Kasson        Jiri Kraus      Carsten
>>> Kutzner
>>>
>>>     Per Larsson      Justin A. Lemkul   Magnus Lundborg   Pieter
>>> Meulenhoff
>>>    Erik Marklund      Teemu Murtola       Szilard Pall       Sander Pronk
>>>    Roland Schulz     Alexey Shvetsov     Michael Shirts     Alfons
>>> Sijbers
>>>    Peter Tieleman    Teemu Virolainen  Christian Wennberg    Maarten Wolf
>>>                            and the project leaders:
>>>         Mark Abraham, Berk Hess, Erik Lindahl, and David van der Spoel
>>>
>>> Copyright (c) 1991-2000, University of Groningen, The Netherlands.
>>> Copyright (c) 2001-2015, The GROMACS development team at
>>> Uppsala University, Stockholm University and
>>> the Royal Institute of Technology, Sweden.
>>> check out http://www.gromacs.org for more information.
>>>
>>> GROMACS is free software; you can redistribute it and/or modify it
>>> under the terms of the GNU Lesser General Public License
>>> as published by the Free Software Foundation; either version 2.1
>>> of the License, or (at your option) any later version.
>>>
>>> GROMACS:      gmx mdrun, VERSION 5.1.2
>>> Executable:   /usr/local/gromacs/bin/gmx
>>> Data prefix:  /usr/local/gromacs
>>> Command line:
>>>   gmx mdrun -deffnm md_0_2 -nb gpu
>>>
>>>
>>> Back Off! I just backed up md_0_2.log to ./#md_0_2.log.1#
>>>
>>> Running on 1 node with total 4 cores, 8 logical cores, 1 compatible GPU
>>> Hardware detected:
>>>   CPU info:
>>>     Vendor: GenuineIntel
>>>     Brand:  Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
>>>     SIMD instructions most likely to fit this hardware: AVX2_256
>>>     SIMD instructions selected at GROMACS compile time: AVX2_256
>>>   GPU info:
>>>     Number of GPUs detected: 1
>>>     #0: NVIDIA GeForce GTX 1080, compute cap.: 6.1, ECC:  no, stat:
>>> compatible
>>>
>>> Reading file md_0_2.tpr, VERSION 5.1.2 (single precision)
>>> Changing nstlist from 10 to 40, rlist from 1 to 1.099
>>>
>>> Using 1 MPI thread
>>> Using 8 OpenMP threads
>>>
>>> 1 compatible GPU is present, with ID 0
>>> 1 GPU auto-selected for this run.
>>> Mapping of GPU ID to the 1 PP rank in this node: 0
>>>
>>>
>>> Back Off! I just backed up md_0_2.trr to ./#md_0_2.trr.1#
>>>
>>> Back Off! I just backed up md_0_2.xtc to ./#md_0_2.xtc.1#
>>>
>>> Back Off! I just backed up md_0_2.edr to ./#md_0_2.edr.1#
>>> starting mdrun 'Protein in water'
>>> 500000 steps,   1000.0 ps.
>>>
>>>
>>> - the output of nvidia-smi dmon executing *during* mdrun's execution
>>>
>>>
>>> heres a snapshot:
>>>
>>> # gpu   pwr  temp    sm   mem   enc   dec  mclk  pclk
>>>
>>>> # Idx     W     C     %     %     %     %   MHz   MHz
>>>>  0   117    51    56     3     0     0  4513  1847
>>>>     0   115    52    54     3     0     0  4513  1847
>>>>     0   117    52    47     3     0     0  4513  1847
>>>>     0   101    52    52     3     0     0  4513  1848
>>>>     0   119    53    54     3     0     0  4513  1847
>>>>     0   115    53    54     3     0     0  4513  1848
>>>>     0   106    54    54     3     0     0  4513  1848
>>>>     0   118    54    54     3     0     0  4513  1849
>>>>     0    94    54    52     3     0     0  4513  1848
>>>>     0   110    55    52     3     0     0  4513  1849
>>>>     0   116    55    53     3     0     0  4513  1854
>>>>     0   126    55    51     3     0     0  4513  1854
>>>>     0   119    56    51     3     0     0  4513  1855
>>>>     0   116    56    52     3     0     0  4513  1855
>>>>     0   116    56    53     3     0     0  4513  1855
>>>>     0   120    57    53     3     0     0  4513  1855
>>>>     0   113    57    54     3     0     0  4513  1855
>>>>
>>>>
>>> and here's the tail of the performance in .log
>>>
>>> Reading file md_0_2.tpr, VERSION 5.1.2 (single precision)
>>>
>>>> Changing nstlist from 10 to 40, rlist from 1 to 1.099
>>>> Using 1 MPI thread
>>>> Using 8 OpenMP threads
>>>> 1 compatible GPU is present, with ID 0
>>>> 1 GPU auto-selected for this run.
>>>> Mapping of GPU ID to the 1 PP rank in this node: 0
>>>>
>>>> Back Off! I just backed up md_0_2.trr to ./#md_0_2.trr.1#
>>>> Back Off! I just backed up md_0_2.xtc to ./#md_0_2.xtc.1#
>>>> Back Off! I just backed up md_0_2.edr to ./#md_0_2.edr.1#
>>>> starting mdrun 'Protein in water'
>>>> 500000 steps,   1000.0 ps.
>>>> Writing final coordinates.
>>>> Back Off! I just backed up md_0_2.gro to ./#md_0_2.gro.1#
>>>>                Core t (s)   Wall t (s)        (%)
>>>>        Time:    25285.344     3409.337      741.6
>>>>                          56:49
>>>>                  (ns/day)    (hour/ns)
>>>> Performance:       25.342        0.947
>>>> gcq#11: "She's Not Bad, She's Just Genetically Mean" (Captain Beefheart)
>>>>
>>>>
>>>> As we can see, there is *something* running on the GPU (process id:
>>> 30784),
>>> but its never more than 135MiB of memory being used and only uses a very
>>> small portion of the gpus power.
>>>
>>> Any thoughts are greatly appreciated!
>>>
>>>
>> You cut out the part of the .log file that shows how mdrun actually spent
>> its time.  This is perhaps the most valuable information.  This is why
>> Szilard asked for you to share a complete .log file (upload it somewhere,
>> post a URL) so we can see all relevant details.
>>
>> -Justin
>>
>>
>>
>>>
>>>
>>>
>>>
>>>
>>> On Thu, Jul 14, 2016 at 4:54 PM, Szilárd Páll <pall.szilard at gmail.com>
>>> wrote:
>>>
>>> Hi,
>>>>
>>>> If mdrun claims it's using the GPU and you see no errors (and results
>>>> look reasonable), the GPU is likely being used. That nvidia-smi is not
>>>> showing it is admittedly strange. However, this truncated log output
>>>> does not confirm much, e.g. it does not show the performance table and
>>>> the final performance.
>>>>
>>>> Could you please run again and look at (and share if approapriate):
>>>> - the entire log, not just parts, in particular we want to see the
>>>> header, perf table
>>>> - the output of nvidia-smi dmon executing *during* mdrun's execution
>>>> (this monitors the device continuously)
>>>>
>>>> Cheers,
>>>>
>>>> --
>>>> Szilárd
>>>>
>>>>
>>>> On Thu, Jul 14, 2016 at 8:28 PM, Vito Spadavecchio
>>>> <spadavecchio at gmail.com> wrote:
>>>>
>>>>> Hello
>>>>>
>>>>> After building GROMACS with GPU support and running some simple MD, I
>>>>> am
>>>>> only getting ~20ns/day (which is what I would expect to get out of my
>>>>> i7-6700k cpu only. My machine currently has a 1080 GTX in it, which
>>>>>
>>>> should
>>>>
>>>>> even on moderately large systems, be getting ~100ns/day)
>>>>>
>>>>>  I've tried launching a mdrun with with the following command:
>>>>>
>>>>>   gmx mdrun -deffnm md_0_1 -gpu_id 0 -nb gpu
>>>>>
>>>>>
>>>>>
>>>>> With the following in my md.mdp file:
>>>>>
>>>>> ; Neighborsearching
>>>>>
>>>>> cutoff-scheme   = Verlet
>>>>>
>>>>>
>>>>>
>>>>> The top of the log file for the mdrun notes:
>>>>>
>>>>> Running on 1 node with total 4 cores, 8 logical cores, 1 compatible GPU
>>>>>
>>>>>> Hardware detected:
>>>>>>   CPU info:
>>>>>>     Vendor: GenuineIntel
>>>>>>     Brand:  Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
>>>>>>     SIMD instructions most likely to fit this hardware: AVX2_256
>>>>>>     SIMD instructions selected at GROMACS compile time: AVX2_256
>>>>>>   GPU info:
>>>>>>     Number of GPUs detected: 1
>>>>>>     #0: NVIDIA GeForce GTX 1080, compute cap.: 6.1, ECC:  no, stat:
>>>>>> compatible
>>>>>> Reading file md_0_1.tpr, VERSION 5.1.2 (single precision)
>>>>>> Changing nstlist from 10 to 40, rlist from 1 to 1.099
>>>>>> Using 1 MPI thread
>>>>>> Using 8 OpenMP threads
>>>>>> 1 GPU user-selected for this run.
>>>>>> Mapping of GPU ID to the 1 PP rank in this node: 0
>>>>>>
>>>>>>
>>>>> The output of *nvidia-smi *on my machine is:
>>>>>
>>>>> hu Jul 14 11:25:04 2016
>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>> +-----------------------------------------------------------------------------+
>>>>
>>>>> | NVIDIA-SMI 367.27                 Driver Version: 367.27
>>>>>>    |
>>>>>>
>>>>>>
>>>>>>
>>>> |-------------------------------+----------------------+----------------------+
>>>>
>>>>> | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile
>>>>>>
>>>>> Uncorr.
>>>>
>>>>> ECC |
>>>>>> | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util
>>>>>>
>>>>> Compute
>>>>
>>>>> M. |
>>>>>>
>>>>>>
>>>>>>
>>>> |===============================+======================+======================|
>>>>
>>>>> |   0  GeForce GTX 1080    Off  | 0000:01:00.0      On |
>>>>>>  N/A |
>>>>>> | 30%   41C    P8     7W / 180W |    730MiB /  8113MiB |      3%
>>>>>>  Default |
>>>>>>
>>>>>>
>>>>>>
>>>> +-------------------------------+----------------------+----------------------+
>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>> +-----------------------------------------------------------------------------+
>>>>
>>>>> | Processes:                                                       GPU
>>>>>> Memory |
>>>>>> |  GPU       PID  Type  Process name
>>>>>>  Usage
>>>>>>    |
>>>>>>
>>>>>>
>>>>>>
>>>> |=============================================================================|
>>>>
>>>>> |    0      1035    G   /usr/lib/xorg/Xorg
>>>>>> 424MiB |
>>>>>> |    0      2132    G   compiz
>>>>>> 103MiB |
>>>>>> |    0      5296    G   ...s-passed-by-fd --v8-snapshot-passed-by-fd
>>>>>>  59MiB |
>>>>>> |    0     14079    G   ...s-passed-by-fd --v8-snapshot-passed-by-fd
>>>>>>  31MiB |
>>>>>> |    0     14234    G   ...ing<BrotliEncoding --disable-features=Doc
>>>>>> 108MiB |
>>>>>>
>>>>>>
>>>>>>
>>>> +-----------------------------------------------------------------------------+
>>>>
>>>>>
>>>>>
>>>>> and output of *nvcc --version* is:
>>>>>
>>>>> nvcc: NVIDIA (R) Cuda compiler driver
>>>>>
>>>>>> Copyright (c) 2005-2015 NVIDIA Corporation
>>>>>> Built on Tue_Aug_11_14:27:32_CDT_2015
>>>>>> Cuda compilation tools, release 7.5, V7.5.17
>>>>>>
>>>>>
>>>>>
>>>>> Does anyone have any ideas what the problem might be?
>>>>>
>>>>> Thanks!
>>>>> --
>>>>> Gromacs Users mailing list
>>>>>
>>>>> * Please search the archive at
>>>>>
>>>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
>>>> posting!
>>>>
>>>>>
>>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>>>>
>>>>> * For (un)subscribe requests visit
>>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>>>>>
>>>> send a mail to gmx-users-request at gromacs.org.
>>>> --
>>>> Gromacs Users mailing list
>>>>
>>>> * Please search the archive at
>>>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
>>>> posting!
>>>>
>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>>>
>>>> * For (un)subscribe requests visit
>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>>>> send a mail to gmx-users-request at gromacs.org.
>>>>
>>>>
>> --
>> ==================================================
>>
>> Justin A. Lemkul, Ph.D.
>> Ruth L. Kirschstein NRSA Postdoctoral Fellow
>>
>> Department of Pharmaceutical Sciences
>> School of Pharmacy
>> Health Sciences Facility II, Room 629
>> University of Maryland, Baltimore
>> 20 Penn St.
>> Baltimore, MD 21201
>>
>> jalemkul at outerbanks.umaryland.edu | (410) 706-7441
>> http://mackerell.umaryland.edu/~jalemkul
>>
>> ==================================================
>>
>> --
>> Gromacs Users mailing list
>>
>> * Please search the archive at
>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
>> posting!
>>
>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>
>> * For (un)subscribe requests visit
>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>> send a mail to gmx-users-request at gromacs.org.
>>
>
>


More information about the gromacs.org_gmx-users mailing list