[gmx-users] Gromacs 2019.2 on Power9 + Volta GPUs (building and running)

Alex nedomacho at gmail.com
Thu May 2 01:18:54 CEST 2019


Well, unless something important has changed within a year, I distinctly 
remember being advised here not to offload anything to GPU for EM. Not 
that we ever needed to, to be honest...

In any case, we appear to be dealing with build issues here.

Alex

On 5/1/2019 5:09 PM, Kevin Boyd wrote:
> Hi,
>
>> Of course, i am not. This is the EM. ;)
> I haven't looked back at the code, but IIRC EM can use GPUs for the
> nonbondeds, just not the PME. I just double-checked on one of my systems
> with 10 cores and a GTX 1080 Ti, offloading to the GPU more than doubled
> the minimization speed.
>
> Kevin
>
> On Wed, May 1, 2019 at 6:33 PM Alex <nedomacho at gmail.com> wrote:
>
>> Of course, i am not. This is the EM. ;)
>>
>> On Wed, May 1, 2019, 4:30 PM Kevin Boyd <kevin.boyd at uconn.edu> wrote:
>>
>>> Hi,
>>>
>>> In addition to what Mark said (and I've also found pinning to be critical
>>> for performance), you're also not using the GPUs with "-pme cpu -nb cpu".
>>>
>>> Kevin
>>>
>>> On Wed, May 1, 2019 at 5:56 PM Alex <nedomacho at gmail.com> wrote:
>>>
>>>> Well, my experience so far has been with the EM, because the rest of
>> the
>>>> script (with all the dynamic things) needed that to finish. And it
>>>> "finished" by hitting the wall. However, your comment does touch upon
>>> what
>>>> to do with thread pinning and I will try to set '-pin on' throughout to
>>> see
>>>> if things make a difference for the better. I am less confident about
>>>> setting strides because it is unclear what the job manager provides in
>>>> terms of the available core numbers. I will play around some more and
>>>> report here.
>>>>
>>>> Thanks!
>>>>
>>>> Alex
>>>>
>>>> On Wed, May 1, 2019 at 3:49 PM Mark Abraham <mark.j.abraham at gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> As with x86, GROMACS uses SIMD intrinsics on POWER9 and is thus
>> fairly
>>>>> insensitive to the compiler's vectorisation abilities. GCC is the
>> only
>>>>> compiler we've tested, as xlc can't compile simple C++11. As
>>> everywhere,
>>>>> you should use the latest version of gcc, as IBM spent quite some
>> years
>>>>> landing improvements for POWER9.
>>>>>
>>>>> EM is useless as a performance indicator of a dynamical simulation,
>>> avoid
>>>>> that - it runs serial code much much more often.
>>>>>
>>>>> Your run deliberately didn't fill the available cores, so just like
>> on
>>>> x86,
>>>>> mdrun will leave the thread affinity handling to the environment,
>> which
>>>> is
>>>>> often a path to bad performance. So, if you plan on doing that often,
>>>>> you'll want to check out the mdrun performance guide docs about the
>>> mdrun
>>>>> -pin and related options.
>>>>>
>>>>> Mark
>>>>>
>>>>>
>>>>> On Wed., 1 May 2019, 23:21 Alex, <nedomacho at gmail.com> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> Our institution decided to be all fancy, so now we have a bunch of
>>>> Power9
>>>>>> nodes, each with 80 cores + 4 Volta GPUs. Stuff is managed by
>> slurm.
>>>>> Today
>>>>>> I did a simple EM ('gmx mdrun -ntomp 4 -ntmpi 4 -pme cpu -nb cpu')
>>> and
>>>>> the
>>>>>> performance is abysmal, I would guess 100 times slower than on
>>> anything
>>>>>> I've ever seen before.
>>>>>>
>>>>>> Our admin person emailed me the following:
>>>>>> "-- it would not surprise me if the GCC compilers were relatively
>> bad
>>>> at
>>>>>> taking advantage of POWER9 vectorization, they're likely optimized
>>> for
>>>>>> x86_64 vector stuff like SSE and AVX operations.  This was an issue
>>> in
>>>>> the
>>>>>> build, I selected "-DGMX_SIMD=IBM_VSX" for the config, but
>> according
>>> to
>>>>> my
>>>>>> notes, that was part of an attempt to fix the "unimplemented SIMD"
>>>> error
>>>>>> that was dogging me at first, and/but which was eventually cleared
>> by
>>>>>> switching to gcc-6."
>>>>>>
>>>>>> Does anyone have any comments/suggestions on building and running
>> GMX
>>>> on
>>>>>> Power9?
>>>>>>
>>>>>> Thank you,
>>>>>>
>>>>>> Alex
>>>>>> --
>>>>>> Gromacs Users mailing list
>>>>>>
>>>>>> * Please search the archive at
>>>>>>
>> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists%2FGMX-Users_List&amp;data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018052656&amp;sdata=zejDS0OvUCl%2BSch%2BzVtxic%2B%2BDFIPEhB1DygmpmQ2dvw%3D&amp;reserved=0
>>>> before
>>>>>> posting!
>>>>>>
>>>>>> * Can't post? Read
>> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists&amp;data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018052656&amp;sdata=X87Kk%2FtkodePJ9uhDb9XPIA0Xhaqi52e6Z9%2FhqY35fo%3D&amp;reserved=0
>>>>>> * For (un)subscribe requests visit
>>>>>>
>> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmaillist.sys.kth.se%2Fmailman%2Flistinfo%2Fgromacs.org_gmx-users&amp;data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018052656&amp;sdata=mJ%2FfYvTgmL49ZCAUYzSRJqz%2FJY8MxQdGpoYwKtbN39U%3D&amp;reserved=0
>>>> or
>>>>>> send a mail to gmx-users-request at gromacs.org.
>>>>>>
>>>>> --
>>>>> Gromacs Users mailing list
>>>>>
>>>>> * Please search the archive at
>>>>>
>> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists%2FGMX-Users_List&amp;data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018052656&amp;sdata=zejDS0OvUCl%2BSch%2BzVtxic%2B%2BDFIPEhB1DygmpmQ2dvw%3D&amp;reserved=0
>>>> before
>>>>> posting!
>>>>>
>>>>> * Can't post? Read
>> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists&amp;data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018052656&amp;sdata=X87Kk%2FtkodePJ9uhDb9XPIA0Xhaqi52e6Z9%2FhqY35fo%3D&amp;reserved=0
>>>>> * For (un)subscribe requests visit
>>>>>
>> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmaillist.sys.kth.se%2Fmailman%2Flistinfo%2Fgromacs.org_gmx-users&amp;data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018052656&amp;sdata=mJ%2FfYvTgmL49ZCAUYzSRJqz%2FJY8MxQdGpoYwKtbN39U%3D&amp;reserved=0
>>>> or
>>>>> send a mail to gmx-users-request at gromacs.org.
>>>>>
>>>> --
>>>> Gromacs Users mailing list
>>>>
>>>> * Please search the archive at
>>>>
>> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists%2FGMX-Users_List&amp;data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018052656&amp;sdata=zejDS0OvUCl%2BSch%2BzVtxic%2B%2BDFIPEhB1DygmpmQ2dvw%3D&amp;reserved=0
>>>> before posting!
>>>>
>>>> * Can't post? Read
>>>>
>> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists&amp;data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018052656&amp;sdata=X87Kk%2FtkodePJ9uhDb9XPIA0Xhaqi52e6Z9%2FhqY35fo%3D&amp;reserved=0
>>>> * For (un)subscribe requests visit
>>>>
>>>>
>> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmaillist.sys.kth.se%2Fmailman%2Flistinfo%2Fgromacs.org_gmx-users&amp;data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018062645&amp;sdata=jxfG7UF4cZYh%2FOnDgFR%2F%2FcNgCrD7DytsWNXgvtd8wlU%3D&amp;reserved=0
>>>> or send a mail to gmx-users-request at gromacs.org.
>>>>
>>> --
>>> Gromacs Users mailing list
>>>
>>> * Please search the archive at
>>>
>> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists%2FGMX-Users_List&amp;data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018062645&amp;sdata=VXuy%2Bwx5Pcv3vgqNNLcXVbcsiAgWHI3nLdhqD4Yc7Io%3D&amp;reserved=0
>> before
>>> posting!
>>>
>>> * Can't post? Read
>> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists&amp;data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018062645&amp;sdata=hNfd9HQH4i7TmU4wVUaos%2BzxqkRuKpEcVyswRFBrnyk%3D&amp;reserved=0
>>> * For (un)subscribe requests visit
>>>
>> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmaillist.sys.kth.se%2Fmailman%2Flistinfo%2Fgromacs.org_gmx-users&amp;data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018062645&amp;sdata=jxfG7UF4cZYh%2FOnDgFR%2F%2FcNgCrD7DytsWNXgvtd8wlU%3D&amp;reserved=0
>> or
>>> send a mail to gmx-users-request at gromacs.org.
>>>
>> --
>> Gromacs Users mailing list
>>
>> * Please search the archive at
>> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists%2FGMX-Users_List&amp;data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018062645&amp;sdata=VXuy%2Bwx5Pcv3vgqNNLcXVbcsiAgWHI3nLdhqD4Yc7Io%3D&amp;reserved=0
>> before posting!
>>
>> * Can't post? Read
>> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists&amp;data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018062645&amp;sdata=hNfd9HQH4i7TmU4wVUaos%2BzxqkRuKpEcVyswRFBrnyk%3D&amp;reserved=0
>>
>> * For (un)subscribe requests visit
>>
>> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmaillist.sys.kth.se%2Fmailman%2Flistinfo%2Fgromacs.org_gmx-users&amp;data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018062645&amp;sdata=jxfG7UF4cZYh%2FOnDgFR%2F%2FcNgCrD7DytsWNXgvtd8wlU%3D&amp;reserved=0
>> or send a mail to gmx-users-request at gromacs.org.
>>


More information about the gromacs.org_gmx-users mailing list