[gmx-users] Gromacs 2019.2 on Power9 + Volta GPUs (building and running)
Szilárd Páll
pall.szilard at gmail.com
Thu May 2 09:47:58 CEST 2019
Power9 (for HPC) is 4-way SMT, so make sure to try 1,2, and 4 threads per
core (stride 4, 2, and 1 respectively). Especially if you are offloading
all force computing to the GPU, what remains on the couch may not be able
to benefit from more than 1-2 threads per core.
--
Szilárd
On Thu, May 2, 2019, 01:19 Alex <nedomacho at gmail.com> wrote:
> Well, unless something important has changed within a year, I distinctly
> remember being advised here not to offload anything to GPU for EM. Not
> that we ever needed to, to be honest...
>
> In any case, we appear to be dealing with build issues here.
>
> Alex
>
> On 5/1/2019 5:09 PM, Kevin Boyd wrote:
> > Hi,
> >
> >> Of course, i am not. This is the EM. ;)
> > I haven't looked back at the code, but IIRC EM can use GPUs for the
> > nonbondeds, just not the PME. I just double-checked on one of my systems
> > with 10 cores and a GTX 1080 Ti, offloading to the GPU more than doubled
> > the minimization speed.
> >
> > Kevin
> >
> > On Wed, May 1, 2019 at 6:33 PM Alex <nedomacho at gmail.com> wrote:
> >
> >> Of course, i am not. This is the EM. ;)
> >>
> >> On Wed, May 1, 2019, 4:30 PM Kevin Boyd <kevin.boyd at uconn.edu> wrote:
> >>
> >>> Hi,
> >>>
> >>> In addition to what Mark said (and I've also found pinning to be
> critical
> >>> for performance), you're also not using the GPUs with "-pme cpu -nb
> cpu".
> >>>
> >>> Kevin
> >>>
> >>> On Wed, May 1, 2019 at 5:56 PM Alex <nedomacho at gmail.com> wrote:
> >>>
> >>>> Well, my experience so far has been with the EM, because the rest of
> >> the
> >>>> script (with all the dynamic things) needed that to finish. And it
> >>>> "finished" by hitting the wall. However, your comment does touch upon
> >>> what
> >>>> to do with thread pinning and I will try to set '-pin on' throughout
> to
> >>> see
> >>>> if things make a difference for the better. I am less confident about
> >>>> setting strides because it is unclear what the job manager provides in
> >>>> terms of the available core numbers. I will play around some more and
> >>>> report here.
> >>>>
> >>>> Thanks!
> >>>>
> >>>> Alex
> >>>>
> >>>> On Wed, May 1, 2019 at 3:49 PM Mark Abraham <mark.j.abraham at gmail.com
> >
> >>>> wrote:
> >>>>
> >>>>> Hi,
> >>>>>
> >>>>> As with x86, GROMACS uses SIMD intrinsics on POWER9 and is thus
> >> fairly
> >>>>> insensitive to the compiler's vectorisation abilities. GCC is the
> >> only
> >>>>> compiler we've tested, as xlc can't compile simple C++11. As
> >>> everywhere,
> >>>>> you should use the latest version of gcc, as IBM spent quite some
> >> years
> >>>>> landing improvements for POWER9.
> >>>>>
> >>>>> EM is useless as a performance indicator of a dynamical simulation,
> >>> avoid
> >>>>> that - it runs serial code much much more often.
> >>>>>
> >>>>> Your run deliberately didn't fill the available cores, so just like
> >> on
> >>>> x86,
> >>>>> mdrun will leave the thread affinity handling to the environment,
> >> which
> >>>> is
> >>>>> often a path to bad performance. So, if you plan on doing that often,
> >>>>> you'll want to check out the mdrun performance guide docs about the
> >>> mdrun
> >>>>> -pin and related options.
> >>>>>
> >>>>> Mark
> >>>>>
> >>>>>
> >>>>> On Wed., 1 May 2019, 23:21 Alex, <nedomacho at gmail.com> wrote:
> >>>>>
> >>>>>> Hi all,
> >>>>>>
> >>>>>> Our institution decided to be all fancy, so now we have a bunch of
> >>>> Power9
> >>>>>> nodes, each with 80 cores + 4 Volta GPUs. Stuff is managed by
> >> slurm.
> >>>>> Today
> >>>>>> I did a simple EM ('gmx mdrun -ntomp 4 -ntmpi 4 -pme cpu -nb cpu')
> >>> and
> >>>>> the
> >>>>>> performance is abysmal, I would guess 100 times slower than on
> >>> anything
> >>>>>> I've ever seen before.
> >>>>>>
> >>>>>> Our admin person emailed me the following:
> >>>>>> "-- it would not surprise me if the GCC compilers were relatively
> >> bad
> >>>> at
> >>>>>> taking advantage of POWER9 vectorization, they're likely optimized
> >>> for
> >>>>>> x86_64 vector stuff like SSE and AVX operations. This was an issue
> >>> in
> >>>>> the
> >>>>>> build, I selected "-DGMX_SIMD=IBM_VSX" for the config, but
> >> according
> >>> to
> >>>>> my
> >>>>>> notes, that was part of an attempt to fix the "unimplemented SIMD"
> >>>> error
> >>>>>> that was dogging me at first, and/but which was eventually cleared
> >> by
> >>>>>> switching to gcc-6."
> >>>>>>
> >>>>>> Does anyone have any comments/suggestions on building and running
> >> GMX
> >>>> on
> >>>>>> Power9?
> >>>>>>
> >>>>>> Thank you,
> >>>>>>
> >>>>>> Alex
> >>>>>> --
> >>>>>> Gromacs Users mailing list
> >>>>>>
> >>>>>> * Please search the archive at
> >>>>>>
> >>
> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists%2FGMX-Users_List&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018052656&sdata=zejDS0OvUCl%2BSch%2BzVtxic%2B%2BDFIPEhB1DygmpmQ2dvw%3D&reserved=0
> >>>> before
> >>>>>> posting!
> >>>>>>
> >>>>>> * Can't post? Read
> >>
> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018052656&sdata=X87Kk%2FtkodePJ9uhDb9XPIA0Xhaqi52e6Z9%2FhqY35fo%3D&reserved=0
> >>>>>> * For (un)subscribe requests visit
> >>>>>>
> >>
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmaillist.sys.kth.se%2Fmailman%2Flistinfo%2Fgromacs.org_gmx-users&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018052656&sdata=mJ%2FfYvTgmL49ZCAUYzSRJqz%2FJY8MxQdGpoYwKtbN39U%3D&reserved=0
> >>>> or
> >>>>>> send a mail to gmx-users-request at gromacs.org.
> >>>>>>
> >>>>> --
> >>>>> Gromacs Users mailing list
> >>>>>
> >>>>> * Please search the archive at
> >>>>>
> >>
> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists%2FGMX-Users_List&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018052656&sdata=zejDS0OvUCl%2BSch%2BzVtxic%2B%2BDFIPEhB1DygmpmQ2dvw%3D&reserved=0
> >>>> before
> >>>>> posting!
> >>>>>
> >>>>> * Can't post? Read
> >>
> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018052656&sdata=X87Kk%2FtkodePJ9uhDb9XPIA0Xhaqi52e6Z9%2FhqY35fo%3D&reserved=0
> >>>>> * For (un)subscribe requests visit
> >>>>>
> >>
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmaillist.sys.kth.se%2Fmailman%2Flistinfo%2Fgromacs.org_gmx-users&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018052656&sdata=mJ%2FfYvTgmL49ZCAUYzSRJqz%2FJY8MxQdGpoYwKtbN39U%3D&reserved=0
> >>>> or
> >>>>> send a mail to gmx-users-request at gromacs.org.
> >>>>>
> >>>> --
> >>>> Gromacs Users mailing list
> >>>>
> >>>> * Please search the archive at
> >>>>
> >>
> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists%2FGMX-Users_List&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018052656&sdata=zejDS0OvUCl%2BSch%2BzVtxic%2B%2BDFIPEhB1DygmpmQ2dvw%3D&reserved=0
> >>>> before posting!
> >>>>
> >>>> * Can't post? Read
> >>>>
> >>
> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018052656&sdata=X87Kk%2FtkodePJ9uhDb9XPIA0Xhaqi52e6Z9%2FhqY35fo%3D&reserved=0
> >>>> * For (un)subscribe requests visit
> >>>>
> >>>>
> >>
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmaillist.sys.kth.se%2Fmailman%2Flistinfo%2Fgromacs.org_gmx-users&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018062645&sdata=jxfG7UF4cZYh%2FOnDgFR%2F%2FcNgCrD7DytsWNXgvtd8wlU%3D&reserved=0
> >>>> or send a mail to gmx-users-request at gromacs.org.
> >>>>
> >>> --
> >>> Gromacs Users mailing list
> >>>
> >>> * Please search the archive at
> >>>
> >>
> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists%2FGMX-Users_List&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018062645&sdata=VXuy%2Bwx5Pcv3vgqNNLcXVbcsiAgWHI3nLdhqD4Yc7Io%3D&reserved=0
> >> before
> >>> posting!
> >>>
> >>> * Can't post? Read
> >>
> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018062645&sdata=hNfd9HQH4i7TmU4wVUaos%2BzxqkRuKpEcVyswRFBrnyk%3D&reserved=0
> >>> * For (un)subscribe requests visit
> >>>
> >>
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmaillist.sys.kth.se%2Fmailman%2Flistinfo%2Fgromacs.org_gmx-users&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018062645&sdata=jxfG7UF4cZYh%2FOnDgFR%2F%2FcNgCrD7DytsWNXgvtd8wlU%3D&reserved=0
> >> or
> >>> send a mail to gmx-users-request at gromacs.org.
> >>>
> >> --
> >> Gromacs Users mailing list
> >>
> >> * Please search the archive at
> >>
> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists%2FGMX-Users_List&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018062645&sdata=VXuy%2Bwx5Pcv3vgqNNLcXVbcsiAgWHI3nLdhqD4Yc7Io%3D&reserved=0
> >> before posting!
> >>
> >> * Can't post? Read
> >>
> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018062645&sdata=hNfd9HQH4i7TmU4wVUaos%2BzxqkRuKpEcVyswRFBrnyk%3D&reserved=0
> >>
> >> * For (un)subscribe requests visit
> >>
> >>
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmaillist.sys.kth.se%2Fmailman%2Flistinfo%2Fgromacs.org_gmx-users&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018062645&sdata=jxfG7UF4cZYh%2FOnDgFR%2F%2FcNgCrD7DytsWNXgvtd8wlU%3D&reserved=0
> >> or send a mail to gmx-users-request at gromacs.org.
> >>
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
>
More information about the gromacs.org_gmx-users
mailing list