[gmx-users] Gromacs 2019.2 on Power9 + Volta GPUs (building and running)
Kevin Boyd
kevin.boyd at uconn.edu
Thu May 2 01:09:35 CEST 2019
Hi,
>Of course, i am not. This is the EM. ;)
I haven't looked back at the code, but IIRC EM can use GPUs for the
nonbondeds, just not the PME. I just double-checked on one of my systems
with 10 cores and a GTX 1080 Ti, offloading to the GPU more than doubled
the minimization speed.
Kevin
On Wed, May 1, 2019 at 6:33 PM Alex <nedomacho at gmail.com> wrote:
> Of course, i am not. This is the EM. ;)
>
> On Wed, May 1, 2019, 4:30 PM Kevin Boyd <kevin.boyd at uconn.edu> wrote:
>
> > Hi,
> >
> > In addition to what Mark said (and I've also found pinning to be critical
> > for performance), you're also not using the GPUs with "-pme cpu -nb cpu".
> >
> > Kevin
> >
> > On Wed, May 1, 2019 at 5:56 PM Alex <nedomacho at gmail.com> wrote:
> >
> > > Well, my experience so far has been with the EM, because the rest of
> the
> > > script (with all the dynamic things) needed that to finish. And it
> > > "finished" by hitting the wall. However, your comment does touch upon
> > what
> > > to do with thread pinning and I will try to set '-pin on' throughout to
> > see
> > > if things make a difference for the better. I am less confident about
> > > setting strides because it is unclear what the job manager provides in
> > > terms of the available core numbers. I will play around some more and
> > > report here.
> > >
> > > Thanks!
> > >
> > > Alex
> > >
> > > On Wed, May 1, 2019 at 3:49 PM Mark Abraham <mark.j.abraham at gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > As with x86, GROMACS uses SIMD intrinsics on POWER9 and is thus
> fairly
> > > > insensitive to the compiler's vectorisation abilities. GCC is the
> only
> > > > compiler we've tested, as xlc can't compile simple C++11. As
> > everywhere,
> > > > you should use the latest version of gcc, as IBM spent quite some
> years
> > > > landing improvements for POWER9.
> > > >
> > > > EM is useless as a performance indicator of a dynamical simulation,
> > avoid
> > > > that - it runs serial code much much more often.
> > > >
> > > > Your run deliberately didn't fill the available cores, so just like
> on
> > > x86,
> > > > mdrun will leave the thread affinity handling to the environment,
> which
> > > is
> > > > often a path to bad performance. So, if you plan on doing that often,
> > > > you'll want to check out the mdrun performance guide docs about the
> > mdrun
> > > > -pin and related options.
> > > >
> > > > Mark
> > > >
> > > >
> > > > On Wed., 1 May 2019, 23:21 Alex, <nedomacho at gmail.com> wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > Our institution decided to be all fancy, so now we have a bunch of
> > > Power9
> > > > > nodes, each with 80 cores + 4 Volta GPUs. Stuff is managed by
> slurm.
> > > > Today
> > > > > I did a simple EM ('gmx mdrun -ntomp 4 -ntmpi 4 -pme cpu -nb cpu')
> > and
> > > > the
> > > > > performance is abysmal, I would guess 100 times slower than on
> > anything
> > > > > I've ever seen before.
> > > > >
> > > > > Our admin person emailed me the following:
> > > > > "-- it would not surprise me if the GCC compilers were relatively
> bad
> > > at
> > > > > taking advantage of POWER9 vectorization, they're likely optimized
> > for
> > > > > x86_64 vector stuff like SSE and AVX operations. This was an issue
> > in
> > > > the
> > > > > build, I selected "-DGMX_SIMD=IBM_VSX" for the config, but
> according
> > to
> > > > my
> > > > > notes, that was part of an attempt to fix the "unimplemented SIMD"
> > > error
> > > > > that was dogging me at first, and/but which was eventually cleared
> by
> > > > > switching to gcc-6."
> > > > >
> > > > > Does anyone have any comments/suggestions on building and running
> GMX
> > > on
> > > > > Power9?
> > > > >
> > > > > Thank you,
> > > > >
> > > > > Alex
> > > > > --
> > > > > Gromacs Users mailing list
> > > > >
> > > > > * Please search the archive at
> > > > >
> > >
> >
> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists%2FGMX-Users_List&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018052656&sdata=zejDS0OvUCl%2BSch%2BzVtxic%2B%2BDFIPEhB1DygmpmQ2dvw%3D&reserved=0
> > > before
> > > > > posting!
> > > > >
> > > > > * Can't post? Read
> > >
> >
> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018052656&sdata=X87Kk%2FtkodePJ9uhDb9XPIA0Xhaqi52e6Z9%2FhqY35fo%3D&reserved=0
> > > > >
> > > > > * For (un)subscribe requests visit
> > > > >
> > >
> >
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmaillist.sys.kth.se%2Fmailman%2Flistinfo%2Fgromacs.org_gmx-users&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018052656&sdata=mJ%2FfYvTgmL49ZCAUYzSRJqz%2FJY8MxQdGpoYwKtbN39U%3D&reserved=0
> > > or
> > > > > send a mail to gmx-users-request at gromacs.org.
> > > > >
> > > > --
> > > > Gromacs Users mailing list
> > > >
> > > > * Please search the archive at
> > > >
> > >
> >
> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists%2FGMX-Users_List&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018052656&sdata=zejDS0OvUCl%2BSch%2BzVtxic%2B%2BDFIPEhB1DygmpmQ2dvw%3D&reserved=0
> > > before
> > > > posting!
> > > >
> > > > * Can't post? Read
> > >
> >
> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018052656&sdata=X87Kk%2FtkodePJ9uhDb9XPIA0Xhaqi52e6Z9%2FhqY35fo%3D&reserved=0
> > > >
> > > > * For (un)subscribe requests visit
> > > >
> > >
> >
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmaillist.sys.kth.se%2Fmailman%2Flistinfo%2Fgromacs.org_gmx-users&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018052656&sdata=mJ%2FfYvTgmL49ZCAUYzSRJqz%2FJY8MxQdGpoYwKtbN39U%3D&reserved=0
> > > or
> > > > send a mail to gmx-users-request at gromacs.org.
> > > >
> > > --
> > > Gromacs Users mailing list
> > >
> > > * Please search the archive at
> > >
> >
> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists%2FGMX-Users_List&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018052656&sdata=zejDS0OvUCl%2BSch%2BzVtxic%2B%2BDFIPEhB1DygmpmQ2dvw%3D&reserved=0
> > > before posting!
> > >
> > > * Can't post? Read
> > >
> >
> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018052656&sdata=X87Kk%2FtkodePJ9uhDb9XPIA0Xhaqi52e6Z9%2FhqY35fo%3D&reserved=0
> > >
> > > * For (un)subscribe requests visit
> > >
> > >
> >
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmaillist.sys.kth.se%2Fmailman%2Flistinfo%2Fgromacs.org_gmx-users&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018062645&sdata=jxfG7UF4cZYh%2FOnDgFR%2F%2FcNgCrD7DytsWNXgvtd8wlU%3D&reserved=0
> > > or send a mail to gmx-users-request at gromacs.org.
> > >
> > --
> > Gromacs Users mailing list
> >
> > * Please search the archive at
> >
> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists%2FGMX-Users_List&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018062645&sdata=VXuy%2Bwx5Pcv3vgqNNLcXVbcsiAgWHI3nLdhqD4Yc7Io%3D&reserved=0
> before
> > posting!
> >
> > * Can't post? Read
> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018062645&sdata=hNfd9HQH4i7TmU4wVUaos%2BzxqkRuKpEcVyswRFBrnyk%3D&reserved=0
> >
> > * For (un)subscribe requests visit
> >
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmaillist.sys.kth.se%2Fmailman%2Flistinfo%2Fgromacs.org_gmx-users&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018062645&sdata=jxfG7UF4cZYh%2FOnDgFR%2F%2FcNgCrD7DytsWNXgvtd8wlU%3D&reserved=0
> or
> > send a mail to gmx-users-request at gromacs.org.
> >
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists%2FGMX-Users_List&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018062645&sdata=VXuy%2Bwx5Pcv3vgqNNLcXVbcsiAgWHI3nLdhqD4Yc7Io%3D&reserved=0
> before posting!
>
> * Can't post? Read
> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018062645&sdata=hNfd9HQH4i7TmU4wVUaos%2BzxqkRuKpEcVyswRFBrnyk%3D&reserved=0
>
> * For (un)subscribe requests visit
>
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmaillist.sys.kth.se%2Fmailman%2Flistinfo%2Fgromacs.org_gmx-users&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018062645&sdata=jxfG7UF4cZYh%2FOnDgFR%2F%2FcNgCrD7DytsWNXgvtd8wlU%3D&reserved=0
> or send a mail to gmx-users-request at gromacs.org.
>
More information about the gromacs.org_gmx-users
mailing list