[gmx-users] Gromacs 2019.2 on Power9 + Volta GPUs (building and running)
Kevin Boyd
kevin.boyd at uconn.edu
Thu May 2 00:30:38 CEST 2019
Hi,
In addition to what Mark said (and I've also found pinning to be critical
for performance), you're also not using the GPUs with "-pme cpu -nb cpu".
Kevin
On Wed, May 1, 2019 at 5:56 PM Alex <nedomacho at gmail.com> wrote:
> Well, my experience so far has been with the EM, because the rest of the
> script (with all the dynamic things) needed that to finish. And it
> "finished" by hitting the wall. However, your comment does touch upon what
> to do with thread pinning and I will try to set '-pin on' throughout to see
> if things make a difference for the better. I am less confident about
> setting strides because it is unclear what the job manager provides in
> terms of the available core numbers. I will play around some more and
> report here.
>
> Thanks!
>
> Alex
>
> On Wed, May 1, 2019 at 3:49 PM Mark Abraham <mark.j.abraham at gmail.com>
> wrote:
>
> > Hi,
> >
> > As with x86, GROMACS uses SIMD intrinsics on POWER9 and is thus fairly
> > insensitive to the compiler's vectorisation abilities. GCC is the only
> > compiler we've tested, as xlc can't compile simple C++11. As everywhere,
> > you should use the latest version of gcc, as IBM spent quite some years
> > landing improvements for POWER9.
> >
> > EM is useless as a performance indicator of a dynamical simulation, avoid
> > that - it runs serial code much much more often.
> >
> > Your run deliberately didn't fill the available cores, so just like on
> x86,
> > mdrun will leave the thread affinity handling to the environment, which
> is
> > often a path to bad performance. So, if you plan on doing that often,
> > you'll want to check out the mdrun performance guide docs about the mdrun
> > -pin and related options.
> >
> > Mark
> >
> >
> > On Wed., 1 May 2019, 23:21 Alex, <nedomacho at gmail.com> wrote:
> >
> > > Hi all,
> > >
> > > Our institution decided to be all fancy, so now we have a bunch of
> Power9
> > > nodes, each with 80 cores + 4 Volta GPUs. Stuff is managed by slurm.
> > Today
> > > I did a simple EM ('gmx mdrun -ntomp 4 -ntmpi 4 -pme cpu -nb cpu') and
> > the
> > > performance is abysmal, I would guess 100 times slower than on anything
> > > I've ever seen before.
> > >
> > > Our admin person emailed me the following:
> > > "-- it would not surprise me if the GCC compilers were relatively bad
> at
> > > taking advantage of POWER9 vectorization, they're likely optimized for
> > > x86_64 vector stuff like SSE and AVX operations. This was an issue in
> > the
> > > build, I selected "-DGMX_SIMD=IBM_VSX" for the config, but according to
> > my
> > > notes, that was part of an attempt to fix the "unimplemented SIMD"
> error
> > > that was dogging me at first, and/but which was eventually cleared by
> > > switching to gcc-6."
> > >
> > > Does anyone have any comments/suggestions on building and running GMX
> on
> > > Power9?
> > >
> > > Thank you,
> > >
> > > Alex
> > > --
> > > Gromacs Users mailing list
> > >
> > > * Please search the archive at
> > >
> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists%2FGMX-Users_List&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C4c05490f75ba4dc9658e08d6ce7fd451%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923445772493263&sdata=hInjXVJw1xyIo23W3Q%2Fnt5UlXy%2Bx5mok7re4cpCopG8%3D&reserved=0
> before
> > > posting!
> > >
> > > * Can't post? Read
> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C4c05490f75ba4dc9658e08d6ce7fd451%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923445772493263&sdata=iiux5GTZD%2F7xh56kyGi%2BCImX55GOgP9gdi1Bx6lUEOM%3D&reserved=0
> > >
> > > * For (un)subscribe requests visit
> > >
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmaillist.sys.kth.se%2Fmailman%2Flistinfo%2Fgromacs.org_gmx-users&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C4c05490f75ba4dc9658e08d6ce7fd451%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923445772493263&sdata=fA46t2G3%2FRErO9ephu1d2QoOcWoLadgzG6DkhSG9Los%3D&reserved=0
> or
> > > send a mail to gmx-users-request at gromacs.org.
> > >
> > --
> > Gromacs Users mailing list
> >
> > * Please search the archive at
> >
> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists%2FGMX-Users_List&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C4c05490f75ba4dc9658e08d6ce7fd451%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923445772493263&sdata=hInjXVJw1xyIo23W3Q%2Fnt5UlXy%2Bx5mok7re4cpCopG8%3D&reserved=0
> before
> > posting!
> >
> > * Can't post? Read
> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C4c05490f75ba4dc9658e08d6ce7fd451%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923445772493263&sdata=iiux5GTZD%2F7xh56kyGi%2BCImX55GOgP9gdi1Bx6lUEOM%3D&reserved=0
> >
> > * For (un)subscribe requests visit
> >
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmaillist.sys.kth.se%2Fmailman%2Flistinfo%2Fgromacs.org_gmx-users&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C4c05490f75ba4dc9658e08d6ce7fd451%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923445772493263&sdata=fA46t2G3%2FRErO9ephu1d2QoOcWoLadgzG6DkhSG9Los%3D&reserved=0
> or
> > send a mail to gmx-users-request at gromacs.org.
> >
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists%2FGMX-Users_List&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C4c05490f75ba4dc9658e08d6ce7fd451%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923445772503260&sdata=9bnLD7gtKcDhz%2Fad2gYXB2YzFk1Z4UFyICs1SuFSMV0%3D&reserved=0
> before posting!
>
> * Can't post? Read
> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C4c05490f75ba4dc9658e08d6ce7fd451%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923445772503260&sdata=Vd3yF1au0wtlRoUVfFAaAvLvHWuREe6sa71VMo5TFXE%3D&reserved=0
>
> * For (un)subscribe requests visit
>
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmaillist.sys.kth.se%2Fmailman%2Flistinfo%2Fgromacs.org_gmx-users&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C4c05490f75ba4dc9658e08d6ce7fd451%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923445772503260&sdata=0CjcPnXDU5%2FcBFzTXTVCZ3MzXS1bbcd7KffZRRNu3%2Bk%3D&reserved=0
> or send a mail to gmx-users-request at gromacs.org.
>
More information about the gromacs.org_gmx-users
mailing list