[gmx-users] Gromacs 2019.2 on Power9 + Volta GPUs (building and running)
Alex
nedomacho at gmail.com
Thu May 2 00:33:06 CEST 2019
Of course, i am not. This is the EM. ;)
On Wed, May 1, 2019, 4:30 PM Kevin Boyd <kevin.boyd at uconn.edu> wrote:
> Hi,
>
> In addition to what Mark said (and I've also found pinning to be critical
> for performance), you're also not using the GPUs with "-pme cpu -nb cpu".
>
> Kevin
>
> On Wed, May 1, 2019 at 5:56 PM Alex <nedomacho at gmail.com> wrote:
>
> > Well, my experience so far has been with the EM, because the rest of the
> > script (with all the dynamic things) needed that to finish. And it
> > "finished" by hitting the wall. However, your comment does touch upon
> what
> > to do with thread pinning and I will try to set '-pin on' throughout to
> see
> > if things make a difference for the better. I am less confident about
> > setting strides because it is unclear what the job manager provides in
> > terms of the available core numbers. I will play around some more and
> > report here.
> >
> > Thanks!
> >
> > Alex
> >
> > On Wed, May 1, 2019 at 3:49 PM Mark Abraham <mark.j.abraham at gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > As with x86, GROMACS uses SIMD intrinsics on POWER9 and is thus fairly
> > > insensitive to the compiler's vectorisation abilities. GCC is the only
> > > compiler we've tested, as xlc can't compile simple C++11. As
> everywhere,
> > > you should use the latest version of gcc, as IBM spent quite some years
> > > landing improvements for POWER9.
> > >
> > > EM is useless as a performance indicator of a dynamical simulation,
> avoid
> > > that - it runs serial code much much more often.
> > >
> > > Your run deliberately didn't fill the available cores, so just like on
> > x86,
> > > mdrun will leave the thread affinity handling to the environment, which
> > is
> > > often a path to bad performance. So, if you plan on doing that often,
> > > you'll want to check out the mdrun performance guide docs about the
> mdrun
> > > -pin and related options.
> > >
> > > Mark
> > >
> > >
> > > On Wed., 1 May 2019, 23:21 Alex, <nedomacho at gmail.com> wrote:
> > >
> > > > Hi all,
> > > >
> > > > Our institution decided to be all fancy, so now we have a bunch of
> > Power9
> > > > nodes, each with 80 cores + 4 Volta GPUs. Stuff is managed by slurm.
> > > Today
> > > > I did a simple EM ('gmx mdrun -ntomp 4 -ntmpi 4 -pme cpu -nb cpu')
> and
> > > the
> > > > performance is abysmal, I would guess 100 times slower than on
> anything
> > > > I've ever seen before.
> > > >
> > > > Our admin person emailed me the following:
> > > > "-- it would not surprise me if the GCC compilers were relatively bad
> > at
> > > > taking advantage of POWER9 vectorization, they're likely optimized
> for
> > > > x86_64 vector stuff like SSE and AVX operations. This was an issue
> in
> > > the
> > > > build, I selected "-DGMX_SIMD=IBM_VSX" for the config, but according
> to
> > > my
> > > > notes, that was part of an attempt to fix the "unimplemented SIMD"
> > error
> > > > that was dogging me at first, and/but which was eventually cleared by
> > > > switching to gcc-6."
> > > >
> > > > Does anyone have any comments/suggestions on building and running GMX
> > on
> > > > Power9?
> > > >
> > > > Thank you,
> > > >
> > > > Alex
> > > > --
> > > > Gromacs Users mailing list
> > > >
> > > > * Please search the archive at
> > > >
> >
> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists%2FGMX-Users_List&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C4c05490f75ba4dc9658e08d6ce7fd451%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923445772493263&sdata=hInjXVJw1xyIo23W3Q%2Fnt5UlXy%2Bx5mok7re4cpCopG8%3D&reserved=0
> > before
> > > > posting!
> > > >
> > > > * Can't post? Read
> >
> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C4c05490f75ba4dc9658e08d6ce7fd451%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923445772493263&sdata=iiux5GTZD%2F7xh56kyGi%2BCImX55GOgP9gdi1Bx6lUEOM%3D&reserved=0
> > > >
> > > > * For (un)subscribe requests visit
> > > >
> >
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmaillist.sys.kth.se%2Fmailman%2Flistinfo%2Fgromacs.org_gmx-users&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C4c05490f75ba4dc9658e08d6ce7fd451%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923445772493263&sdata=fA46t2G3%2FRErO9ephu1d2QoOcWoLadgzG6DkhSG9Los%3D&reserved=0
> > or
> > > > send a mail to gmx-users-request at gromacs.org.
> > > >
> > > --
> > > Gromacs Users mailing list
> > >
> > > * Please search the archive at
> > >
> >
> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists%2FGMX-Users_List&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C4c05490f75ba4dc9658e08d6ce7fd451%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923445772493263&sdata=hInjXVJw1xyIo23W3Q%2Fnt5UlXy%2Bx5mok7re4cpCopG8%3D&reserved=0
> > before
> > > posting!
> > >
> > > * Can't post? Read
> >
> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C4c05490f75ba4dc9658e08d6ce7fd451%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923445772493263&sdata=iiux5GTZD%2F7xh56kyGi%2BCImX55GOgP9gdi1Bx6lUEOM%3D&reserved=0
> > >
> > > * For (un)subscribe requests visit
> > >
> >
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmaillist.sys.kth.se%2Fmailman%2Flistinfo%2Fgromacs.org_gmx-users&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C4c05490f75ba4dc9658e08d6ce7fd451%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923445772493263&sdata=fA46t2G3%2FRErO9ephu1d2QoOcWoLadgzG6DkhSG9Los%3D&reserved=0
> > or
> > > send a mail to gmx-users-request at gromacs.org.
> > >
> > --
> > Gromacs Users mailing list
> >
> > * Please search the archive at
> >
> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists%2FGMX-Users_List&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C4c05490f75ba4dc9658e08d6ce7fd451%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923445772503260&sdata=9bnLD7gtKcDhz%2Fad2gYXB2YzFk1Z4UFyICs1SuFSMV0%3D&reserved=0
> > before posting!
> >
> > * Can't post? Read
> >
> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C4c05490f75ba4dc9658e08d6ce7fd451%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923445772503260&sdata=Vd3yF1au0wtlRoUVfFAaAvLvHWuREe6sa71VMo5TFXE%3D&reserved=0
> >
> > * For (un)subscribe requests visit
> >
> >
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmaillist.sys.kth.se%2Fmailman%2Flistinfo%2Fgromacs.org_gmx-users&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C4c05490f75ba4dc9658e08d6ce7fd451%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923445772503260&sdata=0CjcPnXDU5%2FcBFzTXTVCZ3MzXS1bbcd7KffZRRNu3%2Bk%3D&reserved=0
> > or send a mail to gmx-users-request at gromacs.org.
> >
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
>
More information about the gromacs.org_gmx-users
mailing list