[gmx-users] GPU acceleration

Wed Jun 10 01:39:12 CEST 2015

Thanks for your help, but nevermind this whole thing. I incidentally found
out that our cluster had a node with two Tesla cards, which I am now using
with GMX 5.0.5 freshly compiled with the NVIDIA library.
I am getting massive (up to x10) acceleration compared with CPU-only runs.
I think I posted yesterday about a fatal error due to CPU-GPU sync issue,
but seems like I was able to take care of it.
Very impressed here, about to abandon all of my CPU-only runs and switch to
the GPU node as my workhorse.

Alex

On Tue, Jun 9, 2015 at 3:43 PM, Szilárd Páll <pall.szilard at gmail.com> wrote:

> No, it's not. Is this 5.0 or master branch?
> --
> Szilárd
>
>
> On Fri, Jun 5, 2015 at 11:48 PM, Alex <nedomacho at gmail.com> wrote:
> > Thanks for the info. I tried with GMX_EMULATE_GPU=1 preceding the mdrun
> > line, crashed after 100 steps. Is that the expected behavior?
> >
> > On Fri, Jun 5, 2015 at 2:46 PM, Szilárd Páll <pall.szilard at gmail.com>
> wrote:
> >
> >> On Fri, Jun 5, 2015 at 10:39 PM, Alex <nedomacho at gmail.com> wrote:
> >> > In that case, I am a bit lost, as I was sure GMXRC needed
> modification.
> >> > :)). Since I never ran anything on GPU and have an mdrun line that
> looks
> >> > like
> >> >
> >> > mdrun -nt 4 -s <tpr> -c <final structure>
> >>
> >> gmx mdrun -ntomp 4
> >>
> >> No need to do domain decomposition. Even with CPU runs it'll be faster
> >> if you don't.
> >>
> >> > what should be added to trigger the behavior you're suggesting? I
> >> > understand that it will likely blow up the system, but for the heck of
> >> > it...
> >>
> >> those environment variables I mentioned. Plus timer resetting, e.g.
> >> with -resethway if you want reliable timings.
> >>
> >> > I am now looking at the stats and see that (NxN QSTab + LJ) and (NxN
> >> QSTab
> >> > [F]) take up about 86% of all flops in my simulation. Which of those
> get
> >> > offloaded?
> >>
> >> All of that will be offloaded. However, those are theoretical (useful)
> >> flops, the timing table on the bottom is more relevant. Set the
> >> GMX_CYCLE_SUBCOUNTER=ON breakdown of the "Force".
> >>
> >> However, if you just follow the previous emulate+skip non-bondeds
> >> advise, you'll get a run that does the offload except that it skips
> >> calculating - which is useful because you have no GPU to calculate
> >> with.
> >>
> >> > Thank you,
> >> >
> >> > Alex
> >> >
> >> >
> >> > On Fri, Jun 5, 2015 at 2:26 PM, Szilárd Páll <pall.szilard at gmail.com>
> >> wrote:
> >> >
> >> >> Hi Alex,
> >> >>
> >> >> Why do you think GMXRC needs to be changed? You simply need to set
> >> >> those env. vars to trigger the GPU code-path - and you will likely
> >> >> have to freeze the system, otherwise it will probably not run more
> >> >> than a few steps without seeing an explosion. Based on this you'll
> >> >> know how fast you can run with GPU offload, but you can also
> >> >> guesstimate how fast of a GPU you need (based on the PME+Bonded force
> >> >> calculation time + publicly available CUDA kernel performance).
> >> >>
> >> >> Cheers,
> >> >> --
> >> >> Szilárd
> >> >>
> >> >>
> >> >> On Fri, Jun 5, 2015 at 10:13 PM, Alex <nedomacho at gmail.com> wrote:
> >> >> > Hi Szilárd,
> >> >> >
> >> >> > Well, yes, it is an old CPU, but I am actually quite content with
> the
> >> >> > ns/day performance it churns out on all four cores. I'm looking at
> >> >> >
> >> >>
> >>
> http://manual.gromacs.org/documentation/5.1-beta1/user-guide/environment-variables.html
> >> >> > and it is my understanding that GMXRC needs to be changed to
> include
> >> >> those
> >> >> > flags, but no recompiling is required. Is that correct?
> >> >> >
> >> >> > Thanks,
> >> >> >
> >> >> > Alex
> >> >> >
> >> >> >
> >> >> > On Fri, Jun 5, 2015 at 1:52 PM, Szilárd Páll <
> pall.szilard at gmail.com>
> >> >> wrote:
> >> >> >
> >> >> >> Hi Alex,
> >> >> >>
> >> >> >> That's a quite old CPU so it's hard to tell how fast is it, you'll
> >> >> >> have to try. There are ways to "guesstimate" how fast does your
> GPU
> >> >> >> have to be to keep up with your CPU cores (hint search for
> >> >> >> GMX_EMULATE_GPU / GMX_NO_NONBONDED on the list).
> >> >> >>
> >> >> >> In any case, unless your CPU is so slow that it balances well with
> >> the
> >> >> >> 750Ti, I would strongly recommend that you get a GTX 960 which
> will
> >> >> >> cost just a bit more, but it will be 1.6-1.8x faster at computing
> >> >> >> non-bonded interactions.
> >> >> >>
> >> >> >>  On Fri, Jun 5, 2015 at 9:08 PM, Alex <nedomacho at gmail.com>
> wrote:
> >> >> >> > Hi all,
> >> >> >> >
> >> >> >> > We're about to start a bit of a toy project to test
> GPU-accelerated
> >> >> GMX.
> >> >> >> > What I have here is my old workstation (Dell Precision T3500:
> >> >> Quad-core
> >> >> >> >  Xeon W3530 @ 2.8 GHz, 12 gigs of RAM) running Linux Mint 17.1.
> The
> >> >> plan
> >> >> >> is
> >> >> >> > to get something like GeForce GTX 750 (
> >> >> >> >
> >> >> >>
> >> >>
> >>
> http://www.amazon.com/GeForce-750Ti-GDDR5-Graphics-GTX750TI-OC-2GD5/dp/B00IB9P1KG/ref=sr_1_3?ie=UTF8&qid=1433529993&sr=8-3&keywords=Nvidia+GTX750
> >> >> >> > )
> >> >> >> > and maybe a new PSU.
> >> >> >> >
> >> >> >> > Questions:
> >> >> >> >
> >> >> >> > 1. With a card like this, how much acceleration (very
> >> approximately)
> >> >> can
> >> >> >> we
> >> >> >> > expect in a "standard" simulation (protein+lipid+water, PME
> >> >> >> electrostatics)?
> >> >> >>
> >> >> >> In PME simulations you should see about 3x speedup, more if you
> are
> >> >> >> lucky and possibly a little less if you are not so lucky with the
> >> >> >> computational workload. What FF are you using?
> >> >> >>
> >> >> >> > 2. Should we put any extra considerations into cooling? We
> expect
> >> >> >> week-long
> >> >> >> > runs, and I am not sure these cards have this type of hardcore
> >> gaming
> >> >> in
> >> >> >> > mind. :)
> >> >> >>
> >> >> >> I don't think so, but it depends how stuffed or dusty is that Dell
> >> >> >> workstation. These Maxwell generation (96x) GPUs run quite cool.
> >> >> >>
> >> >> >> > Any additional comments are highly appreciated. Please keep in
> mind
> >> >> that
> >> >> >> > this is a toy project to utilize an old workstation.
> >> >> >>
> >> >> >> There is a GPU fan speed issue with desktop NVIDIA cards, the fan
> >> >> >> rotation is limited to 60% and cards can overheat. You can force
> the
> >> >> >> fan speed higher using nvidia-settings (you'll need an X server
> >> >> >> running, though). For more details you can check slide 28 here:
> >> >> >> https://goo.gl/7DnSri
> >> >> >>
> >> >> >> Cheers,
> >> >> >> --
> >> >> >> Szilárd
> >> >> >>
> >> >> >> > Thank you,
> >> >> >> >
> >> >> >> > Alex
> >> >> >> > --
> >> >> >> > Gromacs Users mailing list
> >> >> >> >
> >> >> >> > * Please search the archive at
> >> >> >> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List
> before
> >> >> >> posting!
> >> >> >> >
> >> >> >> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >> >> >> >
> >> >> >> > * For (un)subscribe requests visit
> >> >> >> >
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users
> >> or
> >> >> >> send a mail to gmx-users-request at gromacs.org.
> >> >> >> --
> >> >> >> Gromacs Users mailing list
> >> >> >>
> >> >> >> * Please search the archive at
> >> >> >> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List
> before
> >> >> >> posting!
> >> >> >>
> >> >> >> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >> >> >>
> >> >> >> * For (un)subscribe requests visit
> >> >> >>
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users
> >> or
> >> >> >> send a mail to gmx-users-request at gromacs.org.
> >> >> > --
> >> >> > Gromacs Users mailing list
> >> >> >
> >> >> > * Please search the archive at
> >> >> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> >> >> posting!
> >> >> >
> >> >> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >> >> >
> >> >> > * For (un)subscribe requests visit
> >> >> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users
> or
> >> >> send a mail to gmx-users-request at gromacs.org.
> >> >> --
> >> >> Gromacs Users mailing list
> >> >>
> >> >> * Please search the archive at
> >> >> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> >> >> posting!
> >> >>
> >> >> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >> >>
> >> >> * For (un)subscribe requests visit
> >> >> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users
> or
> >> >> send a mail to gmx-users-request at gromacs.org.
> >> >>
> >> > --
> >> > Gromacs Users mailing list
> >> >
> >> > * Please search the archive at
> >> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> >> posting!
> >> >
> >> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >> >
> >> > * For (un)subscribe requests visit
> >> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> >> send a mail to gmx-users-request at gromacs.org.
> >> --
> >> Gromacs Users mailing list
> >>
> >> * Please search the archive at
> >> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> >> posting!
> >>
> >> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >>
> >> * For (un)subscribe requests visit
> >> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> >> send a mail to gmx-users-request at gromacs.org.
> >>
> > --
> > Gromacs Users mailing list
> >
> > * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
> >
> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >
> > * For (un)subscribe requests visit
> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.