[gmx-developers] Double precision on GPU's

[Szilárd Páll]

On Wed, Jul 24, 2019 at 4:17 PM Erik Lindahl <erik.lindahl at gmail.com> wrote:
>
> Hi,
>
> Just to add some minor things to what Berk already said:
>
> 1. With current nvidia cards, for the Tesla line the double precision floprate is 50% of single. However, for the consumer cards it's only 1/8-1/12th. In practice the latter is so low that it's often better to run on the CPU instead.

Note that it's FP64 throughput is 32x lower than FP32.

Additionally, atomic operations are not supported on FP64, so that
would further slow down many of the algorithms.

> This is the main reason we haven't bothered with it yet - most academic labs use GeForce :-)
>
> 2. You are likely well aware of this, but for users in general another problem with skipping thermostats for complex biophysical systems is that the potential energy often goes down as the system relaxes - which will make the temperature go up in an NVE ensemble.
>
> 3. Having said that, I think it should be a fairly straightforward minor project for someone. It's probably mostly a matter of changing variable types so they compile to double if GMX_DOUBLE is set.

Additionally at least the following will also be necessary (either to
make the code work or to get meaningful performance):
- reimplement reductions (shuffle 64-bit)
- implement/assess atomics for FP64 (not hard, but it will be slow)
- FP64 requires 2x the storage so some of the local/shared memory use
needs to be reconsidered

Considering that the performance characteristics will greatly change,
one might also want to revise the algorithmic setup; if a considerable
performance drop is a not a concern this step can however be omitted
-- and one can also omit optimizations in the above (e.g. by using
unotimized reductions).

--
Szilárd

>
> Cheers,
>
> Erik
>
> Erik Lindahl <erik.lindahl at scilifelab.se>
> Professor of Biophysics
> Science for Life Laboratory
> Stockholm University & KTH
> Office (SciLifeLab): +46 8 524 81567
> Cell (Sweden): +46 73 4618050
> Cell (US): +1 (650) 924 7674
>
>
>
> On 24 Jul 2019, at 15:25, Berk Hess <hess at kth.se> wrote:
>
> Hi,
>
> The energy conservation check can still be done with a thermostat, as GROMACS keeps track of the amount of energy the thermostat adds to or removes from the system.
>
> I don't understand what you mean exactly with "energy balance". Either you are interested in energy dissipation between different parts of the system, in which case you often can not use a thermostat, or you are not and then you can, and probably should, use a thermostat (and still keep track of energy conservation).
>
> Cheers,
>
> Berk
>
> On 2019-07-23 21:19 , James wrote:
>
> Hi Berk,
>
> Thank you for the information. I prefer not to use thermostats, at least when trying to get quantitative values, because (and I may be misunderstanding thermostats, but since changing the temperature changes the energy, I assume this is true) then I cannot use "energy in = energy out" as a sanity check. I'm more interested in energy balance than in maintaining a given temperature.
>
> If I were to be able to get the double precision GPU stuff coded, would the team be willing to maintain it? Or, since I don't know the code, perhaps a better question is: Does someone who is already familiar with the relevant code have time to do this as a side project (with compensation)? If anyone is interested, please feel free to contact me off list.
>
> Thanks,
> James
>
> =======================================
>
> Hi,
>
> IIRC all Nvidia Tesla cards have always had double precision, at half
> the throughput of single precision. But there are very few cases where
> double precision is needed. Energy drift in single precision is never an
> issue, unless you really can not use a thermostat.
>
> But having said that, making the GPU code, either CUDA or OpenCL work in
> double precision is probably not much effort. But making it work
> efficiently requires optimizing several algorithmic parameters and maybe
> changing the arrangement of some data in the different GPU memory levels.
>
> Cheers,
>
> Berk
>
> On 7/22/19 10:10 PM, James wrote:
> > Hi,
> >
> > My apologies if this question has been previously discussed. I just
> > joined the list and all I know is that from reading the docs and
> > release comments, writing code for double precision on GPU's is not a
> > priority.
> >
> > However, I believe all recent upper-end Nvidia cards have native
> > double precision (which was not true several generations ago). So, you
> > don't have to have a real "scientific computing" GPU to take advantage
> > of this -- most people probably already have the hardware. Still, I
> > understand that most people do not need/want to run double precision.
> > But, some do (and you have to if you are concerned with conservation
> > of energy -- the energy drift in single precision is substantial).
> >
> > So, I would like to ask what the level of effort to do this is
> > believed to be? Would it require a lot of new code, or would it be
> > porting the single precision code to double precision?
> >
> > Sincerely,
> > James
> >
>
>
>
>
> --
> Gromacs Developers mailing list
>
> * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List before posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers or send a mail to gmx-developers-request at gromacs.org.
>
> --
> Gromacs Developers mailing list
>
> * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List before posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers or send a mail to gmx-developers-request at gromacs.org.