[gmx-users] AMD vs Intel, nvidia 20xx

[Szilárd Páll]

On Wed, Sep 19, 2018 at 11:58 AM Tamas Hegedus <tamas at hegelab.org> wrote:

> Hi,
>
> I am planning to buy 1 or 2 GPU workstations/servers for stand alone
> gromacs and gromacs+plumed (w plumed I can efficiently use only 1 GPU).
> I would like to get some suggestions for the best performance/price
> ratio. More specifically, some info on the correlation between PCI lines
> and performance.
>

Have you considered questions like: What are the type of runs that you plan
to do? Do you need to scale or will you do throughput runs too? Typically
many runs can you/are you planning to do concurrently per node?


> Based on google, gmx mail list, and my experience I think that
> * i9 with 44PCI lines
> * 4 gtx 1080Ti
> are OK and fits the budget.
>
> However, there are several upcoming issues:
>
> 1. I expect the 4 GPUs running at 70% (1PME and 3PP) with 16 cores. I do
> not know if the 44PCI lines are limiting or not?
>

Somewhat but details will matter. With most FFs & settings 2+1 GPUs for
PP/PME is generally the most scaling you get in r2018 -- PME offload was
not optimized for parallel runs. Also, scaling is "costly" and should be
done only if you need to maximize the single-run throughput (and if the
system is large enough).


> 2. 4x8 channel are OK? Should I look for a mother board with a PLX chip
> that makes 64 virtual pci lines? Do you have experience with this type
> of mobo? I have red that the average will be better than 4x8, but others
> say that its makes the system slower.
>

The impact will vary but typically not likely to be very large (haven't
looked recently to be honest); also, if you have at least 2 ranks per GPU
(which is generally the right way to go), the impact will lower.

3. An option might be go with AMD CPUs with higher PCI line, but I do
> not trust AMD when high CPU performance is needed. But this might come
> from the past and I should forget this. Have you used recent AMD
> processors with gromacs with GPU acceleration?
>

I strongly recommend to reconsider that. Both Ryzens and Threadrippers are
very competitive not only in performance/price, but generally get close to
similar Intel CPUs without GPU offload and beat them in performance if GPU
offload is used (as Intel's advantage with wider SIMD is gone).

4. I have the feeling using gmx 2018 that I would not gain performance
> with (physical) 32 cores and 4 GPUs, since 16 already is sufficient and
> the 70% GPU usage is not because waiting for the CPU. I do not know
> about this (how to check this). What is your experience?
>

Again, depends on the type of simulation, but generally >4-5 cores per GPU
with PME offload will not have significant performance advantage. However,
as soon as you do some additional computation that runs on the CPU (e.g.
free energy perturbation or all-bonds constrained with vsites), you might
need a  few more more cores.


> 5. I am afraid to buy more GPU power than necessary.
> It might be better to buy 2 faster and 2 slower GPU or only 3 newer GPUs
> to optimize the system.
> However, I most likely can not get 1070Ti. In addition, is better for
> the system not to run at 100%.
> BUT: there will be new RTX 20xx out and likely I can get lower version
> number for less power consumption and the same computing capacity (e.g.
> 2070 instead of 1080).
> BUT: it seems for me that RTX2070 has much less cores (2304) than 1080Ti
> and 2080Ti has more cores than 1080Ti.
> 1080, 4*3584 cores, 64 PCI lines
> 2080, 3*4352 cores, 48 PCI lines
> However, in the letter case more PP may run on the GPU and again the
> CPU/GPU communication may become the limiting factor.
>

RTX 20xx cards seem to be pricey right now for not much performance gain,
but I've not tried, so can't comment in detail.
1070 Ti, 1080 and 1080 Ti are however good options. You can also consider
buying fewer GPUs per system and adding more if needed; I'd recommend
against mixing GPUs in one system unless you have a single scaled out
workload in mind and you know that e.g. you'll use 2PP + 1PME and the GPU
for PME can be a 1070 next to two 1080 Tis.

--
Szilárd

Thanks for all the opinion and suggestion,
> Tamas
>
> --
> Tamas Hegedus, PhD
> Senior Research Fellow
> Department of Biophysics and Radiation Biology
> Semmelweis University     | phone: (36) 1-459 1500/60233
> Tuzolto utca 37-47        | mailto:tamas at hegelab.org
> Budapest, 1094, Hungary   | http://www.hegelab.org
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
>