[gmx-users] Performance issues with Gromacs 2020 on GPUs - slower than 2019.5

[Andreas Baer]

Hello,

from a set of benchmark tests with large systems using Gromacs versions 
2019.5 and 2020, I obtained some unexpected results:
With the same set of parameters and the 2020 version, I obtain a 
performance that is about 2/3 of the 2019.5 version. Interestingly, 
according to nvidia-smi, the GPU usage is about 20% higher for the 2020 
version.
Also from the log files it seems, that the 2020 version does the 
computations more efficiently, but spends so much more time waiting, 
that the overall performance drops.

Some background info on the benchmarks:
- System contains about 2.1 million atoms.
- Hardware: 2x Intel Xeon Gold 6134 („Skylake“) @3.2 GHz = 16 cores + 
SMT; 4x NVIDIA Tesla V100
   (similar results with less significant performance drop (~15%) on a 
different machine: 2 or 4 nodes with each [2x Intel Xeon 2660v2 („Ivy 
Bridge“) @ 2.2GHz = 20 cores + SMT; 2x NVIDIA Kepler K20])
- Several options for -ntmpi, -ntomp, -bonded, -pme are used to find the 
optimal set. However the performance drop seems to be persistent for all 
such options.

Two representative log files are attached.
Does anyone have an idea, where this drop comes from, and how to choose 
the parameters for the 2020 version to circumvent this?

Regards,
Andreas