[gmx-users] Worse GROMACS performance with better specs?

[Jason Loo Siau Ee]

Dear gmx users,



I recently purchased a second GPU workstation and tried compiling GROMACS on it, but despite the better (and more expensive) specs the performance is significantly worse on my test system. To test things I standardized the version (2016.4). Some details below:



Workstation 1:

2 x Intel Xeon E5-2680v4 (14 cores), 2 x GTX 1080

gmx mdrun -ntmpi 8 -ntomp 7 -gpu_id 00001111

Performance: 61ns/day



Workstation 2:

2 x Intel Xeon Gold 6126 (12 cores), 2 x GTX 1080Ti

gmx mdrun -ntmpi 8 -ntomp 6 -gpu_id 00001111

Performance: 49ns/day



I'm guessing it's an issue during compilation but I can't figure it out. I wouldn't claim to have any knowledge about how GROMACS interacts with the hardware, so some observations as below (not sure idea which is actually relevant):



- Compilation command for both: cmake .. -DGMX_BUILD_OWN_FFTW=ON  -DREGRESSIONTEST_DOWNLOAD=ON  -DGMX_GPU=ON  -DCMAKE_INSTALL_PREFIX=/opt/gromacs



- When compiling on Workstation 2 I originally got a CMake error "Cannot find AVX512F compiler flag". Updated my gcc version to 5.3.1 to solve this.



- Some regression tests fail for Workstation 2 during compilation: 4 -FTUnitTest (SEGFAULT), 16 - CorrelationTest



- mdrun logs show Hardware Topology: Basic versus Hardware Topology: Only logical processor count



- Running CPU-only (export CUDA_VISIBLE_DEVICES="") I get 21ns/day versus 23 ns/day , so the CPUs in Workstation 2 are definitely faster.



- Upgraded both to 2018.rc1 (used cmake3) I get a regression test fail for Workstation 1 (9 - GPUUtilUnitTest) and Workstation 2 (8 - FFTUnitTests, 9 - GPUUtilsUnitTest, 26 - CorrelationTest). Performance is 66.5ns/day versus 51.95ns/day. The GPU load actually looks similar to previous versions (~70% for Workstation 1 and 50-60% for Workstation 2). I actually got the best performance with Workstation 1 running 2016.1 (69ns/day).



Any help on how I can optimize performance on Workstation 2 would be appreciated. If there are certain files that would be helpful let me know and I'll send a link.



Cheers,
Jason







Jason Loo

PhD, MPharm, RPh

Lecturer

School of Pharmacy

Faculty of Health and Medical Sciences

Taylor's University



This message (including any attachments) is intended only for
the use of the individual or entity to which it is addressed and
may contain information that is non-public, proprietary,
privileged, confidential, and exempt from disclosure under
applicable law or may constitute as attorney work product.
If you are not the intended recipient, you are hereby notified
that any use, dissemination, distribution, or copying of this
communication is strictly prohibited. If you have received this
communication in error, notify us immediately by telephone and
(i) destroy this message if a facsimile or (ii) delete this message
immediately if this is an electronic communication.

Thank you.