[gmx-users] Worse GROMACS performance with better specs?

Sun Feb 25 14:14:48 CET 2018

BS”D

 Someone has pointed out to me that the “Silver” line is not meant for HPC.  For HPC, you need to go with the “Gold” series, even if you don’t want 4 sockets.

The difference presumably lies is the fact that the Gold has 2 FMA units, and the Silver series has 1.

Harry

On 20 Feb 2018, at 6:42 PM, Szilárd Páll <pall.szilard at gmail.com<mailto:pall.szilard at gmail.com>> wrote:

On Fri, Jan 12, 2018 at 2:35 AM, Jason Loo Siau Ee
<JasonSiauEe.Loo at taylors.edu.my<mailto:JasonSiauEe.Loo at taylors.edu.my>> wrote:
Dear Carsten,

Look's like we're seeing the same thing here, but only when using gcc 4.5.3:

Original performance (gcc 5.3.1, AVX512, no hwloc support): 49 ns/day

With hwloc support:
gcc 4.5.3, AVX2_256 = 67 ns/day
gcc 4.5.3, AVX512 = can't compile
gcc 5.3.1, AVX2_256 = 36 ns/day
gcc 5.3.1, AVX512 = 60 ns/day

At the very least it's now comparable to my previous workstation. Will have to try with the gromacs 2018 next.

Have you tried? Any feedback?

I'd also recommend using gcc >=6.

Cheers for the info.

Jason

------------------------------

Message: 2
Date: Wed, 10 Jan 2018 08:39:22 +0000
From: "Kutzner, Carsten" <ckutzne at gwdg.de<mailto:ckutzne at gwdg.de>>
To: "<gmx-users at gromacs.org<mailto:gmx-users at gromacs.org>> GROMACS users" <gmx-users at gromacs.org<mailto:gmx-users at gromacs.org>>
Subject: Re: [gmx-users] Worse GROMACS performance with better specs?
Message-ID: <816DEA54-1E01-42AC-A78A-3680FDEF097E at gwdg.de<mailto:816DEA54-1E01-42AC-A78A-3680FDEF097E at gwdg.de>>
Content-Type: text/plain; charset="us-ascii"

Dear Jason,

1.)
we have observed a similar behavior comparing Intel Silver 4114 against E5-2630v4
processors in a server with one GTX 1080Ti. Both CPUs have 10 cores and run at
2.2 GHz. Using our standard benchmark systems (see https://arxiv.org/abs/1507.00898)
we were able to get 74.4 ns/day on the E5 but only 65.7 ns/day on the Silver machine
for the MEM (80k atoms), and 4.4 vs. 4.2 ns/day for the RIB (2M atoms) system.

Although the 4114 supports AVX512 SIMD instructions, compiling with AVX2_256
yielded higher performance (this is already included in the above numbers, so
all benchmarks were run with the exact same mdrun executable).

2.)
We recently bought a bunch of workstations with 2x Gold 6146 CPUs and 2x GTX1080Ti GPUs
and were very surprised to find that initial GROMACS 2016 performances differ a
lot between the identical machines (some were up to 40% slower).

This was the reported difference in the md.log files:
- mdrun logs show Hardware Topology: Basic versus Hardware Topology: Only logical processor count

For some reason, the logical processor count was reported in a mixed-up way
for the slower machines so that mdrun is unable to correctly determine the
hardware topology. We solved this CPU problem by installing GROMACS with hwloc
support. Then performances were equal over all 6146 nodes.

Best,
 Carsten

On 10. Jan 2018, at 07:45, Jason Loo Siau Ee <JasonSiauEe.Loo at taylors.edu.my<mailto:JasonSiauEe.Loo at taylors.edu.my>> wrote:

Dear gmx users,

I recently purchased a second GPU workstation and tried compiling GROMACS on it, but despite the better (and more expensive) specs the performance is significantly worse on my test system. To test things I standardized the version (2016.4). Some details below:

Workstation 1:

2 x Intel Xeon E5-2680v4 (14 cores), 2 x GTX 1080

gmx mdrun -ntmpi 8 -ntomp 7 -gpu_id 00001111

Performance: 61ns/day

Workstation 2:

2 x Intel Xeon Gold 6126 (12 cores), 2 x GTX 1080Ti

gmx mdrun -ntmpi 8 -ntomp 6 -gpu_id 00001111

Performance: 49ns/day

I'm guessing it's an issue during compilation but I can't figure it out. I wouldn't claim to have any knowledge about how GROMACS interacts with the hardware, so some observations as below (not sure idea which is actually relevant):

- Compilation command for both: cmake .. -DGMX_BUILD_OWN_FFTW=ON  -DREGRESSIONTEST_DOWNLOAD=ON  -DGMX_GPU=ON  -DCMAKE_INSTALL_PREFIX=/opt/gromacs

- When compiling on Workstation 2 I originally got a CMake error "Cannot find AVX512F compiler flag". Updated my gcc version to 5.3.1 to solve this.

- Some regression tests fail for Workstation 2 during compilation: 4 -FTUnitTest (SEGFAULT), 16 - CorrelationTest

- mdrun logs show Hardware Topology: Basic versus Hardware Topology: Only logical processor count

- Running CPU-only (export CUDA_VISIBLE_DEVICES="") I get 21ns/day versus 23 ns/day , so the CPUs in Workstation 2 are definitely faster.

- Upgraded both to 2018.rc1 (used cmake3) I get a regression test fail for Workstation 1 (9 - GPUUtilUnitTest) and Workstation 2 (8 - FFTUnitTests, 9 - GPUUtilsUnitTest, 26 - CorrelationTest). Performance is 66.5ns/day versus 51.95ns/day. The GPU load actually looks similar to previous versions (~70% for Workstation 1 and 50-60% for Workstation 2). I actually got the best performance with Workstation 1 running 2016.1 (69ns/day).

Any help on how I can optimize performance on Workstation 2 would be appreciated. If there are certain files that would be helpful let me know and I'll send a link.

Cheers,
Jason

Jason Loo

PhD, MPharm, RPh

Lecturer

School of Pharmacy

Faculty of Health and Medical Sciences

Taylor's University

This message (including any attachments) is intended only for
the use of the individual or entity to which it is addressed and
may contain information that is non-public, proprietary,
privileged, confidential, and exempt from disclosure under
applicable law or may constitute as attorney work product.
If you are not the intended recipient, you are hereby notified
that any use, dissemination, distribution, or copying of this
communication is strictly prohibited. If you have received this
communication in error, notify us immediately by telephone and
(i) destroy this message if a facsimile or (ii) delete this message
immediately if this is an electronic communication.

Thank you.
--
Gromacs Users mailing list

* Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!

* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists

* For (un)subscribe requests visit
https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a mail to gmx-users-request at gromacs.org<mailto:gmx-users-request at gromacs.org>.

--
Dr. Carsten Kutzner
Max Planck Institute for Biophysical Chemistry
Theoretical and Computational Biophysics
Am Fassberg 11, 37077 Goettingen, Germany
Tel. +49-551-2012313, Fax: +49-551-2012302
http://www.mpibpc.mpg.de/grubmueller/kutzner
http://www.mpibpc.mpg.de/grubmueller/sppexa

This message (including any attachments) is intended only for
the use of the individual or entity to which it is addressed and
may contain information that is non-public, proprietary,
privileged, confidential, and exempt from disclosure under
applicable law or may constitute as attorney work product.
If you are not the intended recipient, you are hereby notified
that any use, dissemination, distribution, or copying of this
communication is strictly prohibited. If you have received this
communication in error, notify us immediately by telephone and
(i) destroy this message if a facsimile or (ii) delete this message
immediately if this is an electronic communication.

Thank you.
--
Gromacs Users mailing list

* Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!

* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists

* For (un)subscribe requests visit
https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a mail to gmx-users-request at gromacs.org.
--
Gromacs Users mailing list

* Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!

* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists

* For (un)subscribe requests visit
https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a mail to gmx-users-request at gromacs.org<mailto:gmx-users-request at gromacs.org>.

--------------------------------------------------------------------
Harry M. Greenblatt
Associate Staff Scientist
Dept of Structural Biology           harry.greenblatt at weizmann.ac.il<../../owa/redir.aspx?C=QQgUExlE8Ueu2zs5OGxuL5gubHf97c8IyXxHOfOIqyzCgIQtXppXx1YBYaN5yrHbaDn2xAb8moU.&URL=mailto%3aharry.greenblatt%40weizmann.ac.il>
Weizmann Institute of Science        Phone:  972-8-934-6340
234 Herzl St.                        Facsimile:   972-8-934-3361
Rehovot, 7610001
Israel