[gmx-users] Gromacs 4.5.1 on 48 core magny-cours AMDs
pronk at cbr.su.se
Thu Oct 21 16:44:23 CEST 2010
Thanks for the information; the OpenMPI recommendation is probably because OpenMPI goes to great lengths trying to avoid process migration. The numactl doesn't prevent migration as far as I can tell: it controls where memory gets allocated if it's NUMA.
For gromacs the setting should of course always be
As far as thread migration goes: that might also be a kernel issue. If I remember correctly, this issue has only recently been addressed in the Linux kernel. I'll check whether there's anything we can do specifically for Gromacs.
On 21 Oct 2010, at 14:04 , Carsten Kutzner wrote:
> Hi Sander,
> On Oct 21, 2010, at 12:27 PM, Sander Pronk wrote:
>> Hi Carsten,
>> As Berk noted, we haven't had problems on 24-core machines, but quite frankly I haven't looked at thread migration.
> I did not have any problems on 32-core machines as well, only on 48-core ones.
>> Currently, the wait states actively yield to the scheduler, which is an opportunity for the scheduler to re-assign threads to different cores. I could set harder thread affinity but that could compromise system responsiveness (when running mdrun on a desktop machine without active yielding, the system slows down noticeably).
>> One thing you could try is to turn on the THREAD_MPI_WAIT_FOR_NO_ONE option in cmake. That turns off the yielding which might change the migration behavior.
> I will try that, thanks!
>> BTW What do you mean with bad performance, and how do you notice thread migration issues?
> A while ago I benchmarked a ~80,000 atom test system (membrane+channel+water, 2 fs time
> step, cutoffs @ 1 nm) on a 48-core 1.9 GHz AMD node. My first try gave a lousy 7.5 ns/day
> using Gromacs 4.0.7 and IntelMPI. According to AMD, parallel applications should be
> run under control of numactl to be compliant to the new memory hierarchy. Also, they
> suggest using OpenMPI rather than other MPI libs. With OpenMPI and numactl - which pins
> the processes to the cores - the performance was nearly doubled to 14.3 ns/day. Using
> Gromacs 4.5 I got 14.0 ns/day with OpenMPI+numactl and 15.2 ns/day with threads (here no
> pinning was necessary for the threaded version!)
> Now on another machine with identical hardware (but another Linux) I get 4.5.1 timings that
> vary a lot (see g_tune_pme snippet below) even between identical runs. One run actually approaches
> the expected 15 ns/day, while the others with also 20 PME-only nodes) do not. I cannot be shure
> that thread migration is the problem here, but correct pinning might be necessary here.
> g_tune_pme output snippet for mdrun with threads:
> Benchmark steps : 1000
> dlb equilibration steps : 100
> Repeats for each test : 4
> No. scaling rcoulomb nkx nky nkz spacing rvdw tpr file
> 0 -input- 1.000000 90 88 80 0.119865 1.000000 ./Aquaporin_gmx4_bench00.tpr
> Individual timings for input file 0 (./Aquaporin_gmx4_bench00.tpr):
> PME nodes Gcycles ns/day PME/f Remark
> 24 1804.442 8.736 1.703 OK.
> 24 1805.655 8.730 1.689 OK.
> 24 1260.351 12.505 0.647 OK.
> 24 1954.314 8.064 1.488 OK.
> 20 1753.386 8.992 1.960 OK.
> 20 1981.032 7.958 2.190 OK.
> 20 1344.375 11.721 1.180 OK.
> 20 1103.340 14.287 0.896 OK.
> 16 1876.134 8.404 1.713 OK.
> 16 1844.111 8.551 1.525 OK.
> 16 1757.414 8.972 1.845 OK.
> 16 1785.050 8.833 1.208 OK.
> 0 1851.645 8.520 - OK.
> 0 1871.955 8.427 - OK.
> 0 1978.357 7.974 - OK.
> 0 1848.515 8.534 - OK.
> -1( 18) 1926.202 8.182 1.453 OK.
> -1( 18) 1195.456 13.184 0.826 OK.
> -1( 18) 1816.765 8.677 1.853 OK.
> -1( 18) 1218.834 12.931 0.884 OK.
>> On 21 Oct 2010, at 12:03 , Carsten Kutzner wrote:
>>> does anyone have experience with AMD's 12-core Magny-Cours
>>> processors? With 48 cores on a node it is essential that the processes
>>> are properly pinned to the cores for optimum performance. Numactl
>>> can do this, but at the moment I do not get good performance with
>>> 4.5.1 and threads, which still seem to be migrating around.
> gmx-users mailing list gmx-users at gromacs.org
> Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/Search before posting!
> Please don't post (un)subscribe requests to the list. Use the
> www interface or send it to gmx-users-request at gromacs.org.
> Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
More information about the gromacs.org_gmx-users