[gmx-users] Gromacs 4.6.7 with MPI and OpenMP

[Malcolm Tobias]

Szilárd,

On Friday 08 May 2015 20:25:09 Szilárd Páll wrote:
> > I wouldn't expect the CPUSETs to be problematic, I've been using them with Gromacs for over a decade now ;-)
> 
> Thread affinity setting within mdrun has been employed since v4.6 and
> we do it on a per-thread basis and not doing it can leadto pretty
> severe performance degradation when using multi-threading. Depending
> on the Linux kernel, OS jitter, and type/speed/scale of the simulation
> even MPI-only runs will see a benefit from correct affinity settings.
> 
> Hints:
> - some useful mdrun command line arguments: "-pin on", "-pinoffset N"
> (-pinstride N)
> - more details:
> http://www.gromacs.org/Documentation/Acceleration_and_parallelization

Understood.  What's weird is that without pin'ing, it seems to start the 4 threads all on the same CPU-core.

Based on some suggestions by Mark Abraham, I think I might be seeing where the problem is coming from: If I call omp_get_num_procs within a OpenMP thread that is a part of an MPI process, it's only reporting 1 processor.  This seems to happen with one of my MPI implementations, but not others.  I wonder if this would affect the thread affinity: Since the threads only think there is one core, would they all try to run on it?

> > Weird.  I wonder if anyone else has experience using pin'ing with CPUSETs?
> 
> What is your goal with using CPUSETs? Node sharing?

Correct.  While it might be possible to see the cores that have been assigned to the job and do the correct 'pin setting' it would probably be ugly.

Cheers,
Malcolm

 
-- 
Malcolm Tobias
314.362.1594