[gmx-users] optimum number of OpenMP threads for simulation

[Irem Altan]

Hi,

I’m trying to set up a simulation on SDSC’s Comet, using half of the resources on a node. A node contains 24 CPU cores, and 4 GPUs. I want to use 12 cores and 2 GPUs. The resources on the node get shared, meaning that the unused 12 cores and 2 GPUs are used by another user.

I tried the following setup:

#!/bin/bash
#SBATCH --job-name="md_u0"
#SBATCH --output=out0.dat
#SBATCH --ntasks-per-node=12
#SBATCH -c 1
#SBATCH --no-requeue
#SBATCH -p gpu-shared
#SBATCH --gres=gpu:2
#SBATCH -t 2-00:00:00
export OMP_NUM_THREADS=6
module unload intel
module load gromacs
gmx_mpi grompp -f md_umbrella.mdp -c npt0.gro -t npt0.cpt -p topol.top -n index.ndx -maxwarn 1 -o umbrella0.tpr
ibrun gmx_mpi mdrun -v -deffnm umbrella0 -pf pullf-umbrella0.xvg -px pullx-umbrella0.xvg -pin on

which resulted in the following note:

NOTE: Oversubscribing the CPU, will not pin threads

and the simulation was excruciatingly slow. It normally takes about a day, but the output file said

imb F 10% step 10400, will finish Mon Mar  6 20:49:50 2017

When I set OMP_NUM_THREADS=1, Gromacs gives an error saying the optimum number is between 2 and 6, and stops execution. What is the correct value to use for this option? I can’t really do this by trial and error because the job waits on the queue for over a day.

Best,
Irem