[gmx-users] optimum number of OpenMP threads for simulation

[Mark Abraham]

Hi,

The simplest thing to do is to run another replicate of your simulation
(e.g. different velocities) using mdrun -multi, filling the whole node, and
now mdrun will take care of all the details. Even if you ignore the
replicate, this can be more efficient than not managing anything. ;-)

Otherwise, if your cluster implements node sharing, then they have the
burden of documenting how they expect users to be able to keep their jobs
apart (or explaining why they want the resources wasted ;-) ). GROMACS is
extremely sensitive to locality of work, and this is ruined both if GROMACS
threads are moved around by the kernel, or if other jobs on the node are
moved onto cores GROMACS is using. You also want to make sure GROMACS
threads are close together, e.g. on the same CPU socket.

Further, you should note that "tasks per node" means MPI ranks, and
OMP_NUM_THREADS means thread per rank, so your script is asking for 12*6=72
threads, which is not what you intended. You should try 2*6 or maybe 4*3 if
your system is large enough.

More background information is at
http://manual.gromacs.org/documentation/2016.1/user-guide/mdrun-performance.html

Mark

On Tue, Dec 20, 2016 at 8:23 AM Irem Altan <irem.altan at duke.edu> wrote:

> Hi,
>
> I’m trying to set up a simulation on SDSC’s Comet, using half of the
> resources on a node. A node contains 24 CPU cores, and 4 GPUs. I want to
> use 12 cores and 2 GPUs. The resources on the node get shared, meaning that
> the unused 12 cores and 2 GPUs are used by another user.
>
> I tried the following setup:
>
> #!/bin/bash
> #SBATCH --job-name="md_u0"
> #SBATCH --output=out0.dat
> #SBATCH --ntasks-per-node=12
> #SBATCH -c 1
> #SBATCH --no-requeue
> #SBATCH -p gpu-shared
> #SBATCH --gres=gpu:2
> #SBATCH -t 2-00:00:00
> export OMP_NUM_THREADS=6
> module unload intel
> module load gromacs
> gmx_mpi grompp -f md_umbrella.mdp -c npt0.gro -t npt0.cpt -p topol.top -n
> index.ndx -maxwarn 1 -o umbrella0.tpr
> ibrun gmx_mpi mdrun -v -deffnm umbrella0 -pf pullf-umbrella0.xvg -px
> pullx-umbrella0.xvg -pin on
>
> which resulted in the following note:
>
> NOTE: Oversubscribing the CPU, will not pin threads
>
> and the simulation was excruciatingly slow. It normally takes about a day,
> but the output file said
>
> imb F 10% step 10400, will finish Mon Mar  6 20:49:50 2017
>
> When I set OMP_NUM_THREADS=1, Gromacs gives an error saying the optimum
> number is between 2 and 6, and stops execution. What is the correct value
> to use for this option? I can’t really do this by trial and error because
> the job waits on the queue for over a day.
>
> Best,
> Irem
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.