[gmx-users] optimum number of OpenMP threads for simulation

Mon Dec 19 23:44:31 CET 2016

Hi,

On Tue, Dec 20, 2016 at 9:24 AM Irem Altan <irem.altan at duke.edu> wrote:

> Hi,
>
> Thank you for your response. Can I run two different simulations at once?

Yes, mdrun -multi is pretty tolerant (though I recommend mdrun -multidir
because it's easier to manage each simulation in a separate directory).

The simulation that I mentioned is just one window of the umbrella sampling
> I’m trying to do. Can I bundle two windows together? Otherwise, I can just
> use the whole node and run a single simulation on 4 GPUs. I’m trying to use
> half the node because running two simulations on half the node would be
> faster than running one simulation on the entire node, as the simulation
> speed doesn’t double when the resources are doubled.
>

Overall throughput will be highest when you use the least resources per
simulation - see https://arxiv.org/abs/1507.00898

> Does using -pin on make sure that the threads are on the same CPU socket?
>

No, see the link I gave.

> To ask for 2*6 threads, for instance, would I need to ask for 6 tasks per
> node and 2 OpenMP threads?

Depends what you think 2 and 6 mean, but you seem to have swapped the order
of the meanings I was using, which matched the order in which I gave
definitions. Both your cluster's docs and the GROMACS docs have several
useful examples that would be good to read and understand :-)

> If so, what would the number of tasks per CPU be?
>

"CPU" and "task" are ill defined. I can't read your cluster's docs for you
:-)

Mark

> Best,
> Irem
>
> > On Dec 19, 2016, at 4:44 PM, Mark Abraham <mark.j.abraham at gmail.com>
> wrote:
> >
> > Hi,
> >
> > The simplest thing to do is to run another replicate of your simulation
> > (e.g. different velocities) using mdrun -multi, filling the whole node,
> and
> > now mdrun will take care of all the details. Even if you ignore the
> > replicate, this can be more efficient than not managing anything. ;-)
> >
> > Otherwise, if your cluster implements node sharing, then they have the
> > burden of documenting how they expect users to be able to keep their jobs
> > apart (or explaining why they want the resources wasted ;-) ). GROMACS is
> > extremely sensitive to locality of work, and this is ruined both if
> GROMACS
> > threads are moved around by the kernel, or if other jobs on the node are
> > moved onto cores GROMACS is using. You also want to make sure GROMACS
> > threads are close together, e.g. on the same CPU socket.
> >
> > Further, you should note that "tasks per node" means MPI ranks, and
> > OMP_NUM_THREADS means thread per rank, so your script is asking for
> 12*6=72
> > threads, which is not what you intended. You should try 2*6 or maybe 4*3
> if
> > your system is large enough.
> >
> > More background information is at
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__manual.gromacs.org_documentation_2016.1_user-2Dguide_mdrun-2Dperformance.html&d=CwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=r1Wl_e-3DAvYeqhtCRi2Mbok8HBpo_RH4ll0E7Hffr4&m=xmhESE9v69_hhH1s-hCebbtDVAlyAD1I8jPeep4T0W4&s=lPPKZ0aJlT2y6BkqObZv63EydJYi1RS4vJ8F_SKATGw&e=
> >
> > Mark
> >
> > On Tue, Dec 20, 2016 at 8:23 AM Irem Altan <irem.altan at duke.edu> wrote:
> >
> >> Hi,
> >>
> >> I’m trying to set up a simulation on SDSC’s Comet, using half of the
> >> resources on a node. A node contains 24 CPU cores, and 4 GPUs. I want to
> >> use 12 cores and 2 GPUs. The resources on the node get shared, meaning
> that
> >> the unused 12 cores and 2 GPUs are used by another user.
> >>
> >> I tried the following setup:
> >>
> >> #!/bin/bash
> >> #SBATCH --job-name="md_u0"
> >> #SBATCH --output=out0.dat
> >> #SBATCH --ntasks-per-node=12
> >> #SBATCH -c 1
> >> #SBATCH --no-requeue
> >> #SBATCH -p gpu-shared
> >> #SBATCH --gres=gpu:2
> >> #SBATCH -t 2-00:00:00
> >> export OMP_NUM_THREADS=6
> >> module unload intel
> >> module load gromacs
> >> gmx_mpi grompp -f md_umbrella.mdp -c npt0.gro -t npt0.cpt -p topol.top
> -n
> >> index.ndx -maxwarn 1 -o umbrella0.tpr
> >> ibrun gmx_mpi mdrun -v -deffnm umbrella0 -pf pullf-umbrella0.xvg -px
> >> pullx-umbrella0.xvg -pin on
> >>
> >> which resulted in the following note:
> >>
> >> NOTE: Oversubscribing the CPU, will not pin threads
> >>
> >> and the simulation was excruciatingly slow. It normally takes about a
> day,
> >> but the output file said
> >>
> >> imb F 10% step 10400, will finish Mon Mar  6 20:49:50 2017
> >>
> >> When I set OMP_NUM_THREADS=1, Gromacs gives an error saying the optimum
> >> number is between 2 and 6, and stops execution. What is the correct
> value
> >> to use for this option? I can’t really do this by trial and error
> because
> >> the job waits on the queue for over a day.
> >>
> >> Best,
> >> Irem
> >> --
> >> Gromacs Users mailing list
> >>
> >> * Please search the archive at
> >>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.gromacs.org_Support_Mailing-5FLists_GMX-2DUsers-5FList&d=CwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=r1Wl_e-3DAvYeqhtCRi2Mbok8HBpo_RH4ll0E7Hffr4&m=xmhESE9v69_hhH1s-hCebbtDVAlyAD1I8jPeep4T0W4&s=O-m28bj3SAVemEcC-UrtIErnYjAiHo2y-zTDHjnskVM&e=
> before
> >> posting!
> >>
> >> * Can't post? Read
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.gromacs.org_Support_Mailing-5FLists&d=CwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=r1Wl_e-3DAvYeqhtCRi2Mbok8HBpo_RH4ll0E7Hffr4&m=xmhESE9v69_hhH1s-hCebbtDVAlyAD1I8jPeep4T0W4&s=1pr7ilSgSl49tlK40t317FRVxKi4DWkRoJAc4C2PWx0&e=
> >>
> >> * For (un)subscribe requests visit
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__maillist.sys.kth.se_mailman_listinfo_gromacs.org-5Fgmx-2Dusers&d=CwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=r1Wl_e-3DAvYeqhtCRi2Mbok8HBpo_RH4ll0E7Hffr4&m=xmhESE9v69_hhH1s-hCebbtDVAlyAD1I8jPeep4T0W4&s=wBLQHLocJ5pYMxDKY7cqjzTjCsOFlBwDo3QzAqLAarU&e=
> or
> >> send a mail to gmx-users-request at gromacs.org.
> > --
> > Gromacs Users mailing list
> >
> > * Please search the archive at
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.gromacs.org_Support_Mailing-5FLists_GMX-2DUsers-5FList&d=CwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=r1Wl_e-3DAvYeqhtCRi2Mbok8HBpo_RH4ll0E7Hffr4&m=xmhESE9v69_hhH1s-hCebbtDVAlyAD1I8jPeep4T0W4&s=O-m28bj3SAVemEcC-UrtIErnYjAiHo2y-zTDHjnskVM&e=
> before posting!
> >
> > * Can't post? Read
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.gromacs.org_Support_Mailing-5FLists&d=CwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=r1Wl_e-3DAvYeqhtCRi2Mbok8HBpo_RH4ll0E7Hffr4&m=xmhESE9v69_hhH1s-hCebbtDVAlyAD1I8jPeep4T0W4&s=1pr7ilSgSl49tlK40t317FRVxKi4DWkRoJAc4C2PWx0&e=
> >
> > * For (un)subscribe requests visit
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__maillist.sys.kth.se_mailman_listinfo_gromacs.org-5Fgmx-2Dusers&d=CwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=r1Wl_e-3DAvYeqhtCRi2Mbok8HBpo_RH4ll0E7Hffr4&m=xmhESE9v69_hhH1s-hCebbtDVAlyAD1I8jPeep4T0W4&s=wBLQHLocJ5pYMxDKY7cqjzTjCsOFlBwDo3QzAqLAarU&e=
> or send a mail to gmx-users-request at gromacs.org.
>
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.