[gmx-users] optimum number of OpenMP threads for simulation

[Irem Altan]

Hi,

Thank you for your response. Can I run two different simulations at once? The simulation that I mentioned is just one window of the umbrella sampling I’m trying to do. Can I bundle two windows together? Otherwise, I can just use the whole node and run a single simulation on 4 GPUs. I’m trying to use half the node because running two simulations on half the node would be faster than running one simulation on the entire node, as the simulation speed doesn’t double when the resources are doubled.

Does using -pin on make sure that the threads are on the same CPU socket?

To ask for 2*6 threads, for instance, would I need to ask for 6 tasks per node and 2 OpenMP threads? If so, what would the number of tasks per CPU be?

Best,
Irem

> On Dec 19, 2016, at 4:44 PM, Mark Abraham <mark.j.abraham at gmail.com> wrote:
> 
> Hi,
> 
> The simplest thing to do is to run another replicate of your simulation
> (e.g. different velocities) using mdrun -multi, filling the whole node, and
> now mdrun will take care of all the details. Even if you ignore the
> replicate, this can be more efficient than not managing anything. ;-)
> 
> Otherwise, if your cluster implements node sharing, then they have the
> burden of documenting how they expect users to be able to keep their jobs
> apart (or explaining why they want the resources wasted ;-) ). GROMACS is
> extremely sensitive to locality of work, and this is ruined both if GROMACS
> threads are moved around by the kernel, or if other jobs on the node are
> moved onto cores GROMACS is using. You also want to make sure GROMACS
> threads are close together, e.g. on the same CPU socket.
> 
> Further, you should note that "tasks per node" means MPI ranks, and
> OMP_NUM_THREADS means thread per rank, so your script is asking for 12*6=72
> threads, which is not what you intended. You should try 2*6 or maybe 4*3 if
> your system is large enough.
> 
> More background information is at
> https://urldefense.proofpoint.com/v2/url?u=http-3A__manual.gromacs.org_documentation_2016.1_user-2Dguide_mdrun-2Dperformance.html&d=CwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=r1Wl_e-3DAvYeqhtCRi2Mbok8HBpo_RH4ll0E7Hffr4&m=xmhESE9v69_hhH1s-hCebbtDVAlyAD1I8jPeep4T0W4&s=lPPKZ0aJlT2y6BkqObZv63EydJYi1RS4vJ8F_SKATGw&e= 
> 
> Mark
> 
> On Tue, Dec 20, 2016 at 8:23 AM Irem Altan <irem.altan at duke.edu> wrote:
> 
>> Hi,
>> 
>> I’m trying to set up a simulation on SDSC’s Comet, using half of the
>> resources on a node. A node contains 24 CPU cores, and 4 GPUs. I want to
>> use 12 cores and 2 GPUs. The resources on the node get shared, meaning that
>> the unused 12 cores and 2 GPUs are used by another user.
>> 
>> I tried the following setup:
>> 
>> #!/bin/bash
>> #SBATCH --job-name="md_u0"
>> #SBATCH --output=out0.dat
>> #SBATCH --ntasks-per-node=12
>> #SBATCH -c 1
>> #SBATCH --no-requeue
>> #SBATCH -p gpu-shared
>> #SBATCH --gres=gpu:2
>> #SBATCH -t 2-00:00:00
>> export OMP_NUM_THREADS=6
>> module unload intel
>> module load gromacs
>> gmx_mpi grompp -f md_umbrella.mdp -c npt0.gro -t npt0.cpt -p topol.top -n
>> index.ndx -maxwarn 1 -o umbrella0.tpr
>> ibrun gmx_mpi mdrun -v -deffnm umbrella0 -pf pullf-umbrella0.xvg -px
>> pullx-umbrella0.xvg -pin on
>> 
>> which resulted in the following note:
>> 
>> NOTE: Oversubscribing the CPU, will not pin threads
>> 
>> and the simulation was excruciatingly slow. It normally takes about a day,
>> but the output file said
>> 
>> imb F 10% step 10400, will finish Mon Mar  6 20:49:50 2017
>> 
>> When I set OMP_NUM_THREADS=1, Gromacs gives an error saying the optimum
>> number is between 2 and 6, and stops execution. What is the correct value
>> to use for this option? I can’t really do this by trial and error because
>> the job waits on the queue for over a day.
>> 
>> Best,
>> Irem
>> --
>> Gromacs Users mailing list
>> 
>> * Please search the archive at
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.gromacs.org_Support_Mailing-5FLists_GMX-2DUsers-5FList&d=CwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=r1Wl_e-3DAvYeqhtCRi2Mbok8HBpo_RH4ll0E7Hffr4&m=xmhESE9v69_hhH1s-hCebbtDVAlyAD1I8jPeep4T0W4&s=O-m28bj3SAVemEcC-UrtIErnYjAiHo2y-zTDHjnskVM&e=  before
>> posting!
>> 
>> * Can't post? Read https://urldefense.proofpoint.com/v2/url?u=http-3A__www.gromacs.org_Support_Mailing-5FLists&d=CwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=r1Wl_e-3DAvYeqhtCRi2Mbok8HBpo_RH4ll0E7Hffr4&m=xmhESE9v69_hhH1s-hCebbtDVAlyAD1I8jPeep4T0W4&s=1pr7ilSgSl49tlK40t317FRVxKi4DWkRoJAc4C2PWx0&e= 
>> 
>> * For (un)subscribe requests visit
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__maillist.sys.kth.se_mailman_listinfo_gromacs.org-5Fgmx-2Dusers&d=CwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=r1Wl_e-3DAvYeqhtCRi2Mbok8HBpo_RH4ll0E7Hffr4&m=xmhESE9v69_hhH1s-hCebbtDVAlyAD1I8jPeep4T0W4&s=wBLQHLocJ5pYMxDKY7cqjzTjCsOFlBwDo3QzAqLAarU&e=  or
>> send a mail to gmx-users-request at gromacs.org.
> -- 
> Gromacs Users mailing list
> 
> * Please search the archive at https://urldefense.proofpoint.com/v2/url?u=http-3A__www.gromacs.org_Support_Mailing-5FLists_GMX-2DUsers-5FList&d=CwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=r1Wl_e-3DAvYeqhtCRi2Mbok8HBpo_RH4ll0E7Hffr4&m=xmhESE9v69_hhH1s-hCebbtDVAlyAD1I8jPeep4T0W4&s=O-m28bj3SAVemEcC-UrtIErnYjAiHo2y-zTDHjnskVM&e=  before posting!
> 
> * Can't post? Read https://urldefense.proofpoint.com/v2/url?u=http-3A__www.gromacs.org_Support_Mailing-5FLists&d=CwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=r1Wl_e-3DAvYeqhtCRi2Mbok8HBpo_RH4ll0E7Hffr4&m=xmhESE9v69_hhH1s-hCebbtDVAlyAD1I8jPeep4T0W4&s=1pr7ilSgSl49tlK40t317FRVxKi4DWkRoJAc4C2PWx0&e= 
> 
> * For (un)subscribe requests visit
> https://urldefense.proofpoint.com/v2/url?u=https-3A__maillist.sys.kth.se_mailman_listinfo_gromacs.org-5Fgmx-2Dusers&d=CwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=r1Wl_e-3DAvYeqhtCRi2Mbok8HBpo_RH4ll0E7Hffr4&m=xmhESE9v69_hhH1s-hCebbtDVAlyAD1I8jPeep4T0W4&s=wBLQHLocJ5pYMxDKY7cqjzTjCsOFlBwDo3QzAqLAarU&e=  or send a mail to gmx-users-request at gromacs.org.