[gmx-users] GROMACS PBS GPU JOB submission

[Yu Du]

Hi Tuanan,

I think your problem can be separated into several parts:

First, use one PBS script contains 4 GMX commands, shown as follows, may solve your problems:

#PBS -l select=1:ncpus=40:mpiprocs=40:ompthreads=1:ngpus=1
mpirun -machinefile $PBS_NODEFILE -np 40 gmx_mpi mdrun -s nvt-prod1.tpr -deffnm TEST1 -ntomp 1 -gpu_id 0
mpirun -machinefile $PBS_NODEFILE -np 40 gmx_mpi mdrun -s nvt-prod2.tpr -deffnm TEST2 -ntomp 1 -gpu_id 1
mpirun -machinefile $PBS_NODEFILE -np 40 gmx_mpi mdrun -s nvt-prod3.tpr -deffnm TEST3 -ntomp 1 -gpu_id 2
mpirun -machinefile $PBS_NODEFILE -np 40 gmx_mpi mdrun -s nvt-prod4.tpr -deffnm TEST4 -ntomp 1 -gpu_id 3

Then, you should optimize the number of CPU used by one GPU to get the best performance.

Last, check mdrun in the GMX manual, use pin and related options to set the CPU range which each subjob uses to avoid the interference between them.

Cheers,

Du, Yu
PhD Student,
Shanghai Institute of Organic Chemistry
345 Ling Ling Rd., Shanghai, China.
Zip: 200032, Tel: (86) 021 5492 5275
________________________________
From: gromacs.org_gmx-users-bounces at maillist.sys.kth.se <gromacs.org_gmx-users-bounces at maillist.sys.kth.se> on behalf of Tuanan Lourenço <lourenco.tuanan at gmail.com>
Sent: Thursday, April 16, 2020 21:34
To: gmx-users at gromacs.org <gmx-users at gromacs.org>
Subject: [gmx-users] GROMACS PBS GPU JOB submission

Hi everyone

I am using GROMACS 2018 in a node with 80 core and 4 TESLA V1, the queue
system is PBS. I am having some issues with the GPU selection, what I want
is to use 1 GPU per job but GROMACS is always using all the four GPUs.

My submission script is the following:

#PBS -l select=1:ncpus=40:mpiprocs=40:ompthreads=1:ngpus=1

mpirun -machinefile $PBS_NODEFILE -np 40 gmx_mpi mdrun -s nvt-prod.tpr
-deffnm TEST -ntomp 1


However, looking to the gromacs log file I see "On host gn01 4 GPUs
auto-selected for this run".

I know if I use the flag -gpu_id I can tell for GROMACS what GPU I want to
use and in this case, everything is ok, GROMACS does what I say.
But this can be a problem if I submit more than one job to the node,
because the jobs will use the same GPU card.

My question is; there is any way to say for GROMACS or PBS to use the GPU
that is available at that moment? I have 4 GPU in the server, thus, I want
to submit 4 jobs each one using one GPU.


Thank you very much.



--
__________________________________________________
Dr. Tuanan C. Lourenço
--
Gromacs Users mailing list

* Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!

* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists

* For (un)subscribe requests visit
https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a mail to gmx-users-request at gromacs.org.