[gmx-users] performance

gromacs query gromacsquery at gmail.com
Wed Sep 13 23:14:40 CEST 2017 

Hi Szilárd,

Thanks again. I tried now with -multidir like this:

mpirun -np 16 gmx_mpi mdrun -s test -ntomp 2 -maxh 0.1 -multidir t1 t2 t3 t4

So this runs 4 jobs on same node so for each job np is = 16/4, and each job
using 2 GPU. I get now quite improved performance and equal performance for
each job (~ 220 ns) though still slightly less than single independent job
(where I get 300 ns). I can live with that but -

Surprised: There are maximum 40 cores and 8 GPUs per node and thus my 4
jobs should consume 8 GPUS. So I am bit surprised with the fact the same
node on which my four jobs were running was already occupied with jobs by
some other user, which I think should not happen (may be slurm.config admin
issue?). Either my some jobs should have gone in queue or run on other node
if free.

What to do: Importantly though as an individual user I can submit -multidir
job but lets say, which is normally the case, there will be many other
unknown users who submit one or two jobs in that case performance will be
an issue (which is equivalent to my case when I submit many jobs without
-multi/multidir).  I think still they will need -pinoffset. Could you
please suggest what best can be done in such case?


-Jiom




On Wed, Sep 13, 2017 at 9:15 PM, Szilárd Páll <pall.szilard at gmail.com>
wrote:

> Hi,
>
> First off, have you considered options 2) using multi-sim? That would
> allow you to not have to bother manually set offsets. Can you not
> submit your jobs such that you fill at least a node?
>
> How many threads/cores does you node have? Can you share log files?
>
> Cheers,
> --
> Szilárd
>
>
> On Wed, Sep 13, 2017 at 9:14 PM, gromacs query <gromacsquery at gmail.com>
> wrote:
> > Hi Szilárd,
> >
> > Sorry I was bit quick to say its working with pinoffset. I just submitted
> > four same jobs (2 gpus, 4 nprocs) on the same node with -pin on and
> > different -pinoffset to 0, 5, 10, 15 (numbers should be fine as there are
> > 40 cores on node). Still I don't get same performance (all variably less
> > than 50%) as expected from a single independent job. Now am wondering if
> > its still related to overlap of cores as pin on should lock the cores for
> > the same job.
> >
> > -J
> >
> > On Wed, Sep 13, 2017 at 7:33 PM, gromacs query <gromacsquery at gmail.com>
> > wrote:
> >
> >> Hi Szilárd,
> >>
> >> Thanks, option 3 was in my mind but I need to figure out now how :)
> >> Manually fixing pinoffset as of now seems working with some quick test.
> >> I think option 1 would require to ask the admin but I can try option 3
> >> myself. As there are other users from different places who may not
> bother
> >> using option 3. I think I would need to ask the admin to force option 1
> but
> >> before that I will try option 3.
> >>
> >> JIom
> >>
> >> On Wed, Sep 13, 2017 at 7:10 PM, Szilárd Páll <pall.szilard at gmail.com>
> >> wrote:
> >>
> >>> J,
> >>>
> >>> You have a few options:
> >>>
> >>> * Use SLURM to assign not only the set of GPUs, but also the correct
> >>> set of CPU cores to each mdrun process. If you do so, mdrun will
> >>> respect the affinity mask it will inherit and your two mdrun jobs
> >>> should be running on the right set of cores. This has the drawback
> >>> that (AFAIK) SLURM/aprun (or srun) will not allow you to bind each
> >>> application thread to a core/hardware thread (which is what mdrun
> >>> does), only a process to a group of cores/hw threads which can
> >>> sometimes lead to performance loss. (You might be able to compensate
> >>> using some OpenMP library environment variables, though.)
> >>>
> >>> * Run multiple jobs with mdrun "-multi"/"-multidir"  (either two per
> >>> node or mulitple across nodes) and benefit from the rank/thread to
> >>> core/hw thread assignment that's supported also across multiple
> >>> simulations part of a multi-run; e.g.:
> >>> mpirun -np 4 gmx mdrun -multi 4 -ntomp N -multidir
> my_input_dir{1,2,3,4}
> >>> will launch 4 ranks and start 4 simulations in each of the four
> >>> directories passed.
> >>>
> >>> * Write a wrapper script around gmx mdrun which will be what you
> >>> launch with SLURM; you can then inspect the node and decide what
> >>> pinoffset value to pass to your mdrun launch command.
> >>>
> >>>
> >>> I hope one of these will deliver the desired results :)
> >>>
> >>> Cheers,
> >>> --
> >>> Szilárd
> >>>
> >>>
> >>> On Wed, Sep 13, 2017 at 7:47 PM, gromacs query <gromacsquery at gmail.com
> >
> >>> wrote:
> >>> > Hi Szilárd,
> >>> >
> >>> > Thanks for your reply. This is useful but now am thinking because the
> >>> slurm
> >>> > launches job in an automated way it is not really in my control to
> >>> choose
> >>> > the node. So following things can happen; say for two mdrun jobs I
> set
> >>> > -pinoffset 0 and -pinoffset 4;
> >>> >
> >>> > - if they are running on the same node this is good
> >>> > - if jobs run on different nodes (partially occupied or free) whether
> >>> these
> >>> > chosen pinoffsets will make sense or not as I don't know what
> pinoffset
> >>> I
> >>> > would need to set
> >>> > - if I have to submit many jobs together and slurm chooses
> >>> different/same
> >>> > node itself then I think it is difficult to define pinoffset.
> >>> >
> >>> > -
> >>> > J
> >>> >
> >>> > On Wed, Sep 13, 2017 at 6:14 PM, Szilárd Páll <
> pall.szilard at gmail.com>
> >>> > wrote:
> >>> >
> >>> >> My guess is that the two jobs are using the same cores -- either all
> >>> >> cores/threads or only half of them, but the same set.
> >>> >>
> >>> >> You should use -pinoffset; see:
> >>> >>
> >>> >> - Docs and example:
> >>> >> http://manual.gromacs.org/documentation/2016/user-guide/
> >>> >> mdrun-performance.html
> >>> >>
> >>> >> - More explanation on the thread pinning behavior on the old
> website:
> >>> >> http://www.gromacs.org/Documentation/Acceleration_
> >>> >> and_parallelization#Pinning_threads_to_physical_cores
> >>> >>
> >>> >> Cheers,
> >>> >> --
> >>> >> Szilárd
> >>> >>
> >>> >>
> >>> >> On Wed, Sep 13, 2017 at 6:35 PM, gromacs query <
> gromacsquery at gmail.com
> >>> >
> >>> >> wrote:
> >>> >> > Sorry forgot to add; we thought the two jobs are using same GPU
> ids
> >>> but
> >>> >> > cuda visible devices show both jobs are using different ids (0,1
> and
> >>> 2,3)
> >>> >> >
> >>> >> > -
> >>> >> > J
> >>> >> >
> >>> >> > On Wed, Sep 13, 2017 at 5:33 PM, gromacs query <
> >>> gromacsquery at gmail.com>
> >>> >> > wrote:
> >>> >> >
> >>> >> >> Hi All,
> >>> >> >>
> >>> >> >> I have some issues with gromacs performance. There are many nodes
> >>> and
> >>> >> each
> >>> >> >> node has number of gpus and the batch process is controlled by
> >>> slurm.
> >>> >> >> Although I get good performance with some settings of number of
> >>> gpus and
> >>> >> >> nprocs but when I submit same job twice on the same node then the
> >>> >> >> performance is reduced drastically. e.g
> >>> >> >>
> >>> >> >> For 2 GPUs I get 300 ns per day when there is no other job
> running
> >>> on
> >>> >> the
> >>> >> >> node. When I submit same job twice on the same node & at the same
> >>> time,
> >>> >> I
> >>> >> >> get only 17 ns/day for both the jobs. I am using this:
> >>> >> >>
> >>> >> >> mpirun -np 4 gmx_mpi mdrun -deffnm test -ntomp 2 -maxh 0.12
> >>> >> >>
> >>> >> >> Any suggestions highly appreciated.
> >>> >> >>
> >>> >> >> Thanks
> >>> >> >>
> >>> >> >> Jiom
> >>> >> >>
> >>> >> > --
> >>> >> > Gromacs Users mailing list
> >>> >> >
> >>> >> > * Please search the archive at http://www.gromacs.org/
> >>> >> Support/Mailing_Lists/GMX-Users_List before posting!
> >>> >> >
> >>> >> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >>> >> >
> >>> >> > * For (un)subscribe requests visit
> >>> >> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_
> gmx-users
> >>> or
> >>> >> send a mail to gmx-users-request at gromacs.org.
> >>> >> --
> >>> >> Gromacs Users mailing list
> >>> >>
> >>> >> * Please search the archive at http://www.gromacs.org/
> >>> >> Support/Mailing_Lists/GMX-Users_List before posting!
> >>> >>
> >>> >> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >>> >>
> >>> >> * For (un)subscribe requests visit
> >>> >> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users
> or
> >>> >> send a mail to gmx-users-request at gromacs.org.
> >>> > --
> >>> > Gromacs Users mailing list
> >>> >
> >>> > * Please search the archive at http://www.gromacs.org/Support
> >>> /Mailing_Lists/GMX-Users_List before posting!
> >>> >
> >>> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >>> >
> >>> > * For (un)subscribe requests visit
> >>> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users
> or
> >>> send a mail to gmx-users-request at gromacs.org.
> >>> --
> >>> Gromacs Users mailing list
> >>>
> >>> * Please search the archive at http://www.gromacs.org/Support
> >>> /Mailing_Lists/GMX-Users_List before posting!
> >>>
> >>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >>>
> >>> * For (un)subscribe requests visit
> >>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> >>> send a mail to gmx-users-request at gromacs.org.
> >>>
> >>
> >>
> > --
> > Gromacs Users mailing list
> >
> > * Please search the archive at http://www.gromacs.org/
> Support/Mailing_Lists/GMX-Users_List before posting!
> >
> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >
> > * For (un)subscribe requests visit
> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
> --
> Gromacs Users mailing list
>
> * Please search the archive at http://www.gromacs.org/
> Support/Mailing_Lists/GMX-Users_List before posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
>

More information about the gromacs.org_gmx-users mailing list