[gmx-users] performance

[gromacs query]

Hi Szilárd,

Thanks, option 3 was in my mind but I need to figure out now how :)
Manually fixing pinoffset as of now seems working with some quick test.
I think option 1 would require to ask the admin but I can try option 3
myself. As there are other users from different places who may not bother
using option 3. I think I would need to ask the admin to force option 1 but
before that I will try option 3.

JIom

On Wed, Sep 13, 2017 at 7:10 PM, Szilárd Páll <pall.szilard at gmail.com>
wrote:

> J,
>
> You have a few options:
>
> * Use SLURM to assign not only the set of GPUs, but also the correct
> set of CPU cores to each mdrun process. If you do so, mdrun will
> respect the affinity mask it will inherit and your two mdrun jobs
> should be running on the right set of cores. This has the drawback
> that (AFAIK) SLURM/aprun (or srun) will not allow you to bind each
> application thread to a core/hardware thread (which is what mdrun
> does), only a process to a group of cores/hw threads which can
> sometimes lead to performance loss. (You might be able to compensate
> using some OpenMP library environment variables, though.)
>
> * Run multiple jobs with mdrun "-multi"/"-multidir"  (either two per
> node or mulitple across nodes) and benefit from the rank/thread to
> core/hw thread assignment that's supported also across multiple
> simulations part of a multi-run; e.g.:
> mpirun -np 4 gmx mdrun -multi 4 -ntomp N -multidir my_input_dir{1,2,3,4}
> will launch 4 ranks and start 4 simulations in each of the four
> directories passed.
>
> * Write a wrapper script around gmx mdrun which will be what you
> launch with SLURM; you can then inspect the node and decide what
> pinoffset value to pass to your mdrun launch command.
>
>
> I hope one of these will deliver the desired results :)
>
> Cheers,
> --
> Szilárd
>
>
> On Wed, Sep 13, 2017 at 7:47 PM, gromacs query <gromacsquery at gmail.com>
> wrote:
> > Hi Szilárd,
> >
> > Thanks for your reply. This is useful but now am thinking because the
> slurm
> > launches job in an automated way it is not really in my control to choose
> > the node. So following things can happen; say for two mdrun jobs I set
> > -pinoffset 0 and -pinoffset 4;
> >
> > - if they are running on the same node this is good
> > - if jobs run on different nodes (partially occupied or free) whether
> these
> > chosen pinoffsets will make sense or not as I don't know what pinoffset I
> > would need to set
> > - if I have to submit many jobs together and slurm chooses different/same
> > node itself then I think it is difficult to define pinoffset.
> >
> > -
> > J
> >
> > On Wed, Sep 13, 2017 at 6:14 PM, Szilárd Páll <pall.szilard at gmail.com>
> > wrote:
> >
> >> My guess is that the two jobs are using the same cores -- either all
> >> cores/threads or only half of them, but the same set.
> >>
> >> You should use -pinoffset; see:
> >>
> >> - Docs and example:
> >> http://manual.gromacs.org/documentation/2016/user-guide/
> >> mdrun-performance.html
> >>
> >> - More explanation on the thread pinning behavior on the old website:
> >> http://www.gromacs.org/Documentation/Acceleration_
> >> and_parallelization#Pinning_threads_to_physical_cores
> >>
> >> Cheers,
> >> --
> >> Szilárd
> >>
> >>
> >> On Wed, Sep 13, 2017 at 6:35 PM, gromacs query <gromacsquery at gmail.com>
> >> wrote:
> >> > Sorry forgot to add; we thought the two jobs are using same GPU ids
> but
> >> > cuda visible devices show both jobs are using different ids (0,1 and
> 2,3)
> >> >
> >> > -
> >> > J
> >> >
> >> > On Wed, Sep 13, 2017 at 5:33 PM, gromacs query <
> gromacsquery at gmail.com>
> >> > wrote:
> >> >
> >> >> Hi All,
> >> >>
> >> >> I have some issues with gromacs performance. There are many nodes and
> >> each
> >> >> node has number of gpus and the batch process is controlled by slurm.
> >> >> Although I get good performance with some settings of number of gpus
> and
> >> >> nprocs but when I submit same job twice on the same node then the
> >> >> performance is reduced drastically. e.g
> >> >>
> >> >> For 2 GPUs I get 300 ns per day when there is no other job running on
> >> the
> >> >> node. When I submit same job twice on the same node & at the same
> time,
> >> I
> >> >> get only 17 ns/day for both the jobs. I am using this:
> >> >>
> >> >> mpirun -np 4 gmx_mpi mdrun -deffnm test -ntomp 2 -maxh 0.12
> >> >>
> >> >> Any suggestions highly appreciated.
> >> >>
> >> >> Thanks
> >> >>
> >> >> Jiom
> >> >>
> >> > --
> >> > Gromacs Users mailing list
> >> >
> >> > * Please search the archive at http://www.gromacs.org/
> >> Support/Mailing_Lists/GMX-Users_List before posting!
> >> >
> >> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >> >
> >> > * For (un)subscribe requests visit
> >> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> >> send a mail to gmx-users-request at gromacs.org.
> >> --
> >> Gromacs Users mailing list
> >>
> >> * Please search the archive at http://www.gromacs.org/
> >> Support/Mailing_Lists/GMX-Users_List before posting!
> >>
> >> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >>
> >> * For (un)subscribe requests visit
> >> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> >> send a mail to gmx-users-request at gromacs.org.
> > --
> > Gromacs Users mailing list
> >
> > * Please search the archive at http://www.gromacs.org/Support
> /Mailing_Lists/GMX-Users_List before posting!
> >
> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >
> > * For (un)subscribe requests visit
> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
> --
> Gromacs Users mailing list
>
> * Please search the archive at http://www.gromacs.org/Support
> /Mailing_Lists/GMX-Users_List before posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
>