[gmx-users] performance

[Szilárd Páll]

J,

You have a few options:

* Use SLURM to assign not only the set of GPUs, but also the correct
set of CPU cores to each mdrun process. If you do so, mdrun will
respect the affinity mask it will inherit and your two mdrun jobs
should be running on the right set of cores. This has the drawback
that (AFAIK) SLURM/aprun (or srun) will not allow you to bind each
application thread to a core/hardware thread (which is what mdrun
does), only a process to a group of cores/hw threads which can
sometimes lead to performance loss. (You might be able to compensate
using some OpenMP library environment variables, though.)

* Run multiple jobs with mdrun "-multi"/"-multidir"  (either two per
node or mulitple across nodes) and benefit from the rank/thread to
core/hw thread assignment that's supported also across multiple
simulations part of a multi-run; e.g.:
mpirun -np 4 gmx mdrun -multi 4 -ntomp N -multidir my_input_dir{1,2,3,4}
will launch 4 ranks and start 4 simulations in each of the four
directories passed.

* Write a wrapper script around gmx mdrun which will be what you
launch with SLURM; you can then inspect the node and decide what
pinoffset value to pass to your mdrun launch command.


I hope one of these will deliver the desired results :)

Cheers,
--
Szilárd


On Wed, Sep 13, 2017 at 7:47 PM, gromacs query <gromacsquery at gmail.com> wrote:
> Hi Szilárd,
>
> Thanks for your reply. This is useful but now am thinking because the slurm
> launches job in an automated way it is not really in my control to choose
> the node. So following things can happen; say for two mdrun jobs I set
> -pinoffset 0 and -pinoffset 4;
>
> - if they are running on the same node this is good
> - if jobs run on different nodes (partially occupied or free) whether these
> chosen pinoffsets will make sense or not as I don't know what pinoffset I
> would need to set
> - if I have to submit many jobs together and slurm chooses different/same
> node itself then I think it is difficult to define pinoffset.
>
> -
> J
>
> On Wed, Sep 13, 2017 at 6:14 PM, Szilárd Páll <pall.szilard at gmail.com>
> wrote:
>
>> My guess is that the two jobs are using the same cores -- either all
>> cores/threads or only half of them, but the same set.
>>
>> You should use -pinoffset; see:
>>
>> - Docs and example:
>> http://manual.gromacs.org/documentation/2016/user-guide/
>> mdrun-performance.html
>>
>> - More explanation on the thread pinning behavior on the old website:
>> http://www.gromacs.org/Documentation/Acceleration_
>> and_parallelization#Pinning_threads_to_physical_cores
>>
>> Cheers,
>> --
>> Szilárd
>>
>>
>> On Wed, Sep 13, 2017 at 6:35 PM, gromacs query <gromacsquery at gmail.com>
>> wrote:
>> > Sorry forgot to add; we thought the two jobs are using same GPU ids but
>> > cuda visible devices show both jobs are using different ids (0,1 and 2,3)
>> >
>> > -
>> > J
>> >
>> > On Wed, Sep 13, 2017 at 5:33 PM, gromacs query <gromacsquery at gmail.com>
>> > wrote:
>> >
>> >> Hi All,
>> >>
>> >> I have some issues with gromacs performance. There are many nodes and
>> each
>> >> node has number of gpus and the batch process is controlled by slurm.
>> >> Although I get good performance with some settings of number of gpus and
>> >> nprocs but when I submit same job twice on the same node then the
>> >> performance is reduced drastically. e.g
>> >>
>> >> For 2 GPUs I get 300 ns per day when there is no other job running on
>> the
>> >> node. When I submit same job twice on the same node & at the same time,
>> I
>> >> get only 17 ns/day for both the jobs. I am using this:
>> >>
>> >> mpirun -np 4 gmx_mpi mdrun -deffnm test -ntomp 2 -maxh 0.12
>> >>
>> >> Any suggestions highly appreciated.
>> >>
>> >> Thanks
>> >>
>> >> Jiom
>> >>
>> > --
>> > Gromacs Users mailing list
>> >
>> > * Please search the archive at http://www.gromacs.org/
>> Support/Mailing_Lists/GMX-Users_List before posting!
>> >
>> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>> >
>> > * For (un)subscribe requests visit
>> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>> send a mail to gmx-users-request at gromacs.org.
>> --
>> Gromacs Users mailing list
>>
>> * Please search the archive at http://www.gromacs.org/
>> Support/Mailing_Lists/GMX-Users_List before posting!
>>
>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>
>> * For (un)subscribe requests visit
>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>> send a mail to gmx-users-request at gromacs.org.
> --
> Gromacs Users mailing list
>
> * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a mail to gmx-users-request at gromacs.org.