[gmx-users] performance

[gromacs query]

Hi Szilárd,

Sorry I was bit quick to say its working with pinoffset. I just submitted
four same jobs (2 gpus, 4 nprocs) on the same node with -pin on and
different -pinoffset to 0, 5, 10, 15 (numbers should be fine as there are
40 cores on node). Still I don't get same performance (all variably less
than 50%) as expected from a single independent job. Now am wondering if
its still related to overlap of cores as pin on should lock the cores for
the same job.

-J

On Wed, Sep 13, 2017 at 7:33 PM, gromacs query <gromacsquery at gmail.com>
wrote:

> Hi Szilárd,
>
> Thanks, option 3 was in my mind but I need to figure out now how :)
> Manually fixing pinoffset as of now seems working with some quick test.
> I think option 1 would require to ask the admin but I can try option 3
> myself. As there are other users from different places who may not bother
> using option 3. I think I would need to ask the admin to force option 1 but
> before that I will try option 3.
>
> JIom
>
> On Wed, Sep 13, 2017 at 7:10 PM, Szilárd Páll <pall.szilard at gmail.com>
> wrote:
>
>> J,
>>
>> You have a few options:
>>
>> * Use SLURM to assign not only the set of GPUs, but also the correct
>> set of CPU cores to each mdrun process. If you do so, mdrun will
>> respect the affinity mask it will inherit and your two mdrun jobs
>> should be running on the right set of cores. This has the drawback
>> that (AFAIK) SLURM/aprun (or srun) will not allow you to bind each
>> application thread to a core/hardware thread (which is what mdrun
>> does), only a process to a group of cores/hw threads which can
>> sometimes lead to performance loss. (You might be able to compensate
>> using some OpenMP library environment variables, though.)
>>
>> * Run multiple jobs with mdrun "-multi"/"-multidir"  (either two per
>> node or mulitple across nodes) and benefit from the rank/thread to
>> core/hw thread assignment that's supported also across multiple
>> simulations part of a multi-run; e.g.:
>> mpirun -np 4 gmx mdrun -multi 4 -ntomp N -multidir my_input_dir{1,2,3,4}
>> will launch 4 ranks and start 4 simulations in each of the four
>> directories passed.
>>
>> * Write a wrapper script around gmx mdrun which will be what you
>> launch with SLURM; you can then inspect the node and decide what
>> pinoffset value to pass to your mdrun launch command.
>>
>>
>> I hope one of these will deliver the desired results :)
>>
>> Cheers,
>> --
>> Szilárd
>>
>>
>> On Wed, Sep 13, 2017 at 7:47 PM, gromacs query <gromacsquery at gmail.com>
>> wrote:
>> > Hi Szilárd,
>> >
>> > Thanks for your reply. This is useful but now am thinking because the
>> slurm
>> > launches job in an automated way it is not really in my control to
>> choose
>> > the node. So following things can happen; say for two mdrun jobs I set
>> > -pinoffset 0 and -pinoffset 4;
>> >
>> > - if they are running on the same node this is good
>> > - if jobs run on different nodes (partially occupied or free) whether
>> these
>> > chosen pinoffsets will make sense or not as I don't know what pinoffset
>> I
>> > would need to set
>> > - if I have to submit many jobs together and slurm chooses
>> different/same
>> > node itself then I think it is difficult to define pinoffset.
>> >
>> > -
>> > J
>> >
>> > On Wed, Sep 13, 2017 at 6:14 PM, Szilárd Páll <pall.szilard at gmail.com>
>> > wrote:
>> >
>> >> My guess is that the two jobs are using the same cores -- either all
>> >> cores/threads or only half of them, but the same set.
>> >>
>> >> You should use -pinoffset; see:
>> >>
>> >> - Docs and example:
>> >> http://manual.gromacs.org/documentation/2016/user-guide/
>> >> mdrun-performance.html
>> >>
>> >> - More explanation on the thread pinning behavior on the old website:
>> >> http://www.gromacs.org/Documentation/Acceleration_
>> >> and_parallelization#Pinning_threads_to_physical_cores
>> >>
>> >> Cheers,
>> >> --
>> >> Szilárd
>> >>
>> >>
>> >> On Wed, Sep 13, 2017 at 6:35 PM, gromacs query <gromacsquery at gmail.com
>> >
>> >> wrote:
>> >> > Sorry forgot to add; we thought the two jobs are using same GPU ids
>> but
>> >> > cuda visible devices show both jobs are using different ids (0,1 and
>> 2,3)
>> >> >
>> >> > -
>> >> > J
>> >> >
>> >> > On Wed, Sep 13, 2017 at 5:33 PM, gromacs query <
>> gromacsquery at gmail.com>
>> >> > wrote:
>> >> >
>> >> >> Hi All,
>> >> >>
>> >> >> I have some issues with gromacs performance. There are many nodes
>> and
>> >> each
>> >> >> node has number of gpus and the batch process is controlled by
>> slurm.
>> >> >> Although I get good performance with some settings of number of
>> gpus and
>> >> >> nprocs but when I submit same job twice on the same node then the
>> >> >> performance is reduced drastically. e.g
>> >> >>
>> >> >> For 2 GPUs I get 300 ns per day when there is no other job running
>> on
>> >> the
>> >> >> node. When I submit same job twice on the same node & at the same
>> time,
>> >> I
>> >> >> get only 17 ns/day for both the jobs. I am using this:
>> >> >>
>> >> >> mpirun -np 4 gmx_mpi mdrun -deffnm test -ntomp 2 -maxh 0.12
>> >> >>
>> >> >> Any suggestions highly appreciated.
>> >> >>
>> >> >> Thanks
>> >> >>
>> >> >> Jiom
>> >> >>
>> >> > --
>> >> > Gromacs Users mailing list
>> >> >
>> >> > * Please search the archive at http://www.gromacs.org/
>> >> Support/Mailing_Lists/GMX-Users_List before posting!
>> >> >
>> >> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>> >> >
>> >> > * For (un)subscribe requests visit
>> >> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users
>> or
>> >> send a mail to gmx-users-request at gromacs.org.
>> >> --
>> >> Gromacs Users mailing list
>> >>
>> >> * Please search the archive at http://www.gromacs.org/
>> >> Support/Mailing_Lists/GMX-Users_List before posting!
>> >>
>> >> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>> >>
>> >> * For (un)subscribe requests visit
>> >> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>> >> send a mail to gmx-users-request at gromacs.org.
>> > --
>> > Gromacs Users mailing list
>> >
>> > * Please search the archive at http://www.gromacs.org/Support
>> /Mailing_Lists/GMX-Users_List before posting!
>> >
>> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>> >
>> > * For (un)subscribe requests visit
>> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>> send a mail to gmx-users-request at gromacs.org.
>> --
>> Gromacs Users mailing list
>>
>> * Please search the archive at http://www.gromacs.org/Support
>> /Mailing_Lists/GMX-Users_List before posting!
>>
>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>
>> * For (un)subscribe requests visit
>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>> send a mail to gmx-users-request at gromacs.org.
>>
>
>