[gmx-users] Various questions related to Gromacs performance tuning

[Mark Abraham]

Hi,

There could certainly be a bug with -ddorder pp_pme, as there's no testing
done of that. If you can reproduce with a recent version of GROMACS, please
do file a bug report. (Though this week we're moving to new infrastructure,
so leave it for a day or two before trying to report it!

Mark

On Tue, 24 Mar 2020 at 21:03, Kutzner, Carsten <ckutzne at gwdg.de> wrote:

> Hi,
>
> > Am 24.03.2020 um 16:28 schrieb Tobias Klöffel <tobias.kloeffel at fau.de>:
> >
> > Dear all,
> > I am very new to Gromacs so maybe some of my problems are very easy to
> fix:)
> > Currently I am trying to compile and benchmark gromacs on AMD rome cpus,
> the benchmarks are taken from:
> > https://www.mpibpc.mpg.de/grubmueller/bench
> >
> > 1) OpenMP parallelization: Is it done via OpenMP tasks?
> Yes, all over the code loops are parallelized via OpenMP via #pragma omp
> parallel for
> and similar directives.
>
> > If the Intel toolchain is detected and  -DGMX_FFT_LIBRARY=mkl is
> set,-mkl=serial is used, even though -DGMX_OPENMP=on is set.
> GROMACS uses only the serial transposes - allowing mkl to open up its own
> OpenMP threads
> would lead to oversubscription of cores and performance degradation.
>
> > 2) I am trying to use gmx_mpi tune_pme but I never got it to run. I do
> not really understand what I have to specify for -mdrun. I
> Normally you need a serial (read: non-mpi enabled) 'gmx' so that you can
> call
> gmx tune_pme. Most queueing systems don't like it if one parallel program
> calls
> another parallel program.
>
> > tried -mdrun 'gmx_mpi mdrun' and export MPIRUN="mpirun
> -use-hwthread-cpus -np $tmpi -map-by ppr:$tnode:node:pe=$OMP_NUM_THREADS
> --report-bindings" But it just complains that mdrun is not working.
> There should be an output somewhere with the exact command line that
> tune_pme invoked to test whether mdrun works. That should shed some light
> on the issue.
>
> Side note: Tuning is normally only useful on CPU-nodes. If your nodes also
> have GPUs, you will probably not want to do this kind of PME tuning.
>
> > Normal  execution via $MPIRUN gmx_mpi mdrun -s ... works
> >
> >
> > 3) As far as I understood, most time of PME is spent in a 3d FFT and
> hence probably most time is spent in a mpi alltoall communication.
> Yes, but that also depends a lot on the number of nodes you are running on.
> Check for yourself: Do a 'normal' mdrun (without tune_pme) on the number
> of
> nodes that you are interested and check the detailed timings at the end of
> the log file. There you will find how much time is spent in various PME
> routines.
>
> Best,
>   Carsten
>
> > For that reason I would like to place all PME tasks on a separate node
> via -ddorder pp_pme. If I do so, the calculations just hangs. Specifying
> -ddorder interleave or cartesian works without problems. Is this a known
> issue?
> >
> > Kind regards,
> > Tobias Klöffel
> >
> > --
> > M.Sc. Tobias Klöffel
> > =======================================================
> > HPC (High Performance Computing) group
> > Erlangen Regional Computing Center(RRZE)
> > Friedrich-Alexander-Universität Erlangen-Nürnberg
> > Martensstr. 1
> > 91058 Erlangen
> >
> > Room: 1.133
> > Phone: +49 (0) 9131 / 85 - 20101
> >
> > =======================================================
> >
> > E-mail: tobias.kloeffel at fau.de
> >
> > --
> > Gromacs Users mailing list
> >
> > * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
> >
> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >
> > * For (un)subscribe requests visit
> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
>
> --
> Dr. Carsten Kutzner
> Max Planck Institute for Biophysical Chemistry
> Theoretical and Computational Biophysics
> Am Fassberg 11, 37077 Goettingen, Germany
> Tel. +49-551-2012313, Fax: +49-551-2012302
> http://www.mpibpc.mpg.de/grubmueller/kutzner
> http://www.mpibpc.mpg.de/grubmueller/sppexa
>
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.