[gmx-users] g_tune_pme_mpi is not compatible to mdrun_mpi
Mark Abraham
mark.j.abraham at gmail.com
Mon Sep 29 18:40:17 CEST 2014
Hi,
That seems suitable.
Mark
On Mon, Sep 29, 2014 at 6:32 PM, Carsten Kutzner <ckutzne at gwdg.de> wrote:
> Hi,
>
> On 29 Sep 2014, at 18:17, Mark Abraham <mark.j.abraham at gmail.com> wrote:
>
> > Hi,
> >
> > It can't be fixed, because there is no surefire way to run an arbitrary
> tpr
> > on arbitrary number of ranks, regardless of how you guess -npme might
> > succeed.
> What about making this check on two ranks always, regardless of
> what was specified on the g_tune_pme command line? On two ranks,
> we will never have separate PME ranks, so it should always work,
> since we end up with two ranks, doing PP and then PME.
> If the system is so small that you can not decompose it in two
> DD domains, there is no use to do tuning anyway.
>
> So even if you say
> g_tune_pme -np 48 -s input.tpr
>
> we first check with
> mpirun -np 2 mdrun -s input.tpr
>
> and only after that continue with -np 48.
>
> Carsten
>
> > We should just make the check optional, instead of being a deal
> > breaker.
> >
> > Mark
> > On Sep 29, 2014 4:35 PM, "Carsten Kutzner" <ckutzne at gwdg.de> wrote:
> >
> >> Hi,
> >>
> >> I see where the problem is.
> >> There is an initial check in g_tune_pme to make sure that parallel
> >> runs can be executed at all. This is being run with the automatic
> >> number of PME-only ranks, which is 11 for your input file.
> >> Unfortunately, this results in 37 PP ranks, for which no domain
> >> decomposition can be found.
> >>
> >> At some point in the past we discussed that this could happen
> >> and it should be fixed. Will open a bug entry.
> >>
> >> Thanks,
> >> Carsten
> >>
> >>
> >> On 29 Sep 2014, at 15:36, Ebert Maximilian <m.ebert at umontreal.ca>
> wrote:
> >>
> >>> Hi,
> >>>
> >>> this ist he command:
> >>>
> >>> setenv MDRUN mdrun_mpi
> >>>
> >>> g_tune_pme_mpi -np 48 -s ../eq_nvt/1ZG4_nvt.tpr -launch
> >>>
> >>>
> >>> Here the output of perf.out
> >>>
> >>> ------------------------------------------------------------
> >>>
> >>> P E R F O R M A N C E R E S U L T S
> >>>
> >>> ------------------------------------------------------------
> >>> g_tune_pme_mpi for Gromacs VERSION 5.0.1
> >>> Number of ranks : 48
> >>> The mpirun command is : mpirun
> >>> Passing # of ranks via : -np
> >>> The mdrun command is : mdrun_mpi
> >>> mdrun args benchmarks : -resetstep 100 -o bench.trr -x bench.xtc -cpo
> >> bench.cpt -c bench.gro -e bench.edr -g bench.log
> >>> Benchmark steps : 1000
> >>> dlb equilibration steps : 100
> >>> mdrun args at launchtime:
> >>> Repeats for each test : 2
> >>> Input file : ../eq_nvt/1ZG4_nvt.tpr
> >>> PME/PP load estimate : 0.151964
> >>> Number of particles : 39489
> >>> Coulomb type : PME
> >>> Grid spacing x y z : 0.114561 0.114561 0.114561
> >>> Van der Waals type : Cut-off
> >>>
> >>> Will try these real/reciprocal workload settings:
> >>> No. scaling rcoulomb nkx nky nkz spacing rvdw tpr file
> >>> 0 1.000000 1.200000 72 72 72 0.120000 1.200000
> >> ../eq_nvt/1ZG4_nvt_bench00.tpr
> >>> 1 1.100000 1.320000 64 64 64 0.132000 1.320000
> >> ../eq_nvt/1ZG4_nvt_bench01.tpr
> >>> 2 1.200000 1.440000 60 60 60 0.144000 1.440000
> >> ../eq_nvt/1ZG4_nvt_bench02.tpr
> >>>
> >>> Note that in addition to the Coulomb radius and the Fourier grid
> >>> other input settings were also changed (see table above).
> >>> Please check if the modified settings are appropriate.
> >>>
> >>> Individual timings for input file 0 (../eq_nvt/1ZG4_nvt_bench00.tpr):
> >>> PME ranks Gcycles ns/day PME/f Remark
> >>>
> >>> ------------------------------------------------------------
> >>> Cannot run the benchmark simulations! Please check the error message of
> >>> mdrun for the source of the problem. Did you provide a command line
> >>> argument that neither g_tune_pme nor mdrun understands? Offending
> >> command:
> >>>
> >>> mpirun -np 48 mdrun_mpi -npme 11 -s ../eq_nvt/1ZG4_nvt_bench00.tpr
> >> -resetstep 100 -o bench.trr -x bench.xtc -cpo bench.cpt -c bench.gro -e
> >> bench.edr -g bench.log -nsteps 1 -quiet
> >>>
> >>>
> >>>
> >>> and here are parts of the bench.log:
> >>>
> >>> Log file opened on Mon Sep 29 08:56:38 2014
> >>> Host: node-e1-67 pid: 24470 rank ID: 0 number of ranks: 48
> >>> GROMACS: gmx mdrun, VERSION 5.0.1
> >>>
> >>> GROMACS is written by:
> >>> Emile Apol Rossen Apostolov Herman J.C. Berendsen Par
> Bjelkmar
> >>> Aldert van Buuren Rudi van Drunen Anton Feenstra Sebastian
> >> Fritsch
> >>> Gerrit Groenhof Christoph Junghans Peter Kasson Carsten
> Kutzner
> >>> Per Larsson Justin A. Lemkul Magnus Lundborg Pieter
> >> Meulenhoff
> >>> Erik Marklund Teemu Murtola Szilard Pall Sander Pronk
> >>> Roland Schulz Alexey Shvetsov Michael Shirts Alfons Sijbers
> >>> Peter Tieleman Christian Wennberg Maarten Wolf
> >>> and the project leaders:
> >>> Mark Abraham, Berk Hess, Erik Lindahl, and David van der Spoel
> >>>
> >>> Copyright (c) 1991-2000, University of Groningen, The Netherlands.
> >>> Copyright (c) 2001-2014, The GROMACS development team at
> >>> Uppsala University, Stockholm University and
> >>> the Royal Institute of Technology, Sweden.
> >>> check out http://www.gromacs.org for more information.
> >>>
> >>> GROMACS is free software; you can redistribute it and/or modify it
> >>> under the terms of the GNU Lesser General Public License
> >>> as published by the Free Software Foundation; either version 2.1
> >>> of the License, or (at your option) any later version.
> >>>
> >>> GROMACS: gmx mdrun, VERSION 5.0.1
> >>> Executable: /home/apps/Logiciels/gromacs/gromacs-5.0.1/bin/gmx_mpi
> >>> Library dir:
> >> /home/apps/Logiciels/gromacs/gromacs-5.0.1/share/gromacs/top
> >>> Command line:
> >>> mdrun_mpi -npme 11 -s ../eq_nvt/1ZG4_nvt_bench00.tpr -resetstep 100 -o
> >> bench.trr -x bench.xtc -cpo bench.cpt -c bench.gro -e bench.edr -g
> >> bench.log -nsteps 1 -quiet
> >>>
> >>> Gromacs version: VERSION 5.0.1
> >>> Precision: single
> >>> Memory model: 64 bit
> >>> MPI library: MPI
> >>> OpenMP support: enabled
> >>> GPU support: disabled
> >>> invsqrt routine: gmx_software_invsqrt(x)
> >>> SIMD instructions: SSE4.1
> >>> FFT library: fftw-3.3.3-sse2
> >>> RDTSCP usage: enabled
> >>> C++11 compilation: enabled
> >>> TNG support: enabled
> >>> Tracing support: disabled
> >>> Built on: Tue Sep 23 09:58:07 EDT 2014
> >>> Built by: rqchpbib at briaree1 [CMAKE]
> >>> Build OS/arch: Linux 2.6.32-71.el6.x86_64 x86_64
> >>> Build CPU vendor: GenuineIntel
> >>> Build CPU brand: Intel(R) Xeon(R) CPU X5650 @ 2.67GHz
> >>> Build CPU family: 6 Model: 44 Stepping: 2
> >>> Build CPU features: aes apic clfsh cmov cx8 cx16 htt lahf_lm mmx msr
> >> nonstop_tsc pcid pclmuldq pdcm pdpe1gb popcnt pse rdtscp sse2 sse3
> sse4.1
> >> sse4.2 ssse3
> >>> C compiler: /RQusagers/apps/Logiciels/gcc/4.8.1/bin/gcc GNU
> 4.8.1
> >>> C compiler flags: -msse4.1 -Wno-maybe-uninitialized -Wextra
> >> -Wno-missing-field-initializers -Wno-sign-compare -Wpointer-arith -Wall
> >> -Wno-unused -Wunused-value -Wunused-parameter -fomit-frame-pointer
> >> -funroll-all-loops -fexcess-precision=fast -Wno-array-bounds -O3
> -DNDEBUG
> >>> C++ compiler: /RQusagers/apps/Logiciels/gcc/4.8.1/bin/g++ GNU
> 4.8.1
> >>> C++ compiler flags: -msse4.1 -std=c++0x -Wextra
> >> -Wno-missing-field-initializers -Wpointer-arith -Wall
> -Wno-unused-function
> >> -fomit-frame-pointer -funroll-all-loops -fexcess-precision=fast
> >> -Wno-array-bounds -O3 -DNDEBUG
> >>> Boost version: 1.55.0 (internal)
> >>>
> >>>
> >>> ....
> >>>
> >>>
> >>> n = 0
> >>> E-zt:
> >>> n = 0
> >>> swapcoords = no
> >>> adress = FALSE
> >>> userint1 = 0
> >>> userint2 = 0
> >>> userint3 = 0
> >>> userint4 = 0
> >>> userreal1 = 0
> >>> userreal2 = 0
> >>> userreal3 = 0
> >>> userreal4 = 0
> >>> grpopts:
> >>> nrdf: 10175.6 70836.4
> >>> ref-t: 304.65 304.65
> >>> tau-t: 0.5 0.5
> >>> annealing: Single Single
> >>> annealing-npoints: 4 4
> >>> annealing-time [0]: 0.0 200.0 300.0
> 750.0
> >>> annealing-temp [0]: 10.0 100.0 100.0
> 304.6
> >>> annealing-time [1]: 0.0 200.0 300.0
> 750.0
> >>> annealing-temp [1]: 10.0 100.0 100.0
> 304.6
> >>> acc: 0 0 0
> >>> nfreeze: N N N
> >>> energygrp-flags[ 0]: 0
> >>>
> >>> Overriding nsteps with value passed on the command line: 1 steps, 0.002
> >> ps
> >>>
> >>>
> >>> Initializing Domain Decomposition on 48 ranks
> >>> Dynamic load balancing: auto
> >>> Will sort the charge groups at every domain (re)decomposition
> >>> Initial maximum inter charge-group distances:
> >>> two-body bonded interactions: 0.422 nm, LJ-14, atoms 1444 1452
> >>> multi-body bonded interactions: 0.422 nm, Proper Dih., atoms 1444 1452
> >>> Minimum cell size due to bonded interactions: 0.464 nm
> >>> Maximum distance for 5 constraints, at 120 deg. angles, all-trans:
> 0.218
> >> nm
> >>> Estimated maximum distance required for P-LINCS: 0.218 nm
> >>>
> >>> -------------------------------------------------------
> >>> Program mdrun_mpi, VERSION 5.0.1
> >>> Source code file:
> >>
> /RQusagers/rqchpbib/stubbsda/gromacs-5.0.1/src/gromacs/mdlib/domdec_setup.c,
> >> line: 728
> >>>
> >>> Fatal error:
> >>> The number of ranks you selected (37) contains a large prime factor 37.
> >> In most cases this will lead to bad performance. Choose a number with
> >> smaller prime factors or set the decomposition (option -dd) manually.
> >>> For more information and tips for troubleshooting, please check the
> >> GROMACS
> >>> website at http://www.gromacs.org/Documentation/Errors
> >>> -------------------------------------------------------
> >>> -----Original Message-----
> >>> From: gromacs.org_gmx-users-bounces at maillist.sys.kth.se [mailto:
> >> gromacs.org_gmx-users-bounces at maillist.sys.kth.se] On Behalf Of Carsten
> >> Kutzner
> >>> Sent: Montag, 29. September 2014 15:23
> >>> To: gmx-users at gromacs.org
> >>> Subject: Re: [gmx-users] g_tune_pme_mpi is not compatible to mdrun_mpi
> >>>
> >>> Hi,
> >>>
> >>> is this the only output?
> >>>
> >>> Don't you get a perf.out file that lists which settings are optimal?
> >>>
> >>> What exactly was the command line you used?
> >>>
> >>> Carsten
> >>>
> >>>
> >>> On 29 Sep 2014, at 15:01, Ebert Maximilian <m.ebert at umontreal.ca>
> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> I just tried that and I got the following error message (bench.log).
> >> Any idea what could be wrong?
> >>>>
> >>>> Thank you very much,
> >>>>
> >>>> Max
> >>>>
> >>>> Initializing Domain Decomposition on 48 ranks Dynamic load balancing:
> >>>> auto Will sort the charge groups at every domain (re)decomposition
> >>>> Initial maximum inter charge-group distances:
> >>>> two-body bonded interactions: 0.422 nm, LJ-14, atoms 1444 1452
> >>>> multi-body bonded interactions: 0.422 nm, Proper Dih., atoms 1444 1452
> >>>> Minimum cell size due to bonded interactions: 0.464 nm Maximum
> >>>> distance for 5 constraints, at 120 deg. angles, all-trans: 0.218 nm
> >>>> Estimated maximum distance required for P-LINCS: 0.218 nm
> >>>>
> >>>> -------------------------------------------------------
> >>>> Program mdrun_mpi, VERSION 5.0.1
> >>>> Source code file:
> >>>> /RQusagers/rqchpbib/stubbsda/gromacs-5.0.1/src/gromacs/mdlib/domdec_se
> >>>> tup.c, line: 728
> >>>>
> >>>> Fatal error:
> >>>> The number of ranks you selected (37) contains a large prime factor
> 37.
> >> In most cases this will lead to bad performance. Choose a number with
> >> smaller prime factors or set the decomposition (option -dd) manually.
> >>>> For more information and tips for troubleshooting, please check the
> >>>> GROMACS website at http://www.gromacs.org/Documentation/Errors
> >>>> -------------------------------------------------------
> >>>>
> >>>> -----Original Message-----
> >>>> From: gromacs.org_gmx-users-bounces at maillist.sys.kth.se
> >>>> [mailto:gromacs.org_gmx-users-bounces at maillist.sys.kth.se] On Behalf
> >>>> Of Carsten Kutzner
> >>>> Sent: Donnerstag, 25. September 2014 19:29
> >>>> To: gmx-users at gromacs.org
> >>>> Subject: Re: [gmx-users] g_tune_pme_mpi is not compatible to mdrun_mpi
> >>>>
> >>>> Hi,
> >>>>
> >>>> don't invoke g_tune_pme with 'mpirun', because it is a serial
> >> executable that itself invokes parallel MD runs for testing.
> >>>>
> >>>> use
> >>>> export MDRUN=mdrun_mpi
> >>>>
> >>>> g_tune_pme -np 24 -s 1ZG4_nvt.tpr -launch
> >>>>
> >>>> see also
> >>>>
> >>>> g_tune_pme -h
> >>>>
> >>>> You may need to recompile g_tune_pme without MPI enabled (depends on
> >>>> your queueing system)
> >>>>
> >>>> Best,
> >>>> Carsten
> >>>>
> >>>>
> >>>> On 25 Sep 2014, at 15:10, Ebert Maximilian <m.ebert at umontreal.ca>
> >> wrote:
> >>>>
> >>>>> Dear list,
> >>>>>
> >>>>> I tried using g_tune_pme_mpi with the command:
> >>>>>
> >>>>> mpirun -np 24 g_tune_pme_mpi -np 24 -s 1ZG4_nvt.tpr -launch
> >>>>>
> >>>>> on GROMACS 5.0.1 but I get the following error message:
> >>>>> ---------------------------------------------------------------------
> >>>>> -
> >>>>> ---- mpirun was unable to launch the specified application as it
> >>>>> could not find an executable:
> >>>>>
> >>>>> Executable: mdrun
> >>>>> Node: xxxx
> >>>>>
> >>>>> while attempting to start process rank 0.
> >>>>> ---------------------------------------------------------------------
> >>>>> -
> >>>>> ----
> >>>>> 24 total processes failed to start
> >>>>>
> >>>>>
> >>>>> Any idea why this is? Shouldn't g_tune_pme_mpi call mdrun_mpi
> instead?
> >>>>>
> >>>>> Thank you very much,
> >>>>>
> >>>>> Max
> >>>>> --
> >>>>> Gromacs Users mailing list
> >>>>>
> >>>>> * Please search the archive at
> >> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> >> posting!
> >>>>>
> >>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >>>>>
> >>>>> * For (un)subscribe requests visit
> >>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users
> or
> >> send a mail to gmx-users-request at gromacs.org.
> >>>>
> >>>>
> >>>> --
> >>>> Dr. Carsten Kutzner
> >>>> Max Planck Institute for Biophysical Chemistry Theoretical and
> >>>> Computational Biophysics Am Fassberg 11, 37077 Goettingen, Germany
> >>>> Tel. +49-551-2012313, Fax: +49-551-2012302
> >>>> http://www.mpibpc.mpg.de/grubmueller/kutzner
> >>>> http://www.mpibpc.mpg.de/grubmueller/sppexa
> >>>>
> >>>> --
> >>>> Gromacs Users mailing list
> >>>>
> >>>> * Please search the archive at
> >> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> >> posting!
> >>>>
> >>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >>>>
> >>>> * For (un)subscribe requests visit
> >>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> >> send a mail to gmx-users-request at gromacs.org.
> >>>> --
> >>>> Gromacs Users mailing list
> >>>>
> >>>> * Please search the archive at
> >> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> >> posting!
> >>>>
> >>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >>>>
> >>>> * For (un)subscribe requests visit
> >>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> >> send a mail to gmx-users-request at gromacs.org.
> >>>
> >>>
> >>> --
> >>> Dr. Carsten Kutzner
> >>> Max Planck Institute for Biophysical Chemistry Theoretical and
> >> Computational Biophysics Am Fassberg 11, 37077 Goettingen, Germany Tel.
> >> +49-551-2012313, Fax: +49-551-2012302
> >> http://www.mpibpc.mpg.de/grubmueller/kutzner
> >>> http://www.mpibpc.mpg.de/grubmueller/sppexa
> >>>
> >>> --
> >>> Gromacs Users mailing list
> >>>
> >>> * Please search the archive at
> >> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> >> posting!
> >>>
> >>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >>>
> >>> * For (un)subscribe requests visit
> >>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> >> send a mail to gmx-users-request at gromacs.org.
> >>> --
> >>> Gromacs Users mailing list
> >>>
> >>> * Please search the archive at
> >> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> >> posting!
> >>>
> >>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >>>
> >>> * For (un)subscribe requests visit
> >>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> >> send a mail to gmx-users-request at gromacs.org.
> >>
> >>
> >> --
> >> Dr. Carsten Kutzner
> >> Max Planck Institute for Biophysical Chemistry
> >> Theoretical and Computational Biophysics
> >> Am Fassberg 11, 37077 Goettingen, Germany
> >> Tel. +49-551-2012313, Fax: +49-551-2012302
> >> http://www.mpibpc.mpg.de/grubmueller/kutzner
> >> http://www.mpibpc.mpg.de/grubmueller/sppexa
> >>
> >> --
> >> Gromacs Users mailing list
> >>
> >> * Please search the archive at
> >> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> >> posting!
> >>
> >> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >>
> >> * For (un)subscribe requests visit
> >> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> >> send a mail to gmx-users-request at gromacs.org.
> >>
> > --
> > Gromacs Users mailing list
> >
> > * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
> >
> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >
> > * For (un)subscribe requests visit
> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
>
>
> --
> Dr. Carsten Kutzner
> Max Planck Institute for Biophysical Chemistry
> Theoretical and Computational Biophysics
> Am Fassberg 11, 37077 Goettingen, Germany
> Tel. +49-551-2012313, Fax: +49-551-2012302
> http://www.mpibpc.mpg.de/grubmueller/kutzner
> http://www.mpibpc.mpg.de/grubmueller/sppexa
>
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
>
More information about the gromacs.org_gmx-users
mailing list