[gmx-users] Can't do parallell free energy calculation with couple_intramol=no -- workaround?
Szilárd Páll
pall.szilard at gmail.com
Mon Mar 14 17:50:20 CET 2016
Hi Michael,
Great, thanks for the feedback!
Good point about OMP_NUM_THREADS being set in the environment; it is often
possible to set the "width" or "depth" of the ranks in the scheduler, which
will often set the desired OMP_NUM_THREADS (instead of assuming 1) which
makes sure that no override is necessary.
Also note that with "wide" OpenMP-threaded processes, thread pinning is
particularly important, so do make sure that you either get this as default
behavior (which is the case if no process affinity is set externally and
all cores/threads in the code are being used), otherwise pass "-pin on" to
mdrun.
Last, as mentioned before you may want to give a try running PME on
separate cores, a new feature in 5.1, e.g.
mpirun -np 2 mdrun_mpi -npme 1 -ntomp N -ntomp_pme M
or
mdrun -ntmpi 2 -npme 1 -ntomp N -ntomp_pme M
Cheers,
--
Szilárd
On Mon, Mar 14, 2016 at 5:33 PM, Michael Daily <mdaily.work at gmail.com>
wrote:
> Hi Szilard,
>
> I tested running with OpenMP parallelization in gromacs 5.1.2 and it solves
> the problem. Thanks for your advice!
>
> I also found that it is necessary to do deactivate the OMP_NUM_THREADS
> environment variable "unset OMP_NUM_THREADS" prior to running gromacs,
> since if this variable is set to 1 on the node gromacs will default to
> using MPI parallelization and the domain decomposition error will result.
>
> Thanks,
>
> ====================================
> Michael D. Daily
> Research Scientist, Biochemistry and Molecular Biology
> University of Texas Medical Branch at Galveston
> 509-713-0726
>
>
> On Thu, Mar 10, 2016 at 2:04 PM, Michael Daily <mdaily.work at gmail.com>
> wrote:
>
> > Hi Szilard,
> >
> > Thanks for the advice. I will try out v5.1 accordingly. Re attachments,
> > I've now copied my mdp and md output below:
> >
> > ### mdp ###
> > ; Run control
> > integrator = sd ; Langevin dynamics
> > tinit = 0
> > dt = 0.002
> > nsteps = 1000000 ; 2 ns
> > nstcomm = 100
> >
> > define=-DPOSRES
> >
> > ; Output control
> > nstxout = 0
> > nstvout = 0
> > nstfout = 0
> > nstlog = 5000
> > nstenergy = 5000
> > nstxtcout = 5000
> > energygrps = Protein SOL
> > ;energygrp-excl=Protein Protein
> >
> > ; Neighborsearching and short-range nonbonded interactions
> > cutoff-scheme = verlet
> > ;cutoff-scheme=group
> > nstlist = 20
> > ns_type = grid
> > pbc = xyz
> > rlist = 1.2
> > ; Electrostatics
> > coulombtype = PME
> > rcoulomb = 1.2
> > ; van der Waals
> > vdwtype = cutoff
> > vdw-modifier = potential-switch
> > rvdw-switch = 1.0
> > rvdw = 1.2
> > ; Apply long range dispersion corrections for Energy and Pressure
> > DispCorr = EnerPres
> > ; Spacing for the PME/PPPM FFT grid
> > fourierspacing = 0.12
> > ; EWALD/PME/PPPM parameters
> > pme_order = 6
> > ewald_rtol = 1e-06
> > epsilon_surface = 0
> > ; Temperature coupling
> > ; tcoupl is implicitly handled by the sd integrator
> > tc_grps = system
> > tau_t = 1.0
> > ref_t = 300
> > ; Pressure coupling is on for NPT
> > Pcoupl = Parrinello-Rahman
> > Pcoupltype = isotropic
> > refcoord-scaling = com
> > tau_p = 1.0
> > compressibility = 4.5e-05
> > ref_p = 1.0
> > ; Free energy control stuff
> > free_energy = yes
> > init_lambda_state = my_init_lambda
> > delta_lambda = 0
> > calc_lambda_neighbors = 1 ; only immediate neighboring windows
> > ; Vectors of lambda specified here
> > ; Each combination is an index that is retrieved from init_lambda_state
> > for each simulation
> > coul_lambdas = 0.0000 0.0625 0.1250 0.1875 0.2500 0.3125 0.3750 0.4375
> > 0.5000 0.5625 0.6250 0.6875 0.7500 0.8125 0.8750 0.9375 1.0000
> >
> > ; Options for the decoupling
> > sc-alpha = 0.5
> > sc-coul = no ; linear interpolation of Coulomb
> > (none in this case)
> > sc-power = 1.0
> > sc-r-power = 6
> > sc-sigma = 0.3
> >
> > couple-moltype = Protein ; name of moleculetype to (de)couple
> > couple-lambda0 = vdw ; only van der Waals interactions
> > couple-lambda1 = vdw-q ; everything
> > couple-intramol = no
> > ;couple-intramol=yes
> > nstdhdl=5000
> > separate-dhdl-file=no ;more efficient
> >
> > ; Do not generate velocities
> > gen_vel = no
> > ; options for bonds
> > constraints = all-bonds ; we only have C-H bonds here
> > ; Type of constraint algorithm
> > constraint-algorithm = lincs
> > ; Constrain the starting configuration
> > ; since we are continuing from NPT
> > continuation = yes
> > ; Highest order in the expansion of the constraint coupling matrix
> > ;lincs-order = 12
> > table-extension=1.5
> >
> > ### output (part with the error) ###
> > Initializing Domain Decomposition on 16 ranks
> > Dynamic load balancing: auto
> > Will sort the charge groups at every domain (re)decomposition
> > Initial maximum inter charge-group distances:
> > two-body bonded interactions: 2.343 nm, LJC Pairs NB, atoms 4 45
> > multi-body bonded interactions: 0.383 nm, Proper Dih., atoms 1 9
> > Minimum cell size due to bonded interactions: 2.577 nm
> > Maximum distance for 5 constraints, at 120 deg. angles, all-trans: 0.728
> nm
> > Estimated maximum distance required for P-LINCS: 0.728 nm
> > Using 0 separate PME ranks, as there are too few total
> > ranks for efficient splitting
> > Scaling the initial minimum size with 1/0.8 (option -dds) = 1.25
> > Optimizing the DD grid for 16 cells with a minimum initial size of 3.222
> nm
> > The maximum allowed number of cells is: X 0 Y 0 Z 1
> >
> > -------------------------------------------------------
> > Program mdrun, VERSION 5.0.7
> > Source code file:
> > /home1/01777/mdaily/software/gromacs-5.0.7/src/gromacs/mdlib/domdec.c,
> > line: 6902
> >
> > Fatal error:
> > There is no domain decomposition for 16 ranks that is compatible with the
> > given box and a minimum cell size of 3.22187 nm
> > Change the number of ranks or mdrun option -rdd or -dds
> > Look in the log file for details on the domain decomposition
> > For more information and tips for troubleshooting, please check the
> GROMACS
> > website at http://www.gromacs.org/Documentation/Errors
> > -------------------------------------------------------
> >
> >
> >
> > Thanks,
> >
> > ====================================
> > Michael D. Daily
> > Research Scientist, Biochemistry and Molecular Biology
> > University of Texas Medical Branch at Galveston
> > 509-713-0726
> >
> >
> > On Thu, Mar 10, 2016 at 1:55 PM, Szilárd Páll <pall.szilard at gmail.com>
> > wrote:
> >
> >> On Thu, Mar 10, 2016 at 8:14 PM, Michael Daily <mdaily.work at gmail.com>
> >> wrote:
> >>
> >> > Hi,
> >> >
> >> > I am trying to do a solvation free energy calculation on a small
> >> peptide in
> >> > gromacs 5.0. The peptide is in an extended conformation about 2.5 nm
> >> long,
> >> > and I use position restraints on the heavy atoms to keep it in this
> >> > conformation. Due to the nature of the calculation I set
> >> > "couple_intramol=no" in the mdp file (attached for lambda=0).
> >> >
> >> > However, due to the way couple_intramol is implemented in gromacs
> >> > (long-range exclusions), the longest bonded interaction is as long as
> >> the
> >> > molecule, leading to a minimum cell size of over 3 nm, which makes
> >> domain
> >> > decomposition all but impossible (see the attached log file). Is
> there
> >> a
> >> > way to limit long-range exclusions and still get reasonable charging
> >> free
> >> > energies? I have a few ideas but would like some opinions on whether
> >> they
> >> > are physically sound:
> >> >
> >> > 1) change "couple_intramol" to yes and generate intramolecular
> >> exclusions
> >> > manually with some cutoff. Since rlist/rcoulomb/rvdw are 1.2 nm for
> >> this
> >> > simulation, I might use a cutoff of 1.5 to provide a buffer.
> >> > 2) Use "energygrp-excl = Protein Protein" to turn off nonbonded
> >> > interactions within the peptide. The only problem with this is it
> >> forces
> >> > the use of group instead of Verlet cutoff scheme which seems to slow
> >> > performance.
> >> > 3) Revert to Gromacs 4.7 and use particle decomposition
> >> >
> >>
> >> I guess you mean v4.6.7. As far as I know there is the decomposition
> >> should
> >> not interfere with the physics. However, you could instead consider
> using
> >> 5.1 with multi-threading. Unless something is unsupported or you happen
> to
> >> stumble into code not OpenMP-parallelized, you should be able to run on
> up
> >> to 24-32 cores without domain-decomposition (also note that 5.1 support
> >> running separate PME without DD).
> >>
> >>
> >> > The log file from my (attempted) simulation with these parameters is
> >> also
> >> > attached.
> >> >
> >>
> >> The user's list does not accept attachments, so you'll have to share
> links
> >> to files uploaded somewhere else.
> >>
> >> --
> >> Szilárd
> >>
> >>
> >>
> >> >
> >> > Thanks,
> >> >
> >> > ====================================
> >> > Michael D. Daily
> >> > Research Scientist, Biochemistry and Molecular Biology
> >> > University of Texas Medical Branch at Galveston
> >> > 509-713-0726
> >> >
> >> > --
> >> > Gromacs Users mailing list
> >> >
> >> > * Please search the archive at
> >> > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> >> > posting!
> >> >
> >> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >> >
> >> > * For (un)subscribe requests visit
> >> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> >> > send a mail to gmx-users-request at gromacs.org.
> >> >
> >> --
> >> Gromacs Users mailing list
> >>
> >> * Please search the archive at
> >> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> >> posting!
> >>
> >> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >>
> >> * For (un)subscribe requests visit
> >> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> >> send a mail to gmx-users-request at gromacs.org.
> >
> >
> >
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
>
More information about the gromacs.org_gmx-users
mailing list