[gmx-users] grompp -np -shuffle with position restraints
Low Soo Mei
lowsm at bii.a-star.edu.sg
Mon May 12 18:24:26 CEST 2008
Hi Gromacs users,
When I run grompp for 16 nodes to prepare my file for parallel run on my
cluster, and also -shuffle to spread the load evenly over the nodes, I
find that the position restraints get shuffled as well. This is my
command line:
grompp -v -f heat.mdp -c min_end.pdb -p dmpc_npat_cluster.top -po
heattest4shuffleout.mdp -o heattest4shuffle.tpr -pp
heattest4shuffleout.top -np 16 -deshuf heattest4shuffle -shuffle
And this is my grompp output (in full, which is quite long, sorry) :
:-) G R O M A C S (-:
Gallium Rubidium Oxygen Manganese Argon Carbon Silicon
:-) VERSION 3.3.3 (-:
Written by David van der Spoel, Erik Lindahl, Berk Hess, and others.
Copyright (c) 1991-2000, University of Groningen, The Netherlands.
Copyright (c) 2001-2008, The GROMACS development team,
check out http://www.gromacs.org for more information.
This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation; either version 2
of the License, or (at your option) any later version.
:-) /usr/local/gromacs/bin-mpich/grompp (-:
Option Filename Type Description
------------------------------------------------------------
-f heat.mdp Input, Opt! grompp input file with MD parameters
-po heattest4shuffleout.mdp Output grompp input file with MD
parameters
-c min_end.pdb Input Generic structure: gro g96 pdb tpr tpb tpa
xml
-r conf.gro Input, Opt. Generic structure: gro g96 pdb tpr tpb tpa
xml
-rb conf.gro Input, Opt. Generic structure: gro g96 pdb tpr tpb tpa
xml
-n index.ndx Input, Opt. Index file
-deshufattest4shuffle.ndx Output, Opt! Index file
-p dmpc_npat_cluster.top Input Topology file
-pp heattest4shuffleout.top Output, Opt! Topology file
-o heattest4shuffle.tpr Output Generic run input: tpr tpb tpa xml
-t traj.trr Input, Opt. Full precision trajectory: trr trj
-e ener.edr Input, Opt. Generic energy: edr ene
Option Type Value Description
------------------------------------------------------
-[no]h bool no Print help info and quit
-nice int 0 Set the nicelevel
-[no]v bool yes Be loud and noisy
-time real -1 Take frame at or first after this time.
-np int 16 Generate statusfile for # nodes
-[no]shuffle bool yes Shuffle molecules over nodes
-[no]sort bool no Sort molecules according to X coordinate
-[no]rmvsbds bool yes Remove constant bonded interactions with virtual
sites
-load string Releative load capacity of each node on a
parallel machine. Be sure to use quotes around
the string, which should contain a number for
each node
-maxwarn int 10 Number of warnings after which input processing
stops
-[no]check14 bool no Remove 1-4 interactions without Van der Waals
-[no]zero bool no Set parameters for bonded interactions without
defaults to zero instead of generating an error
-[no]renum bool yes Renumber atomtypes and minimize number of
atomtypes
creating statusfile for 16 nodes...
checking input for internal consistency...
calling /usr/bin/cpp...
processing topology...
Generated 483 of the 1653 non-bonded parameter combinations
Generating 1-4 interactions: fudge = 1
Generated 663 of the 1653 1-4 parameter combinations
Excluding 3 bonded neighbours for DMP 128
turning all bonds into constraints...
Excluding 2 bonded neighbours for SOL 3655
turning all bonds into constraints...
Moltype DMP SOL #atoms
CPU 0 8 229 1055
CPU 1 8 229 1055
CPU 2 8 229 1055
CPU 3 8 229 1055
CPU 4 8 229 1055
CPU 5 8 229 1055
CPU 6 8 229 1055
CPU 7 8 228 1052
CPU 8 8 228 1052
CPU 9 8 228 1052
CPU 10 8 228 1052
CPU 11 8 228 1052
CPU 12 8 228 1052
CPU 13 8 228 1052
CPU 14 8 228 1052
CPU 15 8 228 1052
Made a shuffling table with 3783 entries [molecules]
processing coordinates...
Shuffling coordinates...
Entering shuffle_xv
double-checking input for internal consistency...
Reading position restraint coords from min_end.pdb
WARNING shuffling position restraints. Please scheck your results.
renumbering atomtypes...
converting bonded parameters...
# ANGLES: 6784
# PDIHS: 2176
# RBDIHS: 2560
# IDIHS: 384
# LJ14: 3712
# POSRES: 1408
# CONSTR: 5760
# SETTLE: 3655
Walking down the molecule graph to make shake-blocks
initialising group options...
processing index file...
Analysing residue names:
Opening library file /usr/local/gromacs/share/gromacs/top/aminoacids.dat
There are: 3783 OTHER residues
There are: 0 PROTEIN residues
There are: 0 DNA residues
Analysing Other...
Simulated annealing for group DMP: Single, 11 timepoints
Time (ps) Temperature (K)
0.0 0.0
100.0 32.3
200.0 64.6
300.0 96.9
400.0 129.2
500.0 161.5
600.0 193.8
700.0 226.1
800.0 258.4
900.0 290.7
1000.0- 323.0
Simulated annealing for group SOL: Single, 11 timepoints
Time (ps) Temperature (K)
0.0 0.0
100.0 32.3
200.0 64.6
300.0 96.9
400.0 129.2
500.0 161.5
600.0 193.8
700.0 226.1
800.0 258.4
900.0 290.7
1000.0- 323.0
Making dummy/rest group for Acceleration containing 16853 elements
Making dummy/rest group for Freeze containing 16853 elements
Making dummy/rest group for VCM containing 16853 elements
Number of degrees of freedom in T-Coupling group DMP is 11902.94
Number of degrees of freedom in T-Coupling group SOL is 21928.05
Making dummy/rest group for User1 containing 16853 elements
Making dummy/rest group for User2 containing 16853 elements
Making dummy/rest group for XTC containing 16853 elements
Making dummy/rest group for Or. Res. Fit containing 16853 elements
Making dummy/rest group for QMMM containing 16853 elements
T-Coupling has 2 element(s): DMP SOL
Energy Mon. has 2 element(s): DMP SOL
Acceleration has 1 element(s): rest
Freeze has 1 element(s): rest
User1 has 1 element(s): rest
User2 has 1 element(s): rest
VCM has 1 element(s): rest
XTC has 1 element(s): rest
Or. Res. Fit has 1 element(s): rest
QMMM has 1 element(s): rest
Checking consistency between energy and charge groups...
splitting topology...
Walking down the molecule graph to make shake-blocks
There are 7751 charge group borders and 3783 shake borders
There are 3783 total borders
Division over nodes in atoms:
1052 1052 1055 1052 1055 1052 1055 1052
1055 1052 1055 1052 1055 1052 1055 1052
writing run input file...
gcq#40: "Yeah, a Wuzz, Or a Jerk" (F. Black)
-------
The fun comes when I do the mpi mdrun on my cluster. From time 0.044 ps
(step 22) till the time I kill the run to put it out of its misery, I
get SETTLE errors. From time 0.090 ps (step 45) till the time I kill
the run, I get LINCS errors as well.
When I grompp the same files to run on 16 nodes without the -shuffle
flag, the position restraints are fine, the mpi mdrun is smooth, and in
my output trajectory I can see that the position restraints are
definitely working. Of course, since there is no -shuffle flag, the
load balancing is terrible. It takes 15 hours to complete, instead of
the 5.5 hours on a single node that I first tried. I'm not surprised
that it takes longer when spread out over 16 nodes because my cluster is
using gigabit ethernet to communicate. What's surprising is that the
grompp -shuffle flag kills my position restraints.
Using the -sort flag in addition to the -shuffle flag (just by adding
"-sort" to the above grompp command line) gives the same problem of
killing my position restraints.
Anyone has any ideas? My cluster is using 3.3.3 on intel xeon woodcrest
nodes. But I am not very sure if this is a 3.3.3 problem or a cluster
MPI problem, or some interaction between the two.
Many thanks for your advice, in advance.
Cheers,
Soo Mei
More information about the gromacs.org_gmx-users
mailing list