[gmx-users] why it is so slow in Blue gene?

[Justin A. Lemkul]

On 4/24/12 9:42 AM, Albert wrote:
> hello:
>
> I am running a 60,000 atom system with 128 core in a blue gene cluster. and it
> is only 1ns/day.... here is the script I used for submitting jobs:
>
> # @ job_name = gmx_test
> # @ class = kdm-large
> # @ error = gmx_test.err
> # @ output = gmx_test.out
> # @ wall_clock_limit = 00:20:00
> # @ job_type = bluegene
> # @ bg_size = 32
> # @ queue
> mpirun -exe /opt/gromacs/4.5.5/bin/mdrun_mpi_bg -args "-nosum -dlb yes -v -s npt
> _01.tpr -o npt_01.trr -cpo npt_01.cpt -g npt_01.log -launch -nt" -mode VN -np 128
>
>
>
> here is my npt.mdp
>
> title = NPT-01
> cpp = /usr/bin/cpp
> include =
> define = -DPOSRES -DPOSRES_POPE_HEAD
> integrator = md
> dt = 0.001
> nsteps = 5000000
> nstxout = 100000
> nstvout = 100000
> nstlog = 100000
> nstenergy = 50000
> nstxtcout = 50000
> xtc_grps =
> energygrps = Protein SOL ION
> nstcalcenergy = 10
> nstlist = 10
> nstcomm = 10
> comm_mode = Linear
> comm-grps = Protein_POPE Water_and_ions
> ns_type = grid
> rlist = 1.2
> rlistlong = 1.4
> vdwtype = Switch
> rvdw = 1.2
> rvdw_switch = 0.8
> coulombtype = pme
> rcoulomb = 1.2
> rcoulomb_switch = 0.0
> fourierspacing = 0.15
> pme_order = 6
> DispCorr = no
> tcoupl = V-rescale ;nose-hoover
> nhchainlength = 1
> tc-grps = Protein_POPE Water_and_ions
> tau_t = 0.1 0.1
> ref_t = 310 310
> Pcoupl = berendsen ;parrinello-rahman
> Pcoupltype = semiisotropic
> tau_p = 1.0
> compressibility = 4.5e-5 4.5e-5
> ref_p = 1.0 1.0
> pbc = xyz
> refcoord_scaling = com
> gen_vel = no
> optimize_fft = no
> constraints = hbonds
> constraint_algorithm = Lincs
>
>
> Does anybody have any advices?
>

The end of the log file will print information about where performance may have 
been lost.  For 60,000 atoms I would think that 128 cores is too many; you're 
sacrificing performance to communication overhead.  A good ballpark is 1000 
atoms/core.  A few quick benchmark calculations should give you a better idea on 
the setup for optimal performance.

-Justin

-- 
========================================

Justin A. Lemkul
Ph.D. Candidate
ICTAS Doctoral Scholar
MILES-IGERT Trainee
Department of Biochemistry
Virginia Tech
Blacksburg, VA
jalemkul[at]vt.edu | (540) 231-9080
http://www.bevanlab.biochem.vt.edu/Pages/Personal/justin

========================================