[gmx-users] Benchmarking gromacs over large number of cores

[Roland Schulz]

On Thu, Apr 28, 2011 at 10:05 AM, Bruno Monnet <bruno.monnet at hp.com> wrote:

>  Hi,
>
> I'm not really a Gromacs user, but I'm currently benchmarking Gromacs 4.5.4
> on a large cluster. It seems that my communication (PME) is really high and
> gromacs keeps complaining for more PME nodes :
>
>    Average load imbalance: 113.6 %
>  Part of the total run time spent waiting due to load imbalance: 3.3 %
>  Steps where the load balancing was limited by -rdd, -rcon and/or -dds: X 9
> % Y 9 % Z 9 %
>  Average PME mesh/force load: 3.288
>  Part of the total run time spent waiting due to PP/PME imbalance: 32.6 %
>
> NOTE: 32.6 % performance was lost because the PME nodes
>       had more work to do than the PP nodes.
>       You might want to increase the number of PME nodes
>       or increase the cut-off and the grid spacing.
>
>
> I can't modify the original dataset as I only have the TPR file. I switched
> from dlb yes -> dlb auto since it seems to have trouble with more than 6000
> / 8000 cores.
>
You can set the number of PME nodes with -npme. All other nodes are used for
the particle-particle computations (PP).There is also a tool called
g_tune_pme which optimizes it automatically for you.

On that many cores you might see a significant speed-up by using the
prerelease version of GROMACS 4.6. You can obtain that from git using the
branch "threading". It uses threading for the PME nodes. The number of
threads used by the PME nodes is set with the environment variable
GMX_PME_NTHREADS. The total number of cores should be equal to the number of
PP nodes + GMX_PME_NTHREADS * number of PP nodes. Let me know if you try it
- I would be interested in feedback.

I tried to add " -gcom " parameter. This speedup the computation. This
> parameter is not really explained in the Gromacs documentation. Could you
> give me some advice on how I could use it ?
>
It defines how often e.g. the total energy is computed which is important
for pressure and temperature coupling. As long as you don't set it to higher
than 10 it shouldn't affect the accuracy significantly. But it does effect
it minimally thus you should document that in your results.

Roland


>
> Best regards,
> Bruno Monnet
>
>
> --
> gmx-users mailing list    gmx-users at gromacs.org
> http://lists.gromacs.org/mailman/listinfo/gmx-users
> Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/Search before posting!
> Please don't post (un)subscribe requests to the list. Use the
> www interface or send it to gmx-users-request at gromacs.org.
> Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>



-- 
ORNL/UT Center for Molecular Biophysics cmb.ornl.gov
865-241-1537, ORNL PO BOX 2008 MS6309
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://maillist.sys.kth.se/pipermail/gromacs.org_gmx-users/attachments/20110428/0464728b/attachment.html>