[gmx-developers] automated performance testing

[Roland Schulz]

On Fri, Oct 17, 2014 at 12:56 PM, Szilárd Páll <pall.szilard at gmail.com>
wrote:

> On Fri, Oct 17, 2014 at 2:56 AM, Roland Schulz <roland at utk.edu> wrote:
> >
> >
> > On Thu, Oct 16, 2014 at 7:27 PM, Szilárd Páll <pall.szilard at gmail.com>
> > wrote:
> >>
> >> Hi,
> >>
> >> There have been attempts at putting together a benchmark suite, the
> >> first one quite a few years ago resulting in the gmxbench.sh script
> >> and few input systems; more additions have been made a couple of
> >> months ago, everything is here: git.gromacs.org/benchmarks.git
> >
> >
> > I get
> > fatal: remote error: access denied or repository not exported:
> > /benchmarks.git
> >
> > Is this not yet readable to all developers?
>
> Looks like it's not, I was not aware of that.
>
> > Any reason this isn't in gerrit?
>
> No idea, Rossen was managing this server, so he would probably be the
> right person to answer these questions.
>
> IMO git is not the best place for gigabytes of static data, so gerrit
> (unless it has improved in recent versions) will likely be even less
> appropriate.
>

I think a good solution is to put large files on some cloud storage and
have the link to it in git/gerrit.

Roland


>
> Cheers,
> --
> Szilárd
>
>
> > Roland
> >
> >>
> >>
> >>
> >> On Tue, Sep 30, 2014 at 6:22 PM, Mark Abraham <mark.j.abraham at gmail.com
> >
> >> wrote:
> >> > Hi,
> >> >
> >> > Cherry-picking Michael's email into its own thread:
> >> >
> >> >> Is there a plan (long term) to do (essentially) automated performance
> >> >> tests so that we
> >> >> can perform consistent(ish) checks for new changes in code, then post
> >> >> the
> >> >> results in an
> >> >> easy(ish) way to interpret for others?
> >> >
> >> > There's no organized plan.  I've lately been trying to organize a
> >> > dedicated
> >> > machine here so we can start to do some of this - had we had it and
> the
> >> > right kinds of tests then various bugs would not have gone unnoticed.
> >>
> >> While a machine is useful, I think the what and how to benchmarks are
> >> to clarify first; given the difficulties in concretizing the benchmark
> >> setup in the past, these aspects require attention earlier rather than
> >> later, I think. Some of the questions that previous attempts brought
> >> up:
> >> - facilitate comparing results (to other codes or older version of
> >> GROMACS) while avoiding the pitfall of using of "smallest" common
> >> denominator features/algorithms (like JACC or the STFC benchmarks);
> >> - test algorithms or functionality: one may be interested in the
> >> algorithmic performance while others want to know how fast can one
> >> compute X.
> >>
> >> > In
> >> > principle, people could run that test suite on their own hardware, of
> >> > course.
> >>
> >> I think the best would be if many people ran on many different hardware.
> >>
> >> However, I think reproducibility is quite tricky. It can only be
> >> ensured if we get a couple of identical machines, set them up with
> >> identical software and avoid most upgrades (e.g. kernel, libc[++],
> >> etc.) that can affect performance, keeping a machine as backup to
> >> avoid having to rerun all reference runs when the hardware breaks.
> >> Even this will only ensure observing performance on one particular
> >> piece of hardware with the set of algorithms, parallelization,
> >> optimizations actually used.
> >>
> >> > One option I've been toying with lately is dumping the mdrun
> performance
> >> > data matrix to XML (or something) so that some existing plotting
> >> > machinery
> >> > can show the trend over time (and also observe a per-commit delta
> large
> >> > enough to vote -1 on Jenkins).
> >>
> >> Isn't per commit is an overkill an ill-scaling setup? I think it's
> >> better to do less frequent (e.g. weekly) and per-reqeust performance
> >> regression tests. Proper testing anyway requires running dozens of
> >> combinations of input+launch configurations on several platforms. It's
> >> not a fun task, I know because I've been doing quite extensive
> >> performance testing semi-manually.
> >>
> >> --
> >> Szilárd
> >>
> >> > I also mean to have a poke around with
> >> > http://www.phoromatic.com/ to see if maybe it already has
> infrastructure
> >> > we
> >> > could use.
> >> >
> >> > Mark
> >> >
> >> > --
> >> > Gromacs Developers mailing list
> >> >
> >> > * Please search the archive at
> >> > http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List
> before
> >> > posting!
> >> >
> >> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >> >
> >> > * For (un)subscribe requests visit
> >> >
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers
> >> > or
> >> > send a mail to gmx-developers-request at gromacs.org.
> >> --
> >> Gromacs Developers mailing list
> >>
> >> * Please search the archive at
> >> http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List before
> >> posting!
> >>
> >> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >>
> >> * For (un)subscribe requests visit
> >> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers
> or
> >> send a mail to gmx-developers-request at gromacs.org.
> >
> >
> >
> >
> > --
> > ORNL/UT Center for Molecular Biophysics cmb.ornl.gov
> > 865-241-1537, ORNL PO BOX 2008 MS6309
> >
> > --
> > Gromacs Developers mailing list
> >
> > * Please search the archive at
> > http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List before
> > posting!
> >
> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >
> > * For (un)subscribe requests visit
> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers
> or
> > send a mail to gmx-developers-request at gromacs.org.
> --
> Gromacs Developers mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers
> or send a mail to gmx-developers-request at gromacs.org.
>



-- 
ORNL/UT Center for Molecular Biophysics cmb.ornl.gov
865-241-1537, ORNL PO BOX 2008 MS6309
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://maillist.sys.kth.se/pipermail/gromacs.org_gmx-developers/attachments/20141017/ee0d91eb/attachment.html>