[gmx-developers] gmx performance-log database

[Szilárd Páll]

Hi,

As Berk said, we have considered creating a benchmark result
database/submission both for showing statistics on the web as well as
for performance regression testing. Actually, the task was even on a todo
list a while ago, but for various reasons it got dropped. The only result
was a very simple bash benchmark script which can be found here: git://
git.gromacs.org/benchmarks.git

IMHO the biggest barrier to having such a benchmark database/submission
system is the lack of standardized GROMACS benchmark input set. The rest
could be done with very little effort and pretty much anyone could help
with it.

There are several things that would need to be checked for reliable
> number though: correct input tpr, empty machine, etc.
> So we can't publish the numbers as reliable numbers.
>

Sure, but all that is doable and not rocket science, but
mostly straightforward software engineering. If we had
i) a standardized set of benchmark inputs and
ii) some simple working setup (or at least a design and a proof of concept
dumb implementation),
we could ask for help. Without these no one can help with the actual
implementation.

Cheers,
Szilárd


> We were thinking about having two databases, one "certified" one
> and one open one where everyone can submit.
>
> Cheers,
>
> Berk
>
>
> On 10/24/12 17:05 , Raf Ponsaerts wrote:
>
>> Dear gmx-developers,
>>
>> I don't know whether the following idea already has been proposed, but
>> here I go...
>>
>> Would it be useful to have an online database where people would be able
>> to submit their performance log after a run?
>>
>> It is probably not too difficult to automatically generate a small log
>> file, such as md.log that contains the input parameters, "mega-flops
>> accounting output" and the hardware specs (CPU-type, GPU-type, number of
>> nodes, cores, threads, OS, kernel, parameters gmx-compilation...) that
>> were used for the run. Otherwise, easily a script could be written that
>> extracts such data from md.log.
>>
>> Depending on the willingness of the gmx-community, such a
>> performance-logfile could be submitted to an online gmx-database.
>>
>> I guess that putting such data in a database (e.g. PostgreSQL) could be
>> of value to the developers, when statistical information easily can be
>> derived from such a database. Comparing side-by-side benchmarks is
>> always tricky issue (different hardware, OS, GMX versions, ...). But
>> still...
>>
>> Scripts that automatically analyze performance-data might also
>> facilitate or speed up the revision-process, e.g. validate the effect of
>> a git-commit if a developer keeps track of his performance data in a
>> database. It could also be useful to code-testers or production users
>> that would like to keep track - or get an automated overview while
>> tweaking parameters.
>>
>> Since good performance is relevant but not the most important issue,
>> do not take this proposal as some kind of feature-request!
>> I guess that the core-devel team does not really have the time to work
>> on this. But, there might also be other members (gmx-users) of the
>> gmx-community that wish to spend some time to implement this. So, I
>> would like to know how relevant and realistic you think this idea is.
>>
>> I'm curious to read your thoughts on this.
>>
>> regards,
>>
>> raf
>>
>> --------------------
>>
>> Raf Ponsaerts, Phd
>> KULeuven, Belgium
>>
>>
> --
> gmx-developers mailing list
> gmx-developers at gromacs.org
> http://lists.gromacs.org/**mailman/listinfo/gmx-**developers<http://lists.gromacs.org/mailman/listinfo/gmx-developers>
> Please don't post (un)subscribe requests to the list. Use the www
> interface or send it to gmx-developers-request@**gromacs.org<gmx-developers-request at gromacs.org>
> .
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://maillist.sys.kth.se/pipermail/gromacs.org_gmx-developers/attachments/20121024/3b1c76aa/attachment.html>