[gmx-developers] Test Suite, CTest, CDash

[Esztermann, Ansgar]

On Aug 25, 2010, at 15:58 , Mark Abraham wrote:

> > a few years ago, I have been briefly involved in the gcc 
> > project. I was extremely impressed by the huge test suite and by 
> > the way it could be used to quickly detect regressions. Putting 
> > aside for the moment the fact that "correct behaviour" is much 
> > easier to define for a compiler than for a simulation engine, I 
> > think that gromacs would profit from a similar test suite.
> 
> Agreed. There are huge combinatorial and chaotic problems, though. I'd expect there must be some literature on dealing with the former.
> 
> Testing an MD implementation might use a non-trivial amount of computing resources. I've seen more than a few bugs that required a combination of three or more moderately unusual conditions to occur.  Those are the ones that regression suites are really useful for finding. However how do you store the reference results? If you don't store them, you have to recompute them, which requires having the reference version of the code compiled, and some compute time.

As far as the general setup of the build system is concerned, input and reference output should be stored as part of the test case. This would be a problem only if the amount of data becomes unwieldy -- but then again, one would hope that bug triggering inputs can be reduced to a manageable size. As for the computing time required, CTest has at least some support for test subsets, so one could exclude time-consuming tests during "casual" testing. For nightly tests, twelve hour runs or so are quite manageable. Of course, month-long test runs would still be a problem.

> So I'm going to sound a bit negative here, but I'm skeptical that it's possible to do a sound job of a MD regression suite.

I agree that there are formidable problems. Still, I'd like to work on this and see how far I get.

A.
-- 
Ansgar Esztermann
DV-Systemadministration
Max-Planck-Institut für biophysikalische Chemie, Abteilung 105