[gmx-developers] New Test Set

[Roland Schulz]

Hi,

we agreed that we would want to have a test set for 4.6 but so far we
haven't made any progress on it (as far as I know). I want to try to get
this work started by posting here a list of questions I have about the new
test set. Please add your own questions and answer any questions you can
(no need to try to answer all questions).

- Why do the current tests fail? Is it only because of different floating
point rounding or are there other problems? What's the best procedure to
find out why a test fails?
- Which tests should be part of the new test set?
- Should the current tests all be part of the new test set?
- How should the new test be implemented? Should the comparison with the
reference value be done in C (within mdrun), ctest script, python or perl?
- Should the new test execute mdrun for each test? Or should we somehow
(e.g. python wrapper or within mdrun) load the binary only once and run
many test per execution?
- What are the requirements for the new test set? E.g. how easy should it
be to see whats wrong when a test fails? Should the test support being run
under valgrind? Other?
- Do we have any other bugs which have to be solved before the test can be
implemented? E.g. is the problem with shared libraries solved? Are there
any open redmine issues related to the new test set?
- Should we have a policy that everyone who adds a feature also has to
provide tests covering those features?
- Should we have a conference call to discuss the test set? If yes when?
- Should we agree that we won't release 4.6 without the test set to give it
a high priority?
Roland

-- 
ORNL/UT Center for Molecular Biophysics cmb.ornl.gov
865-241-1537, ORNL PO BOX 2008 MS6309
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://maillist.sys.kth.se/pipermail/gromacs.org_gmx-developers/attachments/20120204/cbf7472d/attachment.html>