[gmx-developers] New Test Set

[David Bowman]

Roland, Erik, et.al.
I am not sure if my posting was well understood so I will give an example.

When one has an extremely complex piece of software that has few tests and
there is a need to  be constantly adding new features there are only a
couple of ways of testing it. It can not be well tested at the higher
levels only at the module level.

For example, I would liken the testing of GROMACS to a high level language
compiler designed with complex syntax, many modules and many options.
Compilers have language features, syntax that require runtime support and
finction libraries. They also have supporting library routines or support
programs. No one would ever try to make a test suite for high level use
cases. What is done is to create separate test programs that exercise
language syntax that can be tested without direct calls to internal modules
or directly to library routines (e.g. nonbonded, etc.). Separate tests
programs are written for things like object file creation. These tests have
'right' answers. No attempt is made to test 'use cases' except in the case
of priority 1 bugs. This reduces the scope of testing to a manageable and
defined level. It requires breaking the program into functional groups,
internal routines and external routines and support programs. This is not
difficult to do with GROMACS. GROMACS has .mdp options that amount to
'lanaguage' syntax, internal code routines, external programs and file
formats like a compiler. This is the test approach that I think should be
taken.

In cases where I have worked on compilers that were 'legacy' and mature
products we prioritized the removal of all serious P1 and P2 bugs and
adding new features. When new features were added developers
*started*adding higher level tests. In response to serious P1 bugs in
'broken generated programs' 'e.g. simulations' user provided code (or
simuations) were added to the test suite. Onces I inherited such a
compiler. We had many new features to add, many bugs, little documenation,
virtually no test suite and a short time to release. With 5 people we added
all the new features in 5 months, fixed all the priority 1 and 2
 bugs, and built test programs that exercised or called directly all key
core modules.

I hope this helps.

Cheers
David

 Sat, Feb 11, 2012 at 7:45 PM, Roland Schulz <roland at utk.edu> wrote:

>
>
>  On Sat, Feb 11, 2012 at 11:31 AM, Shirts, Michael (mrs5pt) <
> mrs5pt at eservices.virginia.edu> wrote:
>
>>  > 1) Low-level tests that specifically check the output for several
>> sets of
>> > input for a *module*, i.e. calling routines rather than running a
>> simulation.
>> > The point is that this will isolate errors either to a specific module,
>> or to
>> > modules it depends on. However, when those modules too should have
>> tests it
>> > will be a 5-min job to find what file+routine the bug is likely to be
>> in.
>>
>> I'm not exactly sure how this works.  If we test modules, we have to be
>> writing a bunch of new code that interacts with the modules directly, and
>> so
>> we may miss things that happen in actual simulation cases.  I sort of
>> favor
>> just actually running grompp and mdrun, because the errors that occur will
>> be the errors that people actually see. I haven't found an error yet that
>> is
>> particularly hard to isolate to a given file pretty quickly once it is
>> identified.   Perhaps for particular aspects things (testing that dozens
>> of
>> inner loops give consistent numbers) this makes sense, but I'm not sure it
>> makes sense for everything.  I'm not sure how you _just_ test pressure
>> control, for example.
>>
> One could read virial values from a data file and pass it to the pressure
> control. And then check that the pressure control is behaving as it should.
>
> I think unit tests would be very useful for the planned conversion to C++.
> We will have to convert individual modules one at a time and I assume that
> the overall program will be broken often. In that case unit tests might
> enable us to test individual modules without the overall programming
> working.
>
> Roland
>
>
>>
>> Best,
>> ~~~~~~~~~~~~
>> Michael Shirts
>> Assistant Professor
>> Department of Chemical Engineering
>> University of Virginia
>> michael.shirts at virginia.edu
>> (434)-243-1821
>>
>>
>>
>>
>> --
>> gmx-developers mailing list
>> gmx-developers at gromacs.org
>> http://lists.gromacs.org/mailman/listinfo/gmx-developers
>> Please don't post (un)subscribe requests to the list. Use the
>> www interface or send it to gmx-developers-request at gromacs.org.
>>
>>
>>
>>
>>
>
>
> --
> ORNL/UT Center for Molecular Biophysics cmb.ornl.gov
> 865-241-1537, ORNL PO BOX 2008 MS6309
>
> --
> gmx-developers mailing list
> gmx-developers at gromacs.org
> http://lists.gromacs.org/mailman/listinfo/gmx-developers
> Please don't post (un)subscribe requests to the list. Use the
> www interface or send it to gmx-developers-request at gromacs.org.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://maillist.sys.kth.se/pipermail/gromacs.org_gmx-developers/attachments/20120212/7c612065/attachment.html>