[gmx-developers] Regression tests updated on git

[Mark Abraham]

Hi,

I've done a lot of updates and improvements to the regression tests. 
They're git-ified and available via

git clone git.gromacs.org:regressiontests.git

There are a number of outstanding issues & resolutions:

*) the lack of a definitive GROMACS 3.x version makes life difficult - a 
bugfixed 3.3.4 that we knew produced correct reference data in all 
regression test cases would make the ongoing utility of this regression 
test set much higher

*) As it stands, I am tempted to add a mechanism whereby the reference 
version is recorded by grepping the output of mdrun, and grabbing values 
of relevant environment variables. If/when tests ever need future 
revisions, it'd be best to have a record that was generated 
automatically, rather than relying on the maintainer to have simulated 
and documented correctly.

*) A better solution would be to create a GROMACS utility program that
will return the arguments to configure, and such. Being able to call

gmxconf -configure

or

gmxconf -version

to get this data would have quite a few applications, including
troubleshooting all kinds of issues. libxml2 has such a mechanism, for
example.

*) simple/rb1 used to have what I believe was a bug in the .mdp file
(see
http://lists.gromacs.org/pipermail/gmx-developers/2009-August/003573.html)
but I would like someone who knew how this test was supposed to work to
inspect it, please.

*) complex/field 4.0.5 does not reproduce 3.3.2 - most of the virial
terms at step 0 differ. Is this expected?

*) Likewise, under at least some circumstances many of the complex tests 
and even simple/morse can fail the virial check (and sometimes others) 
at various steps after the first. This looks inevitable - it happens 
even with identical GROMACS versions on different hardware running with 
NOASSEMBLYLOOPS=1. Do we just tell the user that this behaviour is 
expected and not to worry? If so, why bother doing the check?

*) pdb2gmx fails two tests with 4.0.5 on either the original potential
energies generated with the 3.3.2_prerelease version, and on my
recomputed 3.3.2 energies.

Against the original energies:
mabraham.anu.edu.au 386% ./gmxtest.pl pdb2gmx -suffix _405
Will test using executable suffix _405
Here follows a list of the lines in reference_s.log and ener.log which
did not
pass the comparison test within tolerance 0.05
Index  Reference   This test       Error  Description
   11    -33.9883    -29.4849     0.07095  1aml.pdb with G53a6 using
vsite=none and water=spc
   12    -33.9883    -29.4849     0.07095  1aml.pdb with G53a6 using
vsite=none and water=spce
There were 2/45 differences in final energy with the reference file
pdb2gmx tests FAILED

And against my 3.3.2 energies:
mabraham.anu.edu.au 18% ./gmxtest.pl pdb2gmx -suffix _405
Will test using executable suffix _405
Here follows a list of the lines in reference_s.log and ener.log which 
did not
pass the comparison test within tolerance 0.05
Index  Reference   This test       Error  Description
   11    -34.5967    -24.5127    0.170599  1aml.pdb with G53a6 using 
vsite=none and water=spc
   12    -34.5967    -24.5127    0.170599  1aml.pdb with G53a6 using 
vsite=none and water=spce
There were 2/45 differences in final energy with the reference file
pdb2gmx tests FAILED

This error is much larger than those for the other cases. Nothing in the
pdb2gmx/editconf/grompp/mdrun output would seem to account for it. Is
there a known reason for it?

*) I've added grompp4.mdp files generated by grompp 4.0.5 on the 
original grompp.mdp file such that so long as one of grompp.mdp
and this file match mdout.mdp suitably, then the user is not prompted to
check the .mdp files.

*) complex/acetonitrileRF is broken by 3.3.2, since a buggy function in
src/mdlib/rf_util.c calls gmx_sumi without checking for MPI. 3.3 doesn't
call gmx_sumi, and 3.3.3 calls it after checking. I guessed to generate
the reference state with 3.3.3 for this test.

*) The complex tests tend to start with zero velocities, gen_vel = no, 
and sometimes tcoupl = Berendsen, however there is no uniformity. Is 
that a feature or a bug?

Mark