[gmx-developers] Verlet kernel testing in 5.1

Sat Oct 4 08:47:07 CEST 2014

Hi,

My two cents, for what it is worth:

The discussion here focuses mostly on technical  implementation details of
how to organize the testing, but I think an even more important discussion
would be to investigate what we should be testing. A major reason why the
existing regression tests do not test things like -tunepme is that they try
test for exact reproducibility of trajectories in an inherently chaotic
algorithm, and that simply cannot support testing longer runs that would be
required for testing the actual mdrun functionality.

This is where unit tests could help: if there were separate unit tests that
would test the individual forces and energies, the regression tests could
concentrate on testable global properties of longer runs that are still
sensitive to correct  implementation, such as energy conservation, instead
of trying to test every fragile property of the trajectory.

Like Roland said, I think there is nothing inherently bad with an external
script to run the tests (although, if we rewrite the script from scratch,
I'd propose Python). The main benefit of a C++ testing framework comes from
the ability to access internal data structures, which in turn allows more
lightweight tests and assertions on the internal state. But in order to
have such access, substantial refactoring of the code under test may be
required, so this is not a magic bullet. For integration-style tests for
whole executables, an external driver written in a scripting language is
probably a much easier solution.

Finally, if the blocker is the availability of resources for writing such
scripts (and not deciding what they should test), I can volunteer to help
with the script infrastructure, if people think that the most valuable
thing I could be doing. I can't promise to deliver such scripts in O(days)
time, but if the alternative is that it never happens, waiting a few weeks
or months should not be a big deal.

Best regards,
Teemu
On Oct 3, 2014 8:20 PM, "Szilárd Páll" <pall.szilard at gmail.com> wrote:

> On Wed, Oct 1, 2014 at 12:57 AM, Roland Schulz <roland at utk.edu> wrote:
> > Hi,
> >
> > 1) Both regressiontests and unit-tests are important. Unit tests won't be
> > able to find bugs which are in the interplay of different modules. Unit
> > tests make it *much* easier to narrow down an issue and run many tests
> > quickly. Thus even if we have a way to run the verlet tests as unit-tests
> > (which would be awesome) we still needed a regressiontest framework.
>
> I fully agree, I have however not expressed myself correctly by
> listing these two as opposing alternatives - thanks for clarifying.
> However, to get *something* done, wouldn't it be best to focus on one
> thing at a time, at least for the next (minor) release.
>
> Exactly because of the difference between unit and
> regression/integration tests (and even the latter  two are different
> in many ways), I support the idea of having higher level test as a
> first priority. That's because I think it is better to know that
> something is wrong with one of the modules related to the Verlet
> algorithms (e.g. either with the kernels, pair search, or say the
> hardware detection/kernel selection) than to only test the kernels.
>
> > 2) There are multiple serialization frameworks for C++. The issue has a
> lot
> > of overlap with how to communicate complex data structures over MPI. And
> > also a bit with how to offload those data structures to an accelerator.
> Thus
> > it might make sense that we use the same framework for unit tests and
> MPI.
> > See http://redmine.gromacs.org/issues/996.
>
> I have not even considered MPI here, but indeed, it is probably a good
> idea to keep it in mind when considering serialization in the
> integration suite within mdrun.
>
> > 3) I don't think we should move tests from gmxtest.pl to the
> > mdrun-integrationtests until the later is at least as functional (i.e.
> > compares output values). And we should decide which way we go sooner
> rather
> > than later.
>
> I tend to agree, but unfortunately I don't think we have picked a
> strategy, but instead neglected gmxtest to some extent for quite some
> time. More or less as a direct consequence of this AFAIR at least two
> nasty -tunepme bugs have already slipped - that part of mdrun is
> fragile and we are not testing it at all. I'm not comfortable with
> this situation exactly because I experienced myself how the false
> sense of security of a the lacking test setups can result in bad
> scenarios (ref: the headlines of 5.0.2).
>
> Cheers,
> --
> Sz.
>
> > I don't think the perl approach is inherently bad. If we rewrite
> > it partly to make it more extensible it might be a nice long-term
> solution.
> >
> > Roland
> >
> >
> > On Tue, Sep 30, 2014 at 3:50 PM, Szilárd Páll <pall.szilard at gmail.com>
> > wrote:
> >>
> >> Hi,
> >>
> >> As we aim at removing the group kernels for the next release and
> >> making the Verlet kernels the only feature-complete non-bonded
> >> implementation, we will need some form of extensive testing of all
> >> Verlet kernels. We've discussed things offline today and I'd like to
> >> summarize some of the conclusions and hopefully get some input on
> >> which way should we be heading.
> >>
> >> * Build unit tests around the kernels.
> >> This is probably the most sound way of testing the kernels, but it
> >> likely requires a lot of work; in particular, with the current code it
> >> seems to be quite difficult to set up all data structures required for
> >> search and kernels without calling the entire mdrun and based on
> >> Mark's experience serializing the data structures will be a pain too
> >> (although for calling just the kernels only interaction_const_t,
> >> nbnxn_atomdata_t, nbnxn_pairlist_set_t is needed).
> >>
> >>
> >> * Use the mdrun integration test harness
> >> Can't say much about it as I have admittedly not used it myself.
> >> However, what's sure is that compared to proper unit tests it requires
> >> giving up  lightweight-ness. The hardness executes the full mdrun
> >> without the ability to test just a specific kernel but, just like with
> >> the gmxtest-based kernel tests, a specially crafted tpr and quite
> >> likely supporting code changes will be required to exercise only some
> >> code-paths and kernel versions/flavors.
> >>
> >>
> >> * Update gmxtest with Verlet kernel tests
> >> At the moment a most of the (group) kernel testing functionality is in
> >> gmxtest. While gmxtest is getting more and more outdated, for the next
> >> release the regression test suite will surely require a considerable
> >> amount of attention to at least remove group scheme tests. Having done
> >> that, not much will be left, which bring up a couple of weakly related
> >> questions:
> >> - Will (all) current tests will be migrated to Verlet setup?
> >> - Will they be kept in gmxtest?
> >> - ...or most/all will moved all these into some other form of testing?
> >> If there will, what will that be?
> >> If gmxtest is kept and tests will get simply migrated from group to
> >> Verlet setup, it may as well be the least amount of effort to adopt
> >> the kernel testing procedure used for the group kernels. Of course,
> >> this will just prolong the life of gmxtest.pl, but it may still be the
> >> most feasible approach and I believe there has been no consensus on
> >> whether all testing will be moved into the mdrun integration test
> >> harness.
> >>
> >> Cheers,
> >> --
> >> Szilárd
> >> --
> >> Gromacs Developers mailing list
> >>
> >> * Please search the archive at
> >> http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List before
> >> posting!
> >>
> >> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >>
> >> * For (un)subscribe requests visit
> >> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers
> or
> >> send a mail to gmx-developers-request at gromacs.org.
> >
> >
> >
> >
> > --
> > ORNL/UT Center for Molecular Biophysics cmb.ornl.gov
> > 865-241-1537, ORNL PO BOX 2008 MS6309
> >
> > --
> > Gromacs Developers mailing list
> >
> > * Please search the archive at
> > http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List before
> > posting!
> >
> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >
> > * For (un)subscribe requests visit
> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers
> or
> > send a mail to gmx-developers-request at gromacs.org.
> --
> Gromacs Developers mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers
> or send a mail to gmx-developers-request at gromacs.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://maillist.sys.kth.se/pipermail/gromacs.org_gmx-developers/attachments/20141004/662673ba/attachment-0001.html>