[gmx-developers] Verlet kernel testing in 5.1

[Szilárd Páll]

On Wed, Oct 1, 2014 at 12:57 AM, Roland Schulz <roland at utk.edu> wrote:
> Hi,
>
> 1) Both regressiontests and unit-tests are important. Unit tests won't be
> able to find bugs which are in the interplay of different modules. Unit
> tests make it *much* easier to narrow down an issue and run many tests
> quickly. Thus even if we have a way to run the verlet tests as unit-tests
> (which would be awesome) we still needed a regressiontest framework.

I fully agree, I have however not expressed myself correctly by
listing these two as opposing alternatives - thanks for clarifying.
However, to get *something* done, wouldn't it be best to focus on one
thing at a time, at least for the next (minor) release.

Exactly because of the difference between unit and
regression/integration tests (and even the latter  two are different
in many ways), I support the idea of having higher level test as a
first priority. That's because I think it is better to know that
something is wrong with one of the modules related to the Verlet
algorithms (e.g. either with the kernels, pair search, or say the
hardware detection/kernel selection) than to only test the kernels.

> 2) There are multiple serialization frameworks for C++. The issue has a lot
> of overlap with how to communicate complex data structures over MPI. And
> also a bit with how to offload those data structures to an accelerator. Thus
> it might make sense that we use the same framework for unit tests and MPI.
> See http://redmine.gromacs.org/issues/996.

I have not even considered MPI here, but indeed, it is probably a good
idea to keep it in mind when considering serialization in the
integration suite within mdrun.

> 3) I don't think we should move tests from gmxtest.pl to the
> mdrun-integrationtests until the later is at least as functional (i.e.
> compares output values). And we should decide which way we go sooner rather
> than later.

I tend to agree, but unfortunately I don't think we have picked a
strategy, but instead neglected gmxtest to some extent for quite some
time. More or less as a direct consequence of this AFAIR at least two
nasty -tunepme bugs have already slipped - that part of mdrun is
fragile and we are not testing it at all. I'm not comfortable with
this situation exactly because I experienced myself how the false
sense of security of a the lacking test setups can result in bad
scenarios (ref: the headlines of 5.0.2).

Cheers,
--
Sz.

> I don't think the perl approach is inherently bad. If we rewrite
> it partly to make it more extensible it might be a nice long-term solution.
>
> Roland
>
>
> On Tue, Sep 30, 2014 at 3:50 PM, Szilárd Páll <pall.szilard at gmail.com>
> wrote:
>>
>> Hi,
>>
>> As we aim at removing the group kernels for the next release and
>> making the Verlet kernels the only feature-complete non-bonded
>> implementation, we will need some form of extensive testing of all
>> Verlet kernels. We've discussed things offline today and I'd like to
>> summarize some of the conclusions and hopefully get some input on
>> which way should we be heading.
>>
>> * Build unit tests around the kernels.
>> This is probably the most sound way of testing the kernels, but it
>> likely requires a lot of work; in particular, with the current code it
>> seems to be quite difficult to set up all data structures required for
>> search and kernels without calling the entire mdrun and based on
>> Mark's experience serializing the data structures will be a pain too
>> (although for calling just the kernels only interaction_const_t,
>> nbnxn_atomdata_t, nbnxn_pairlist_set_t is needed).
>>
>>
>> * Use the mdrun integration test harness
>> Can't say much about it as I have admittedly not used it myself.
>> However, what's sure is that compared to proper unit tests it requires
>> giving up  lightweight-ness. The hardness executes the full mdrun
>> without the ability to test just a specific kernel but, just like with
>> the gmxtest-based kernel tests, a specially crafted tpr and quite
>> likely supporting code changes will be required to exercise only some
>> code-paths and kernel versions/flavors.
>>
>>
>> * Update gmxtest with Verlet kernel tests
>> At the moment a most of the (group) kernel testing functionality is in
>> gmxtest. While gmxtest is getting more and more outdated, for the next
>> release the regression test suite will surely require a considerable
>> amount of attention to at least remove group scheme tests. Having done
>> that, not much will be left, which bring up a couple of weakly related
>> questions:
>> - Will (all) current tests will be migrated to Verlet setup?
>> - Will they be kept in gmxtest?
>> - ...or most/all will moved all these into some other form of testing?
>> If there will, what will that be?
>> If gmxtest is kept and tests will get simply migrated from group to
>> Verlet setup, it may as well be the least amount of effort to adopt
>> the kernel testing procedure used for the group kernels. Of course,
>> this will just prolong the life of gmxtest.pl, but it may still be the
>> most feasible approach and I believe there has been no consensus on
>> whether all testing will be moved into the mdrun integration test
>> harness.
>>
>> Cheers,
>> --
>> Szilárd
>> --
>> Gromacs Developers mailing list
>>
>> * Please search the archive at
>> http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List before
>> posting!
>>
>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>
>> * For (un)subscribe requests visit
>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers or
>> send a mail to gmx-developers-request at gromacs.org.
>
>
>
>
> --
> ORNL/UT Center for Molecular Biophysics cmb.ornl.gov
> 865-241-1537, ORNL PO BOX 2008 MS6309
>
> --
> Gromacs Developers mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers or
> send a mail to gmx-developers-request at gromacs.org.