[gmx-developers] impending changes to Jenkins verification

[Mark Abraham]

Hi devs,

With 5.1 off the table, we're implementing some much-needed updates to the
way we handle Jenkins verification of GROMACS.

Teemu's rewritten the scripts we use to implement the various kinds of
verification jobs, which will let us maintain and extend in much less
ad-hoc fashion. Some parts of those scripts will now live in the GROMACS
source repository, so that they can change in step with code changes. We've
already submitted that script to master, so when you rebase patches
over 0ce920a017, Jenkins will be able to use its new toys.

We'll probably submit the half of the scripts that live in the releng repo
shortly, and when we do, we'll make a new set of voting verification jobs
named like the old ones, but with "-new-releng" suffix. These will all
fail, unless a patch under review has been rebased to pick up the above
commit. We plan to rebase a few commits to check the new gear is stable.
Please feel free to rebase your commits and upload to Jenkins, but please
check that Jenkins already doesn't have a hundred tasks in its queue, and
don't rebase all your commits at once. Hopefully after a few days, we'll
feel things are working well, and we will de-activate the old verification
jobs and rely on the new ones.

We then plan to rework the matrix of things that we get Jenkins to test
every time code is uploaded to gerrit (ie. pre-submit), and use other
matrices of build configurations and test types that run only after we
submit code, or weekly, etc. This will cover a bunch of known test coverage
gaps, and hopefully not too many unknown bugs will be found :-)

At some point soon, we will probably move the matrix descriptions into the
source repository, so we have a record of what build configurations and
tests were supposed to pass on that code version. That might require
another round of rebasing - you'll hear if that's the case.

At some point soon, we plan to re-implement the build slaves that Jenkins
uses, probably based on Docker containers running in a dynamically
provisioned way on a bunch of old hardware we salvaged. This should give us
reproducible testing behaviour. We should be able to spend less time on
ongoing Jenkins maintenance. It should be easier for us to add a bunch of
hardware during release periods for more rigorous testing. We can probably
separate the compilation phase from the test phase (lowering the pressure
on our machines with real target hardware in them). Debugging a problem
reported in Jenkins will be easier if developers can get the Docker image
that failed, and perhaps then run it on their desktop. Otherwise, hopefully
we can extract the build and test artefacts from the Docker images, and
publish them on Jenkins in the usual way.

Any bright ideas, or other feedback is welcome. We'll keep you posted of
developments :-)

Mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://maillist.sys.kth.se/pipermail/gromacs.org_gmx-developers/attachments/20150902/54747ecd/attachment.html>