[gmx-developers] Please cancel outdated CI pipelines

[Erik Lindahl]

Hi,

Good point, but it also shows we have some homework to do. Our new CI infrastructure was quite expanded (~80 high end cores, 2GB/core, all SSD disks) - but this only seems to have led all of us to happily add tests that took more time.

Looking just briefly at the pipelines, it seems the testing phase is our main culprit. While it is of course nice to have per-change tests, I don't think it's sustainable that we need 10+ CPU hours of testing for every typo fix.

In particular these tests need attention:

- gmx-api. They both take 12-15 minutes on two cores, and there are four of them. 

- TSAN & ASAN. I don't think we can justify using 8 cores for 15-20 min for each of them.

- OpenCL, likely related to slow kernel compiles, which gets even worse when the AMD GPUs become a bottleneck.


I also suspect that quite a few tests asking for lots of cores and memory don't really use all of it (at least not efficiently), but as a result other CI jobs will have to wait.

There's also a huge difference in performance between proper unit tests called on code level vs. the ones that issue commands or even run simulations.

This week is not the one to change things, but IMHO we need to get back to the original model of the CI tests for every change executing FAST. Any test job that doesn't complete in less than ~3 min on a single core does not belong among the ones that are run for every change.

Cheers,

Erik




Erik Lindahl <erik.lindahl at scilifelab.se>
Professor of Biophysics
Science for Life Laboratory
Stockholm University & KTH
Office (SciLifeLab): +46 8 524 81567
Cell (Sweden): +46 73 4618050
Cell (US): +1 (650) 924 7674 



> On 28 Sep 2020, at 16:23, Eric Irrgang <ericirrgang at gmail.com> wrote:
> 
> Hi Devs,
> 
> If you push a new commit to a GitLab branch before the pipelines are finished running for the previous commit, please consider canceling one or the other sets of pipelines.
> 
> You can look at the Pipelines tab of the merge request page (or just go to https://gitlab.com/gromacs/gromacs/-/pipelines). If you have pushed a new commit, you are presumably only interested in one (of the sets of) pipelines. Just click the red X to cancel the pipelines you don't need.
> 
> If you are pushing to a branch that doesn't have an MR yet, you are still generating one pipeline for every push, so please use the web interface to cancel the pipelines that aren't useful to you.
> 
> It will really help all of us to get our CI pipelines to run sooner.
> 
> Thanks!
> M. Eric Irrgang
> -- 
> Gromacs Developers mailing list
> 
> * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List before posting!
> 
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> 
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers or send a mail to gmx-developers-request at gromacs.org.