[gmx-developers] Please cancel outdated CI pipelines

[Paul Bauer]

Hello,

I started to set up tests that can run as jobs after a commit is merged,
and we should just see that we get this code in to reduce the stress on the
hardware. All the slow jobs can then be moved there.

/Paul

On Mon, 28 Sep 2020, 17:00 Erik Lindahl, <erik.lindahl at gmail.com> wrote:

> Hi,
>
> Good point, but it also shows we have some homework to do. Our new CI
> infrastructure was quite expanded (~80 high end cores, 2GB/core, all SSD
> disks) - but this only seems to have led all of us to happily add tests
> that took more time.
>
> Looking just briefly at the pipelines, it seems the testing phase is our
> main culprit. While it is of course nice to have per-change tests, I don't
> think it's sustainable that we need 10+ CPU hours of testing for every typo
> fix.
>
> In particular these tests need attention:
>
> - gmx-api. They both take 12-15 minutes on two cores, and there are four
> of them.
>
> - TSAN & ASAN. I don't think we can justify using 8 cores for 15-20 min
> for each of them.
>
> - OpenCL, likely related to slow kernel compiles, which gets even worse
> when the AMD GPUs become a bottleneck.
>
>
> I also suspect that quite a few tests asking for lots of cores and memory
> don't really use all of it (at least not efficiently), but as a result
> other CI jobs will have to wait.
>
> There's also a huge difference in performance between proper unit tests
> called on code level vs. the ones that issue commands or even run
> simulations.
>
> This week is not the one to change things, but IMHO we need to get back to
> the original model of the CI tests for every change executing FAST. Any
> test job that doesn't complete in less than ~3 min on a single core does
> not belong among the ones that are run for every change.
>
> Cheers,
>
> Erik
>
>
>
>
> Erik Lindahl <erik.lindahl at scilifelab.se>
> Professor of Biophysics
> Science for Life Laboratory
> Stockholm University & KTH
> Office (SciLifeLab): +46 8 524 81567
> Cell (Sweden): +46 73 4618050
> Cell (US): +1 (650) 924 7674
>
>
>
> > On 28 Sep 2020, at 16:23, Eric Irrgang <ericirrgang at gmail.com> wrote:
> >
> > Hi Devs,
> >
> > If you push a new commit to a GitLab branch before the pipelines are
> finished running for the previous commit, please consider canceling one or
> the other sets of pipelines.
> >
> > You can look at the Pipelines tab of the merge request page (or just go
> to https://gitlab.com/gromacs/gromacs/-/pipelines). If you have pushed a
> new commit, you are presumably only interested in one (of the sets of)
> pipelines. Just click the red X to cancel the pipelines you don't need.
> >
> > If you are pushing to a branch that doesn't have an MR yet, you are
> still generating one pipeline for every push, so please use the web
> interface to cancel the pipelines that aren't useful to you.
> >
> > It will really help all of us to get our CI pipelines to run sooner.
> >
> > Thanks!
> > M. Eric Irrgang
> > --
> > Gromacs Developers mailing list
> >
> > * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List before
> posting!
> >
> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >
> > * For (un)subscribe requests visit
> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers
> or send a mail to gmx-developers-request at gromacs.org.
> --
> Gromacs Developers mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers
> or send a mail to gmx-developers-request at gromacs.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://maillist.sys.kth.se/pipermail/gromacs.org_gmx-developers/attachments/20200928/48134fae/attachment.html>