[gmx-developers] jenkins killed by core files

[Berk Hess]

Hi,

I don't see much value in archiving core files. It could be useful to 
generate them on request, but even that doesn't seem very useful to me.

Cheers,

Berk

On 11/09/2017 05:59 PM, Szilárd Páll wrote:
> Hi,
>
> Earlier today (around 15:00 CET) a change resulted in ~1200 failed
> tests and generated a volume of core files that swamped the jenkins
> server. We spent far too much time tracking down the issue and
> recovering from it and identified some critical issues in the setup
> that we believe require changes.
>
> Given that:
> - we all are increasingly drafting and developing in gerrit, not even
> testing locally,
> - we don't have enough space for even the ~1200 cores files of a
> single failed job could generate (let alone multiple iterations of a
> buggy change),
> - we're storing uncompressed cores files that we rarely if ever look at
> we need to take action and prevent such time-consuming failures.
>
> There are two options I see:
> - I'll disable archiving core files (right away so the aforementioned
> change won't bomb jenkins again ;)) -- after devs time saved by having
> jenkins compile for them can now be spent on occasionally testing a
> bit more locally (or on the build slaves if necessary) when hard to
> track down bugs cause crashes;
>
> - We have found a jenkis plugin that compresses artifacts; if this
> reduces the size of archived data enough, we could try to deploy it
> and re-enable core file archival. I have the suspicion that it won't
> work due to a bug prevented us from using it to begin with, but I'll
> have to check.
>
> Cheers,
> --
> Szilárd