[gmx-developers] GitLab CI artifacts pushing 5TB

Paul bauer paul.bauer.q at gmail.com
Fri Sep 30 09:27:44 CEST 2022

On a second note, I just checked our CI settings and the most time we 
keep artifacts is for 1 week, so this can't be the issue. It is more 
likely that pipeline artifacts are being stored indefinitely, and that 
our docker images take up a lot of space.

But I'm open to suggestions here.



On 30/09/2022 09:22, Paul bauer wrote:
> Yes, I concur that we should reduce the time the artifacts are 
> available for most jobs. I'll upload a change for that now.
> Cheers
> Paul
> On 29/09/2022 18:33, Erik Lindahl wrote:
>> I suggest we automatically expire everything in 30 days, just like we 
>> now do for cache.
>> I guess it's always trivial to rerun a pipeline if somebody wants the 
>> artifacts?
>> Cheers,
>> Erik
>> -- 
>> Erik Lindahl, Professor of Biophysics
>> Science for Life Laboratory, Stockholm University & KTH
>> On 29 Sep 2022 at 17:30 +0100, Eric Irrgang <ericirrgang at gmail.com>, 
>> wrote:
>>> Hi Devs.
>>> According to https://gitlab.com/gromacs/gromacs/-/usage_quotas we 
>>> are almost at 5TB of retained job artifacts, and growing.
>>> Do we have any idea of how big that can get before it becomes a problem?
>>> I notice that we have selected the option to override the 
>>> `expire_in` job artifact parameter for successful pipelines and to 
>>> instead retain artifacts for the most recent commit on each ref 
>>> indefinitely.
>>> https://docs.gitlab.com/ee/ci/pipelines/job_artifacts.html#keep-artifacts-from-most-recent-successful-jobs
>>> Is this necessary or appropriate? We have a _lot_ of branches, and 
>>> it is unclear how/whether artifacts are trimmed for refs for 
>>> branches that have been removed.
>>> If the existing value of `expire_in` is too short, maybe we could 
>>> set it to a longer, but finite time instead of "forever".
>>> If we want to further reduce our artifacts volume, we could set 
>>> `expire_in` to something less.
>>> We could also consider generating less artifact volume by merging 
>>> some jobs or stages. The config jobs, for instance, take only 
>>> marginally longer to run than the overhead they generate, so we 
>>> would lose very little CPU/pod time by merging them with the build jobs.
>>> Best,
>>> Eric
>>> --
>>> Gromacs Developers mailing list
>>> * Please search the archive at 
>>> https://www.gromacs.org/gmx-devel.html before posting!
>>> * Can't post? Read https://www.gromacs.org/gmx-devel.html
>>> * For (un)subscribe requests visit
>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers 
>>> or send a mail to gmx-developers-request at gromacs.org.
> -- 
> Paul Bauer, PhD
> GROMACS Development Manager
> KTH Stockholm, SciLifeLab
> 0046737308594

Paul Bauer, PhD
GROMACS Development Manager
KTH Stockholm, SciLifeLab
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://maillist.sys.kth.se/pipermail/gromacs.org_gmx-developers/attachments/20220930/a16a1285/attachment.html>

More information about the gromacs.org_gmx-developers mailing list