[gmx-developers] GitLab CI artifacts pushing 5TB

[Paul bauer]

On a second note, I just checked our CI settings and the most time we 
keep artifacts is for 1 week, so this can't be the issue. It is more 
likely that pipeline artifacts are being stored indefinitely, and that 
our docker images take up a lot of space.

But I'm open to suggestions here.

Cheers

Paul

On 30/09/2022 09:22, Paul bauer wrote:
> Yes, I concur that we should reduce the time the artifacts are 
> available for most jobs. I'll upload a change for that now.
>
> Cheers
>
> Paul
>
> On 29/09/2022 18:33, Erik Lindahl wrote:
>> I suggest we automatically expire everything in 30 days, just like we 
>> now do for cache.
>>
>> I guess it's always trivial to rerun a pipeline if somebody wants the 
>> artifacts?
>>
>> Cheers,
>>
>> Erik
>>
>> -- 
>> Erik Lindahl, Professor of Biophysics
>> Science for Life Laboratory, Stockholm University & KTH
>> On 29 Sep 2022 at 17:30 +0100, Eric Irrgang <ericirrgang at gmail.com>, 
>> wrote:
>>> Hi Devs.
>>>
>>> According to https://gitlab.com/gromacs/gromacs/-/usage_quotas we 
>>> are almost at 5TB of retained job artifacts, and growing.
>>>
>>> Do we have any idea of how big that can get before it becomes a problem?
>>>
>>> I notice that we have selected the option to override the 
>>> `expire_in` job artifact parameter for successful pipelines and to 
>>> instead retain artifacts for the most recent commit on each ref 
>>> indefinitely.
>>> https://docs.gitlab.com/ee/ci/pipelines/job_artifacts.html#keep-artifacts-from-most-recent-successful-jobs
>>>
>>> Is this necessary or appropriate? We have a _lot_ of branches, and 
>>> it is unclear how/whether artifacts are trimmed for refs for 
>>> branches that have been removed.
>>>
>>> If the existing value of `expire_in` is too short, maybe we could 
>>> set it to a longer, but finite time instead of "forever".
>>>
>>> If we want to further reduce our artifacts volume, we could set 
>>> `expire_in` to something less.
>>>
>>> We could also consider generating less artifact volume by merging 
>>> some jobs or stages. The config jobs, for instance, take only 
>>> marginally longer to run than the overhead they generate, so we 
>>> would lose very little CPU/pod time by merging them with the build jobs.
>>>
>>> Best,
>>> Eric
>>> --
>>> Gromacs Developers mailing list
>>>
>>> * Please search the archive at 
>>> https://www.gromacs.org/gmx-devel.html before posting!
>>>
>>> * Can't post? Read https://www.gromacs.org/gmx-devel.html
>>>
>>> * For (un)subscribe requests visit
>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers 
>>> or send a mail to gmx-developers-request at gromacs.org.
>>
>
> -- 
> Paul Bauer, PhD
> GROMACS Development Manager
> KTH Stockholm, SciLifeLab
> 0046737308594


-- 
Paul Bauer, PhD
GROMACS Development Manager
KTH Stockholm, SciLifeLab
0046737308594
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://maillist.sys.kth.se/pipermail/gromacs.org_gmx-developers/attachments/20220930/a16a1285/attachment.html>