[gmx-developers] GitLab CI artifacts pushing 5TB

[Paul bauer]

Yes, I concur that we should reduce the time the artifacts are available 
for most jobs. I'll upload a change for that now.

Cheers

Paul

On 29/09/2022 18:33, Erik Lindahl wrote:
> I suggest we automatically expire everything in 30 days, just like we 
> now do for cache.
>
> I guess it's always trivial to rerun a pipeline if somebody wants the 
> artifacts?
>
> Cheers,
>
> Erik
>
> -- 
> Erik Lindahl, Professor of Biophysics
> Science for Life Laboratory, Stockholm University & KTH
> On 29 Sep 2022 at 17:30 +0100, Eric Irrgang <ericirrgang at gmail.com>, 
> wrote:
>> Hi Devs.
>>
>> According to https://gitlab.com/gromacs/gromacs/-/usage_quotas we are 
>> almost at 5TB of retained job artifacts, and growing.
>>
>> Do we have any idea of how big that can get before it becomes a problem?
>>
>> I notice that we have selected the option to override the `expire_in` 
>> job artifact parameter for successful pipelines and to instead retain 
>> artifacts for the most recent commit on each ref indefinitely.
>> https://docs.gitlab.com/ee/ci/pipelines/job_artifacts.html#keep-artifacts-from-most-recent-successful-jobs
>>
>> Is this necessary or appropriate? We have a _lot_ of branches, and it 
>> is unclear how/whether artifacts are trimmed for refs for branches 
>> that have been removed.
>>
>> If the existing value of `expire_in` is too short, maybe we could set 
>> it to a longer, but finite time instead of "forever".
>>
>> If we want to further reduce our artifacts volume, we could set 
>> `expire_in` to something less.
>>
>> We could also consider generating less artifact volume by merging 
>> some jobs or stages. The config jobs, for instance, take only 
>> marginally longer to run than the overhead they generate, so we would 
>> lose very little CPU/pod time by merging them with the build jobs.
>>
>> Best,
>> Eric
>> --
>> Gromacs Developers mailing list
>>
>> * Please search the archive at https://www.gromacs.org/gmx-devel.html 
>> before posting!
>>
>> * Can't post? Read https://www.gromacs.org/gmx-devel.html
>>
>> * For (un)subscribe requests visit
>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers 
>> or send a mail to gmx-developers-request at gromacs.org.
>

-- 
Paul Bauer, PhD
GROMACS Development Manager
KTH Stockholm, SciLifeLab
0046737308594
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://maillist.sys.kth.se/pipermail/gromacs.org_gmx-developers/attachments/20220930/d6852b00/attachment-0001.html>