[gmx-developers] GitLab CI artifacts pushing 5TB

[Paul bauer]

I changed the project setting for this, waiting an hour to see if the 
storage use will go down now.

On 30/09/2022 09:27, Paul bauer wrote:
> On a second note, I just checked our CI settings and the most time we 
> keep artifacts is for 1 week, so this can't be the issue. It is more 
> likely that pipeline artifacts are being stored indefinitely, and that 
> our docker images take up a lot of space.
>
> But I'm open to suggestions here.
>
> Cheers
>
> Paul
>
> On 30/09/2022 09:22, Paul bauer wrote:
>> Yes, I concur that we should reduce the time the artifacts are 
>> available for most jobs. I'll upload a change for that now.
>>
>> Cheers
>>
>> Paul
>>
>> On 29/09/2022 18:33, Erik Lindahl wrote:
>>> I suggest we automatically expire everything in 30 days, just like 
>>> we now do for cache.
>>>
>>> I guess it's always trivial to rerun a pipeline if somebody wants 
>>> the artifacts?
>>>
>>> Cheers,
>>>
>>> Erik
>>>
>>> -- 
>>> Erik Lindahl, Professor of Biophysics
>>> Science for Life Laboratory, Stockholm University & KTH
>>> On 29 Sep 2022 at 17:30 +0100, Eric Irrgang <ericirrgang at gmail.com>, 
>>> wrote:
>>>> Hi Devs.
>>>>
>>>> According to https://gitlab.com/gromacs/gromacs/-/usage_quotas we 
>>>> are almost at 5TB of retained job artifacts, and growing.
>>>>
>>>> Do we have any idea of how big that can get before it becomes a 
>>>> problem?
>>>>
>>>> I notice that we have selected the option to override the 
>>>> `expire_in` job artifact parameter for successful pipelines and to 
>>>> instead retain artifacts for the most recent commit on each ref 
>>>> indefinitely.
>>>> https://docs.gitlab.com/ee/ci/pipelines/job_artifacts.html#keep-artifacts-from-most-recent-successful-jobs
>>>>
>>>> Is this necessary or appropriate? We have a _lot_ of branches, and 
>>>> it is unclear how/whether artifacts are trimmed for refs for 
>>>> branches that have been removed.
>>>>
>>>> If the existing value of `expire_in` is too short, maybe we could 
>>>> set it to a longer, but finite time instead of "forever".
>>>>
>>>> If we want to further reduce our artifacts volume, we could set 
>>>> `expire_in` to something less.
>>>>
>>>> We could also consider generating less artifact volume by merging 
>>>> some jobs or stages. The config jobs, for instance, take only 
>>>> marginally longer to run than the overhead they generate, so we 
>>>> would lose very little CPU/pod time by merging them with the build 
>>>> jobs.
>>>>
>>>> Best,
>>>> Eric
>>>> --
>>>> Gromacs Developers mailing list
>>>>
>>>> * Please search the archive at 
>>>> https://www.gromacs.org/gmx-devel.html before posting!
>>>>
>>>> * Can't post? Read https://www.gromacs.org/gmx-devel.html
>>>>
>>>> * For (un)subscribe requests visit
>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers 
>>>> or send a mail to gmx-developers-request at gromacs.org.
>>>
>>
>> -- 
>> Paul Bauer, PhD
>> GROMACS Development Manager
>> KTH Stockholm, SciLifeLab
>> 0046737308594
>
>
> -- 
> Paul Bauer, PhD
> GROMACS Development Manager
> KTH Stockholm, SciLifeLab
> 0046737308594


-- 
Paul Bauer, PhD
GROMACS Development Manager
KTH Stockholm, SciLifeLab
0046737308594
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://maillist.sys.kth.se/pipermail/gromacs.org_gmx-developers/attachments/20220930/ab8c706e/attachment-0001.html>