[gmx-developers] src/tests/benchmark sets storage solutions

[Szilárd Páll]

Hi,

Putting the code there might work, but can we really host there:
- a 100-150 Mb tgz with the regression test suite;
- benchmark suite, probably gigabytes in size;

These, in fact already the first one, can easily cause terabytes of traffic
a day. Don't they have some limit?

Cheers,
--
Szilárd


On Mon, Jan 7, 2013 at 10:46 PM, Roland Schulz <roland at utk.edu> wrote:

> Hi,
>
> why don't we put all downloads on github (or any other OpenSource site
> (sourceforge, googlecode, ....)) and link to there from the download page?
> That would solve the problem very easily.
>
> Roland
>
>
> On Mon, Jan 7, 2013 at 4:27 PM, Szilárd Páll <szilard.pall at cbr.su.se>wrote:
>
>>  Hi,
>>
>>  In the future we plan to provide source code, regression tests,
>> benchmark data set, and possibly some binary packages as downloads for
>> GROMACS users. For reasons discussed earlier, with the 4.6 release the
>> regression test suite and validation of GROMACS build/installation is
>> becoming very important. A standardized benchmark set and pre-compiled
>> binaries would also be a great benefit for many.
>>
>>  However, there is one important aspect we need to consider. The source
>> + regression tests suite is already close to 150 Mb which is non-negligible
>> from the point of view of server load. The benchmark set will probably be
>> split up in several packages, some of them gigabytes in size; binaries for
>> the half 6-8 most important platforms will also potentially contribute to
>> the server traffic.
>>
>>  A deluge of large downloads hitting and possibly crippling our servers
>> after the next major/minor release(s) is a very realistic scenario for
>> which we need to be prepared. After discussing with with some developers
>> (thanks Berk & Sander for the tips) the is the suggestion we came up with:
>>
>>  * We should use a separate, dedicated machine/VM to host downloads (are
>> we keeping the current ftp server for this purpose?).
>> * The number of connections and downwind bandwidth for
>> the various GROMACS downloads should be capped to protect the server.
>> * We should provide and *encourage* the use of torrents for downloads,
>> especially the large ones like the benchmark test suite.
>> + We could consider having mirrors for some (larger) or all downloads.
>> However, for this we would need someone to help out with (read: "donate")
>> storage space + bandwidth!
>>  + We could consider commercial storage solutions *if* there are
>> suggestions for relatively cheap and efficient ones.
>>
>>  Legend:
>> *: needs immediate decision/action;
>> +: would be good to get feedback before 4.6.0.
>>
>>  Cheers,
>> --
>> Szilárd
>>
>
>
>
> --
> ORNL/UT Center for Molecular Biophysics cmb.ornl.gov
> 865-241-1537, ORNL PO BOX 2008 MS6309
>
> --
> gmx-developers mailing list
> gmx-developers at gromacs.org
> http://lists.gromacs.org/mailman/listinfo/gmx-developers
> Please don't post (un)subscribe requests to the list. Use the
> www interface or send it to gmx-developers-request at gromacs.org.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://maillist.sys.kth.se/pipermail/gromacs.org_gmx-developers/attachments/20130107/7cae4206/attachment.html>