[gmx-developers] src/tests/benchmark sets storage solutions

[Christoph Junghans]

2013/1/7 Roland Schulz <roland at utk.edu>:
>
>
>
> On Mon, Jan 7, 2013 at 6:09 PM, Szilárd Páll <szilard.pall at cbr.su.se> wrote:
>>
>>
>>
>> I am not a big fan of SF exactly because I can't just wget it but have to
>> go to their page and click
Also SF allows you use wget. Example:
$ wget http://sourceforge.net/projects/libmtp/files/libmtp/1.1.5/libmtp-1.1.5.tar.gz/download
works just fine.
>
>
> Google code doesn't have that problem. So that might be the best option. If
> we want to try that we need to decide that quickly so we can request an
> increased size limit. I don't know how long it takes them to grant it.
Google guys are pretty good about those things, write a short email
and hours later stuff is approved.

I still prefer the gibhub tag solution as we don't need to do anything
except tagging certain version. We could just create a git repository
with binary benchmark programs and purge it whenever it gets too big
with a git force push.

Christoph
>
> Roland
>
>>
>> All in all, I still favor torrent as the encouraged means of downloading +
>> our own file server with capped uplink protection (which will make the
>> torrent in just a few days if not hours have tons of seeders and Mbytes/s
>> downloads speed.
>>
>> --
>> Szilárd
>>
>>
>>>
>>> Roland
>>>
>>>>
>>>>
>>>> Cheers,
>>>> --
>>>> Szilárd
>>>>
>>>>
>>>> On Mon, Jan 7, 2013 at 10:46 PM, Roland Schulz <roland at utk.edu> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> why don't we put all downloads on github (or any other OpenSource site
>>>>> (sourceforge, googlecode, ....)) and link to there from the download page?
>>>>> That would solve the problem very easily.
>>>>>
>>>>> Roland
>>>>>
>>>>>
>>>>> On Mon, Jan 7, 2013 at 4:27 PM, Szilárd Páll <szilard.pall at cbr.su.se>
>>>>> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> In the future we plan to provide source code, regression tests,
>>>>>> benchmark data set, and possibly some binary packages as downloads for
>>>>>> GROMACS users. For reasons discussed earlier, with the 4.6 release the
>>>>>> regression test suite and validation of GROMACS build/installation is
>>>>>> becoming very important. A standardized benchmark set and pre-compiled
>>>>>> binaries would also be a great benefit for many.
>>>>>>
>>>>>> However, there is one important aspect we need to consider. The source
>>>>>> + regression tests suite is already close to 150 Mb which is non-negligible
>>>>>> from the point of view of server load. The benchmark set will probably be
>>>>>> split up in several packages, some of them gigabytes in size; binaries for
>>>>>> the half 6-8 most important platforms will also potentially contribute to
>>>>>> the server traffic.
>>>>>>
>>>>>> A deluge of large downloads hitting and possibly crippling our servers
>>>>>> after the next major/minor release(s) is a very realistic scenario for which
>>>>>> we need to be prepared. After discussing with with some developers (thanks
>>>>>> Berk & Sander for the tips) the is the suggestion we came up with:
>>>>>>
>>>>>> * We should use a separate, dedicated machine/VM to host downloads
>>>>>> (are we keeping the current ftp server for this purpose?).
>>>>>> * The number of connections and downwind bandwidth for the various
>>>>>> GROMACS downloads should be capped to protect the server.
>>>>>> * We should provide and *encourage* the use of torrents for downloads,
>>>>>> especially the large ones like the benchmark test suite.
>>>>>> + We could consider having mirrors for some (larger) or all downloads.
>>>>>> However, for this we would need someone to help out with (read: "donate")
>>>>>> storage space + bandwidth!
>>>>>> + We could consider commercial storage solutions *if* there are
>>>>>> suggestions for relatively cheap and efficient ones.
>>>>>>
>>>>>> Legend:
>>>>>> *: needs immediate decision/action;
>>>>>> +: would be good to get feedback before 4.6.0.
>>>>>>
>>>>>> Cheers,
>>>>>> --
>>>>>> Szilárd
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> ORNL/UT Center for Molecular Biophysics cmb.ornl.gov
>>>>> 865-241-1537, ORNL PO BOX 2008 MS6309
>>>>>
>>>>> --
>>>>> gmx-developers mailing list
>>>>> gmx-developers at gromacs.org
>>>>> http://lists.gromacs.org/mailman/listinfo/gmx-developers
>>>>> Please don't post (un)subscribe requests to the list. Use the
>>>>> www interface or send it to gmx-developers-request at gromacs.org.
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> ORNL/UT Center for Molecular Biophysics cmb.ornl.gov
>>> 865-241-1537, ORNL PO BOX 2008 MS6309
>>>
>>> --
>>> gmx-developers mailing list
>>> gmx-developers at gromacs.org
>>> http://lists.gromacs.org/mailman/listinfo/gmx-developers
>>> Please don't post (un)subscribe requests to the list. Use the
>>> www interface or send it to gmx-developers-request at gromacs.org.
>>
>>
>
>
>
> --
> ORNL/UT Center for Molecular Biophysics cmb.ornl.gov
> 865-241-1537, ORNL PO BOX 2008 MS6309
>
> --
> gmx-developers mailing list
> gmx-developers at gromacs.org
> http://lists.gromacs.org/mailman/listinfo/gmx-developers
> Please don't post (un)subscribe requests to the list. Use the
> www interface or send it to gmx-developers-request at gromacs.org.



--
Christoph Junghans
Web: http://www.compphys.de