[gmx-developers] src/tests/benchmark sets storage solutions

[Szilárd Páll]

Hi,

In the future we plan to provide source code, regression tests, benchmark
data set, and possibly some binary packages as downloads for GROMACS users.
For reasons discussed earlier, with the 4.6 release the regression test
suite and validation of GROMACS build/installation is becoming very
important. A standardized benchmark set and pre-compiled binaries would
also be a great benefit for many.

However, there is one important aspect we need to consider. The source +
regression tests suite is already close to 150 Mb which is non-negligible
from the point of view of server load. The benchmark set will probably be
split up in several packages, some of them gigabytes in size; binaries for
the half 6-8 most important platforms will also potentially contribute to
the server traffic.

A deluge of large downloads hitting and possibly crippling our servers
after the next major/minor release(s) is a very realistic scenario for
which we need to be prepared. After discussing with with some developers
(thanks Berk & Sander for the tips) the is the suggestion we came up with:

* We should use a separate, dedicated machine/VM to host downloads (are
we keeping the current ftp server for this purpose?).
* The number of connections and downwind bandwidth for
the various GROMACS downloads should be capped to protect the server.
* We should provide and *encourage* the use of torrents for downloads,
especially the large ones like the benchmark test suite.
+ We could consider having mirrors for some (larger) or all downloads.
However, for this we would need someone to help out with (read: "donate")
storage space + bandwidth!
+ We could consider commercial storage solutions *if* there are suggestions
for relatively cheap and efficient ones.

Legend:
*: needs immediate decision/action;
+: would be good to get feedback before 4.6.0.

Cheers,
--
Szilárd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://maillist.sys.kth.se/pipermail/gromacs.org_gmx-developers/attachments/20130107/813c45be/attachment.html>