[gmx-users] Archiving all the simulation data
David van der Spoel
spoel at xray.bmc.uu.se
Wed May 17 20:42:11 CEST 2006
Florian Haberl wrote:
> On Wednesday 17 May 2006 01:16, Dallas B. Warren wrote:
>>Not exactly specific GROMACS question, but looking for some insight for
>>those who are sure to be encountering the same issues.
>>How are people now handling archiving the huge data files that are
>>produced when going to large simulation boxes for long periods of time?
>>Until recently it has been find for me to put them onto DVDs, or may be
>>two or three. But as the sims get bigger, that is getting to be a pain
>>What are the options available? Is there actually anything out there
>>that can archive things when you are looking at 20-50G of data files?
>>Or is the best option to just archive onto hard drives now?
> I still think that tape is the most secure and cheapiest solution. Of course
> you should have some arrays of hard disks around for fast unzipping them
> again but after finishing a project you should migrate the data to a cheaper
> and more secure medium and this is still tape.
> Modern tape systems can achieve 120mb/s and around 1tb space on each tape so
> you don`t need to play tape jockey. if you need some old informations you can
> easily transfer it back to faster medium in a real short time.
> Pricing scale per gb with infrastructure (estimated):
> scsi or faster ~ 10 eu (datacentre structure)
> sata < 5-7 eu (NAS or SAN)
> real low cost ~ 2 eu (normal sata raid controller + cheap hard disks)
> tape < 1eu
> Data will always take one step back after certain time ...
It gets more problematic when you have a couple of Tb RAID arrays. Our
latest box with 15 x 500 Gb cost roughly 8000 Euro, that is 1.1 eu/Gb.
1 Tb divided by 120 mb/s is roughly 8000 s, more than two hours, too
slow for my taste.
And in our group 0.5-1 Tb roughly corresponds to a publishable project.
>>Dr. Dallas Warren
>>Department of Pharmaceutical Biology and Pharmacology
>>Victorian College of Pharmacy, Monash University
>>381 Royal Parade, Parkville VIC 3010
>>dallas.warren at vcp.monash.edu.au
>>+61 3 9903 9524
>>When the only tool you own is a hammer, every problem begins to resemble
>>gmx-users mailing list gmx-users at gromacs.org
>>Please don't post (un)subscribe requests to the list. Use the
>>www interface or send it to gmx-users-request at gromacs.org.
>>Can't post? Read http://www.gromacs.org/mailing_lists/users.php
David van der Spoel, PhD, Assoc. Prof., Molecular Biophysics group,
Dept. of Cell and Molecular Biology, Uppsala University.
Husargatan 3, Box 596, 75124 Uppsala, Sweden
phone: 46 18 471 4205 fax: 46 18 511 755
spoel at xray.bmc.uu.se spoel at gromacs.org http://folding.bmc.uu.se
More information about the gromacs.org_gmx-users