[gmx-users] Archiving all the simulation data

[David van der Spoel]

Florian Haberl wrote:
> Hi,
> 
> On Wednesday 17 May 2006 01:16, Dallas B. Warren wrote:
> 
>>Not exactly specific GROMACS question, but looking for some insight for
>>those who are sure to be encountering the same issues.
>>
>>How are people now handling archiving the huge data files that are
>>produced when going to large simulation boxes for long periods of time?
>>
>>Until recently it has been find for me to put them onto DVDs, or may be
>>two or three.  But as the sims get bigger, that is getting to be a pain
>>to do.
>>
>>What are the options available?  Is there actually anything out there
>>that can archive things when you are looking at 20-50G of data files?
>>Or is the best option to just archive onto hard drives now?
> 
> 
> I still think that tape is the most secure and cheapiest solution. Of course 
> you should have some arrays of hard disks around for fast unzipping them 
> again but after finishing a project you should migrate the data to a cheaper 
> and more secure medium and this is still tape.
> 
> Modern tape systems can achieve 120mb/s and around 1tb space on each tape so 
> you don`t need to play tape jockey. if you need some old informations you can 
> easily transfer it back to faster medium in a real short time.
> 
> Pricing scale per gb with infrastructure (estimated):
> 
> scsi or faster  ~ 10 eu (datacentre structure)
> sata                < 5-7 eu (NAS or SAN)
> real low cost     ~  2 eu (normal sata raid controller + cheap hard disks)
> tape                 < 1eu
> 
> Data will always take one step back after certain time ...

It gets more problematic when you have a couple of Tb RAID arrays. Our 
latest box with 15 x 500 Gb cost roughly 8000 Euro, that is 1.1 eu/Gb.

1 Tb divided by 120 mb/s is roughly 8000 s, more than two hours, too 
slow for my taste.

And in our group 0.5-1 Tb roughly corresponds to a publishable project.

> 
> 
> 
> 
>>Catch ya,
>>
>>Dr. Dallas Warren
>>Lecturer
>>Department of Pharmaceutical Biology and Pharmacology
>>Victorian College of Pharmacy, Monash University
>>381 Royal Parade, Parkville VIC 3010
>>dallas.warren at vcp.monash.edu.au
>>+61 3 9903 9524
>>---------------------------------
>>When the only tool you own is a hammer, every problem begins to resemble
>>a nail.
>>_______________________________________________
>>gmx-users mailing list    gmx-users at gromacs.org
>>http://www.gromacs.org/mailman/listinfo/gmx-users
>>Please don't post (un)subscribe requests to the list. Use the
>>www interface or send it to gmx-users-request at gromacs.org.
>>Can't post? Read http://www.gromacs.org/mailing_lists/users.php
> 
> 
> 
> greetings,
> 
> Florian
> 


-- 
David.
________________________________________________________________________
David van der Spoel, PhD, Assoc. Prof., Molecular Biophysics group,
Dept. of Cell and Molecular Biology, Uppsala University.
Husargatan 3, Box 596,  	75124 Uppsala, Sweden
phone:	46 18 471 4205		fax: 46 18 511 755
spoel at xray.bmc.uu.se	spoel at gromacs.org   http://folding.bmc.uu.se
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++