[gmx-users] Archiving all the simulation data

[Mark Abraham]

Dallas B. Warren wrote:
> Not exactly specific GROMACS question, but looking for some insight for
> those who are sure to be encountering the same issues.
> 
> How are people now handling archiving the huge data files that are
> produced when going to large simulation boxes for long periods of time?
> 
> Until recently it has been find for me to put them onto DVDs, or may be
> two or three.  But as the sims get bigger, that is getting to be a pain
> to do.
> 
> What are the options available?  Is there actually anything out there
> that can archive things when you are looking at 20-50G of data files?
> Or is the best option to just archive onto hard drives now?

The other side of the coin is reducing the amount of output produced. 
Frequent full-precision snapshots of positions and velocities are not 
necessary. Unless there's analysis techniques of which I'm unaware, the 
only value the velocities have is to permit restarts. I tend to keep 
such a full-precision position+velocity frame only every few hours of 
compute time. For analysis, I keep positions in .xtc format every 0.1ps 
(which is probably overkill) which is highly compact - bzip2 tends to 
only reduce .xtc by a few percent, and plain .xtc is comparable to .trr 
followed by bzip2.

As a researcher in Australia, you also have potential free access to the 
mass data store facility at APAC. That's easiest to use if you are also 
using their machines, as I gather you are from previous posts. Tens of 
gigabytes are chicken feed there - I got 500GB last request.

Mark