[gmx-developers] Gromacs parallel I/O?

[Shirts, Michael (mrs5pt)]

> BTW: Regarding parallel read of XTC for analysis tools. I suggest we add an
> XTC meta-file to solve the problem of parallel read for XTC. To be able to
> read frames in parallel we need to know the starting positions of the frame.
> Using the bisect search for XTC in parallel will probably give  poor
> performance on most parallel IO systems (small random access IO pattern - is
> what parallel IO systems don't like at all). Using TRR instead for parallel
> analysis is also not such a good idea because even with parallel IO several
> analysis will be IO bound and thus we could benefit from the XTC compression.
> Thus an XTC file with a meta-file containing the starting positions should
> give the best performance. A separate meta-file instead of adding the
> positions to the header has the advantage that we don't change the current
> format and thus don't break compatibility with 3rd party softare. Having a
> separate meta-file has the disadvantage of the required bookkeeping to make
> sure that the XTC file and the metafile are up-to date to each other, but I
> think this shouldn't be to difficult to solve. And if a meta-file is missing
> or not up-to date it is possible to generate it on the fly.

I'm wondering if this is the sort of problem that eventually moving to
something like netCDF might help solve.  Clearly, it would be a difficult
move, and would require interconversion utilities for backward
compatibility.

Best,

~~~~~~~~~~~~
Michael Shirts
Assistant Professor
Department of Chemical Engineering
University of Virginia
michael.shirts at virginia.edu
(434)-243-1821