[gmx-developers] Gromacs parallel I/O?

[Roland Schulz]

Hi,

we would like to implement parallel output later this year.

We would suggest to do the following. Any suggestions are very welcome.

The idea is to write all files at the time when the checkpoint is written
(e.g. every 15min). The reason is, that while the IO bandwidth is not so
much a problem even with multi-million atom systems, the latency of the IO
system can reduce the parallel efficiency significantly. Thus the idea to
store frame in memory until the checkpoint is written and thus making writes
much less frequent.
We don't think this will have any (mayor) disadvantages, since in case the
simulation crashes we don't loose anything because we would need to redo the
simulation from the last checkpoint anyhow. Also the memory usage would be
very low because it would be mainly useful for high parallelization (thus
small number of atoms per node).
This would also allow to do parallel write very easily. Without writing
several frames it is not efficient to do parallel writes because of the
sorting and the low parallel IO bandwidth resulting from small chunk size
(splitting one frame over several nodes would make each chunk small). If N
frames are stored than it is easy to sort and write those N frames on N
nodes also get the higher IO bandwidth of parallel writes possible for large
chunks.

Martin, may I ask why you are interested in parallel output?

BTW: Regarding parallel read of XTC for analysis tools. I suggest we add an
XTC meta-file to solve the problem of parallel read for XTC. To be able to
read frames in parallel we need to know the starting positions of the frame.
Using the bisect search for XTC in parallel will probably give  poor
performance on most parallel IO systems (small random access IO pattern - is
what parallel IO systems don't like at all). Using TRR instead for parallel
analysis is also not such a good idea because even with parallel IO
several analysis will be IO bound and thus we could benefit from the XTC
compression. Thus an XTC file with a meta-file containing the starting
positions should give the best performance. A separate meta-file instead of
adding the positions to the header has the advantage that we don't change
the current format and thus don't break compatibility with 3rd party
softare. Having a separate meta-file has the disadvantage of the required
bookkeeping to make sure that the XTC file and the metafile are up-to date
to each other, but I think this shouldn't be to difficult to solve. And if a
meta-file is missing or not up-to date it is possible to generate it on the
fly.

Roland

On Tue, Jul 6, 2010 at 4:58 PM, Sander Pronk <pronk at cbr.su.se> wrote:

> That's true; right now only the MPI rank 0 process writes trajectories (and
> with -multisim multiple processes get to be rank 0 for their communicator).
> Under normal circumstances, the time spent writing out trajectories should
> be negligible compared to the rest of the MD algorithm. For the version
> beyond the soon-to-be released 4.5, we're looking into parallel I/O for the
> processing tools, but nothing has been decided yet.
>
> On Jul 6, 2010, at 22:44 , Martin M. Ossowski wrote:
>
> > Hello,
> >
> > Can anyone tell me whether one or more MPI processes are involved in
> writing trajectories to the disk.  My feeling is that one process, say rank
> 0, does all the output file writing but I would like to confirm this.
> >
> > -Thank you,
> > -Martin.
> >
> > --
> > gmx-developers mailing list
> > gmx-developers at gromacs.org
> > http://lists.gromacs.org/mailman/listinfo/gmx-developers
> > Please don't post (un)subscribe requests to the list. Use thewww
> interface or send it to gmx-developers-request at gromacs.org.
>
> --
> gmx-developers mailing list
> gmx-developers at gromacs.org
> http://lists.gromacs.org/mailman/listinfo/gmx-developers
> Please don't post (un)subscribe requests to the list. Use the
> www interface or send it to gmx-developers-request at gromacs.org.
>



-- 
ORNL/UT Center for Molecular Biophysics cmb.ornl.gov
865-241-1537, ORNL PO BOX 2008 MS6309
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://maillist.sys.kth.se/pipermail/gromacs.org_gmx-developers/attachments/20100706/a7154237/attachment.html>