[gmx-developers] Collective IO

[Roland Schulz]

On Thu, Sep 30, 2010 at 6:21 PM, Szilárd Páll <szilard.pall at cbr.su.se>wrote:

> Hi Roland,
>
> Nice work, I'll definitely take a look at it!
>
> Any idea on how does this improve scaling in general and at what
> problem size starts to really matter? Does it introduce and overhead
> in smaller simulations or it is only conditionally turned on?
>

At the moment it is always turned on for XTC when compiled with MPI. In
serial or with threads nothing changes. At the moment we buffer at maximum
100 frames. If one uses less than 100 PP nodes than we buffer as many frames
as the number of PP nodes. We also make sure that we don't buffer more than
250MB per node.

The 100 frames and 250MB are both constants which should probably still be
tuned.

If the IO time is negligible than of course you want see any speedup. The IO
time shows mostly up as comm. energies (because only 1 node is in write_traj
and the others are waiting at the next comm. energies). Usually the IO time
because important if you either have more than 100 nodes and/or you write
very frequently.

It should not be slower in any case. The only case might be when one writes
from many IO nodes to a non parallel file-system (e.g. NFS) that the
parallel IO could be slower. We will test this tomorrow. Otherwise it should
always be as fast or faster.

Ryan & Roland


>
>
> On Fri, Oct 1, 2010 at 12:11 AM, Roland Schulz <roland at utk.edu> wrote:
> > Hi,
> > we (Ryan&me) just uploaded our work on buffered MPI writing of XTC
> > trajectories. It can be found in the branch CollectiveIO.
> > We buffer a number of frames and use MPI IO to write those frames from a
> > number of nodes (see previous mails for details). The XTC trajectory is
> > written at least at every checkpoint guaranteeing that no frames are lost
> if
> > a simulation crashes.
> > We have tested it in serial, with PME, with threads, with multi and it
> seems
> > to work in all cases.
> > For 3 million atoms on 8192 writing every 1000 steps the performance is
> > increased from 21ns/day to 34ns/day and the time spent in comm.
> > energies decreases from 47% to 7%.
> > Feedback on the code change is very welcome. If you want to look at the
> > diff, I suggest to use:
> > git difftool afd66e48c4e608    #this is the origin/master from when we
> > uploaded the branch
> > Please let us know what you would like us to change before we merge this
> > into master.
> > Thanks
> > Ryan & Roland
> > --
> > ORNL/UT Center for Molecular Biophysics cmb.ornl.gov
> > 865-241-1537, ORNL PO BOX 2008 MS6309
> >
> > --
> > gmx-developers mailing list
> > gmx-developers at gromacs.org
> > http://lists.gromacs.org/mailman/listinfo/gmx-developers
> > Please don't post (un)subscribe requests to the list. Use the
> > www interface or send it to gmx-developers-request at gromacs.org.
> >
> --
> gmx-developers mailing list
> gmx-developers at gromacs.org
> http://lists.gromacs.org/mailman/listinfo/gmx-developers
> Please don't post (un)subscribe requests to the list. Use the
> www interface or send it to gmx-developers-request at gromacs.org.
>



-- 
ORNL/UT Center for Molecular Biophysics cmb.ornl.gov
865-241-1537, ORNL PO BOX 2008 MS6309
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://maillist.sys.kth.se/pipermail/gromacs.org_gmx-developers/attachments/20100930/38551462/attachment.html>