[gmx-developers] Re: TNG format in Gromacs

Roland Schulz roland at utk.edu
Mon Apr 16 22:08:17 CEST 2012

On Mon, Apr 16, 2012 at 2:51 PM, Erik Lindahl <lindahl at cbr.su.se> wrote:

> Switching this discussion to gmx-developers - please participate if you
> feel you have something to say!
> On Apr 16, 2012, at 8:07 PM, Roland Schulz wrote:
> >
> > Obvious external programs (e.g. VMD) won't bundle HDF5. But I doubt they
> have anything against supporting the HDF5 based formats on systems with
> HDF5 installed.
> It is critical that the new formats work *everywhere*. This might sound
> excessive, but trust me - when we originally developed our current portable
> formats they required the XDR libraries that were available everywhere
> (read: as long as it was unix), and roughly 10 years ago that meant I had
> to spend several months writing a portable XDR implementation so we could
> get Gromacs working on Windows for Folding at Home. We're not going back to
> anything that isn't 100% portable :-)

Obviously, by requiring that an external library works everywhere, we
decide that no external library can ever be used (because it is impossible
to test against all platforms/compilers) :-). Also I don't think it makes
any sense to choose portability over every other important metric
(maintainability, features, ...), no matter how small
the portability difference and how huge the other advantages. Additionally.
I think it should be up to the developers who want a certain portability to
put in the effort to get it (as it is with other features).

I think we should have a list of supported platforms/compilers. As a
developer it should be my duty to make sure that any code changes work on
any of those (ideally Jenkins should do it for me) and I should
be responsible to fix any bugs on any supported platform in my code. On the
other hand, bugs on any non supported platform should be the responsibility
of the person who wants it.

If we would require all developers to support all platforms, we would shift
the responsibility to support esoteric platforms to the developers of new
features, even if the majority has no interest in the platform. This is not
how we handle new features and I don't see any reason why we should do that
for portability.

I think we shouldn't require better portability to external code than we
require for own own.

>  > Is it really easier to fix code some other Gromacs developer wrote?
> Given that the Gromacs code is often not well documented, IMO it is
> sometimes easier to fix code in external libraries than in Gromacs. Thus I
> think this is only an advantage as long as the Gromacs developers who
> originally wrote the code are still around. Also HDF5 is supported by the
> compute centers in the US. Thus we could ask one of the compute centers
> with help on HDF5 if we would required. Also HDF5 has commercial support.
> I don't know about HDF5, but we have never had any significant problems in
> fixing portability issues in the GROMACS codebase. Again, HDF5 might be a
> model of portability, but before even thinking about that alternative the
> portability has to be tested!

I agree it has to be tested.

But I think it is important to remember that maintaining portability in
Gromacs has also problems and significant cost (=developer time). As an
example, looking at Gerrit one sees quite a large number of changes which
had to be reworked to fix MSVC issues. Also the large number of #ifdefs
caused a significant number of bugs. And writing a new container format
would add signification amount of low level (and thus non-trivial to port)

> > There is no question that HDF5 is much more versatile, but do we
> absolutely need that versatility?
> > No. But do we absolutely need NEC and Google nativeclient?  I would
> wager that some of the nice to have but not required HDF5 features are more
> useful than NEC, NC, and Portland together.
> Disagree. Nativeclient is likely to be Google's main vehicle for running
> things in the cloud, and NEC has on several instances had the fastest
> computers in the world (and there is rumor about an update to the Earth
> Simulator).

Both is used by a very small part of the Gromacs community. Thus it
shouldn't be more important than a feature used by that few people.

> And right now Portland is the only major compiler with support for GPU
> directives - it is an important compiler.
GPU directives is a potential useful but currently not used feature. It
should be treated as other comparable features.

> Certainly portability is important. But I don't think portability should
> be taken too far. We should try to decide solution based on the usefulness
> of the code to the Gromacs community (maybe with a slightly more importance
> to the needs of the core developers). Thus if hundreds of user would
> slightly benefit from non-essential tools provided by HDF5, while one user
> has to use an old/limited version (e.g. on native-client), I think the
> majority needs should be more important.
> I think the first step is to be concrete about exactly what useful
> features we will get automatically with HDF5 that will be very difficult or
> impossible to implement in our own format?
It is one option to do that research first. We have started it but are not
finished. Another option is, we first agree on the requirements any
potential external library has to fulfill. I think we currently don't have
any guidelines for this.

> Again, I like HDF5 as a potential more "general" optional format
> corresponding to our current TRR files. In this case it will be close to
> trivial to implement, and it will be very easy to slice data along any
> dimension directly into an analysis program.
> However, here we are primarily talking about the replacement to XTC, and
> with very fancy compression routines I think all those features disappear.

This format needs to be 100% portable

Well even if all of TNG would only be available with HDF5, I don't see why
we would need to have it 100% portable. The XTC compression is not orders
of magnitudes worse.

and easy to provide as a separate library for inclusion in other programs.

My experience is that other projects are less reluctant to have external
dependencies. E.g. VMD already has an optional dependency on NetCDF.

> And, let's focus on what we need rather than fun stuff that might come in
> handy once in a while :-)
I think, this is my main point. I don't see how NEC and PGI doesn't fall
into the "handy once in a while" category :-).


> > BTW: If we now discuss pro/cons of HDF5 and external libraries I suggest
> to move this discussion to gmx-developers. I moved the discussion back to
> private email because we were gathering information and I thought that
> information gathering is not of general interest. But the pro/cons of HDF5
> (and external libraries in general) probably should be discussed on the
> list. Feel free to reply in public or let me know whether it is OK for me
> to forward to the list.
> Good idea - taking it there!
> Cheers,
> Erik

ORNL/UT Center for Molecular Biophysics cmb.ornl.gov
865-241-1537, ORNL PO BOX 2008 MS6309
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://maillist.sys.kth.se/pipermail/gromacs.org_gmx-developers/attachments/20120416/df67e2a7/attachment.html>

More information about the gromacs.org_gmx-developers mailing list