[gmx-developers] Re: TNG format in Gromacs

[Erik Lindahl]

Switching this discussion to gmx-developers - please participate if you feel you have something to say!


On Apr 16, 2012, at 8:07 PM, Roland Schulz wrote:
> 
> Obvious external programs (e.g. VMD) won't bundle HDF5. But I doubt they have anything against supporting the HDF5 based formats on systems with HDF5 installed. 

It is critical that the new formats work *everywhere*. This might sound excessive, but trust me - when we originally developed our current portable formats they required the XDR libraries that were available everywhere (read: as long as it was unix), and roughly 10 years ago that meant I had to spend several months writing a portable XDR implementation so we could get Gromacs working on Windows for Folding at Home. We're not going back to anything that isn't 100% portable :-)

> Is it really easier to fix code some other Gromacs developer wrote? Given that the Gromacs code is often not well documented, IMO it is sometimes easier to fix code in external libraries than in Gromacs. Thus I think this is only an advantage as long as the Gromacs developers who originally wrote the code are still around. Also HDF5 is supported by the compute centers in the US. Thus we could ask one of the compute centers with help on HDF5 if we would required. Also HDF5 has commercial support.

I don't know about HDF5, but we have never had any significant problems in fixing portability issues in the GROMACS codebase. Again, HDF5 might be a model of portability, but before even thinking about that alternative the portability has to be tested!
> 
> There is no question that HDF5 is much more versatile, but do we absolutely need that versatility?
> No. But do we absolutely need NEC and Google nativeclient?  I would wager that some of the nice to have but not required HDF5 features are more useful than NEC, NC, and Portland together.

Disagree. Nativeclient is likely to be Google's main vehicle for running things in the cloud, and NEC has on several instances had the fastest computers in the world (and there is rumor about an update to the Earth Simulator). And right now Portland is the only major compiler with support for GPU directives - it is an important compiler.


> Certainly portability is important. But I don't think portability should be taken too far. We should try to decide solution based on the usefulness of the code to the Gromacs community (maybe with a slightly more importance to the needs of the core developers). Thus if hundreds of user would slightly benefit from non-essential tools provided by HDF5, while one user has to use an old/limited version (e.g. on native-client), I think the majority needs should be more important. 

I think the first step is to be concrete about exactly what useful features we will get automatically with HDF5 that will be very difficult or impossible to implement in our own format?

Again, I like HDF5 as a potential more "general" optional format corresponding to our current TRR files. In this case it will be close to trivial to implement, and it will be very easy to slice data along any dimension directly into an analysis program.

However, here we are primarily talking about the replacement to XTC, and with very fancy compression routines I think all those features disappear. This format needs to be 100% portable and easy to provide as a separate library for inclusion in other programs. And, let's focus on what we need rather than fun stuff that might come in handy once in a while :-)



> BTW: If we now discuss pro/cons of HDF5 and external libraries I suggest to move this discussion to gmx-developers. I moved the discussion back to private email because we were gathering information and I thought that information gathering is not of general interest. But the pro/cons of HDF5 (and external libraries in general) probably should be discussed on the list. Feel free to reply in public or let me know whether it is OK for me to forward to the list.

Good idea - taking it there!

Cheers,

Erik