[gmx-developers] Re: TNG format in Gromacs

[Erik Lindahl]

On Apr 16, 2012, at 10:08 PM, Roland Schulz wrote:
> 
> Obviously, by requiring that an external library works everywhere, we decide that no external library can ever be used (because it is impossible to test against all platforms/compilers) :-). Also I don't think it makes any sense to choose portability over every other important metric (maintainability, features, ...), no matter how small the portability difference and how huge the other advantages. Additionally. I think it should be up to the developers who want a certain portability to put in the effort to get it (as it is with other features).

Unilke other features (such as a better RNG or faster parallel IO), portability has to be a feature of every single line of code in Gromacs. It only takes 30 seconds for anybody to destroy it, while it can be months of time to fix it if it is necessary to reimplement code. You can't "put in" portability separately from the rest of the code.


> I think we should have a list of supported platforms/compilers. As a developer it should be my duty to make sure that any code changes work on any of those (ideally Jenkins should do it for me) and I should be responsible to fix any bugs on any supported platform in my code. On the other hand, bugs on any non supported platform should be the responsibility of the person who wants it.

We already have this list: An ISO C (soon C++) compiler. If your code does not compile on a standards-compliant compiler, the code is buggy.

> If we would require all developers to support all platforms, we would shift the responsibility to support esoteric platforms to the developers of new features, even if the majority has no interest in the platform. This is not how we handle new features and I don't see any reason why we should do that for portability.

Again, the difference between a feature and portability is that you can live without a new feature if it is possible to turn off, while non-portable code means Gromacs won't run at all. Only requiring an ISO compiler is a core value we have had for almost 20 years, and that decision won't be changed easily. '

Historically, the portability to new platforms has been a tremendous advantage for Gromacs.

> I think we shouldn't require better portability to external code than we require for own own.

Indeed - but the whole point is that Gromacs code that isn't portable when used with an ISO C/C++ compiler should result in a redmine issue and be fixed. When this has been urgent (e.g. for a new supercomputer or compiler), we have virtually always managed to fix it within 24h. 

We don't require this of external code or libraries for optional features, of course, but if a library is going to be used for a core Gromacs feature (like FFTs, or the XDR functionality) we have made sure there is a built-in version where we can take responsibility for the portability ourselves.
> 
> 
> > I think the first step is to be concrete about exactly what useful features we will get automatically with HDF5 that will be very difficult or impossible to implement in our > own format?
> It is one option to do that research first. We have started it but are not finished. Another option is, we first agree on the requirements any potential external library has to fulfill. I think we currently don't have any guidelines for this.

The route we have taken this far is that you can use pretty much any external library, as long as there is Gromacs built-in functionality (potentially much slower) to make sure Gromacs always works. For instance, we have FFTPACK built-in although we prefer to use FFTW externally.

> Well even if all of TNG would only be available with HDF5, I don't see why we would need to have it 100% portable. The XTC compression is not orders of magnitudes worse.

1) Because we want to avoid producing XTC for new trajectories. We want a small library that other programs (VMD, PyMol, Babel, etc.) can include to be able to read/write these trajectory formats. Case in point: I believe there are at least  two orders of magnitude more users that want Gromacs file formats they can directly feed into other programs compared to the number of users for which it is important with more efficient parallel IO.

2) Because it would introduce internal incompatibilities in Gromacs. If the default file format isn't available on all platforms we would be back in the situation before we had XTC available everywhere, and Gromacs compiled on host B would not always be able to read the files produced on host A with the very same Gromacs version.

3) Obviously, no code will ever achieve 100% portability in advance without fixes. However, there is a world of difference between a smaller file codebase where we actually *CAN* fix such issues, versus not being able to compile core Gromacs on a platform until an external library has decided to support that platform too.

There is another good example here: NAMD currently can't be compiled or run on the Kei computer, because CHARMM++ hasn't been ported to it yet.

> and easy to provide as a separate library for inclusion in other programs.
> My experience is that other projects are less reluctant to have external dependencies. E.g. VMD already has an optional dependency on NetCDF.

Yes, but read your sentence again: The key word there is "optional"! Second, VMD is a viewer that is unlikely to need to run esoteric embedded hardware.


Cheers,

Erik