[gmx-developers] Re: TNG format in Gromacs

[XAvier Periole]

It could be useful to define/list OS that would be included into the  
100% portability. You may not have the same "list" in mind!

XAvier.

On Apr 17, 2012, at 3:18 PM, Roland Schulz wrote:

>
>
> On Tue, Apr 17, 2012 at 2:44 AM, Erik Lindahl <lindahl at cbr.su.se>  
> wrote:
>
> On Apr 16, 2012, at 10:08 PM, Roland Schulz wrote:
> >
> > Obviously, by requiring that an external library works everywhere,  
> we decide that no external library can ever be used (because it is  
> impossible to test against all platforms/compilers) :-). Also I  
> don't think it makes any sense to choose portability over every  
> other important metric (maintainability, features, ...), no matter  
> how small the portability difference and how huge the other  
> advantages. Additionally. I think it should be up to the developers  
> who want a certain portability to put in the effort to get it (as it  
> is with other features).
>
> Unilke other features (such as a better RNG or faster parallel IO),  
> portability has to be a feature of every single line of code in  
> Gromacs. It only takes 30 seconds for anybody to destroy it, while  
> it can be months of time to fix it if it is necessary to reimplement  
> code. You can't "put in" portability separately from the rest of the  
> code.
> E.g. for parallelization the issue is very similar as it is for  
> portability. Supporting domain decomposition makes it more difficult  
> for everyone and everyone has to make sure that they don't brake it.  
> And it is only included because it essential to Gromacs and used by  
> almost everyone.
>
> > I think we should have a list of supported platforms/compilers. As  
> a developer it should be my duty to make sure that any code changes  
> work on any of those (ideally Jenkins should do it for me) and I  
> should be responsible to fix any bugs on any supported platform in  
> my code. On the other hand, bugs on any non supported platform  
> should be the responsibility of the person who wants it.
>
> We already have this list: An ISO C (soon C++) compiler. If your  
> code does not compile on a standards-compliant compiler, the code is  
> buggy.
> This is only an idea but not a practical rule. It is impossible to  
> write any OS depending code (e.g. file operations) with that  
> requirement. E.g. futil.c only works if the OS is either POSIX or  
> Windows. And also this rule is unclear about how it is implemented/ 
> tested. Is it enough to show that a strict inheritance to the  
> standard is followed (e.g. gcc -pendantic) or does it require to  
> test all possible compilers (impossible).
>
>
> > If we would require all developers to support all platforms, we  
> would shift the responsibility to support esoteric platforms to the  
> developers of new features, even if the majority has no interest in  
> the platform. This is not how we handle new features and I don't see  
> any reason why we should do that for portability.
>
> Again, the difference between a feature and portability is that you  
> can live without a new feature if it is possible to turn off, while  
> non-portable code means Gromacs won't run at all. Only requiring an  
> ISO compiler is a core value we have had for almost 20 years, and  
> that decision won't be changed easily. '
>
> Historically, the portability to new platforms has been a tremendous  
> advantage for Gromacs.
> As far as I know, at least for the last few years portability to  
> exotic platforms (e.g. PS3, K) has been important to only a very  
> small part of users/developers.
>
> > I think we shouldn't require better portability to external code  
> than we require for own own.
>
> Indeed - but the whole point is that Gromacs code that isn't  
> portable when used with an ISO C/C++ compiler should result in a  
> redmine issue and be fixed. When this has been urgent (e.g. for a  
> new supercomputer or compiler), we have virtually always managed to  
> fix it within 24h.
>
> We don't require this of external code or libraries for optional  
> features, of course, but if a library is going to be used for a core  
> Gromacs feature (like FFTs, or the XDR functionality) we have made  
> sure there is a built-in version where we can take responsibility  
> for the portability ourselves.
>
> But I think that "fancy" IO is also an optional feature. I agree  
> that it is a very important feature and it has many disadvantages if  
> the same format is not used everywhere. But it is also non- 
> essential. And at that point it should become a matter of cost- 
> benefit and not a matter of principal. I.e. how many people benefit  
> from features made possible by HDF5 (e.g. because limited developer  
> time wouldn't allow them without HDF5) versus how much of a pain is  
> it to the few people how have to live with XTC (and conversion). And  
> one very important factor in that cost-benefit analysis is the ratio  
> of users.
>
> >
> >
> > > I think the first step is to be concrete about exactly what  
> useful features we will get automatically with HDF5 that will be  
> very difficult or impossible to implement in our > own format?
> > It is one option to do that research first. We have started it but  
> are not finished. Another option is, we first agree on the  
> requirements any potential external library has to fulfill. I think  
> we currently don't have any guidelines for this.
>
> The route we have taken this far is that you can use pretty much any  
> external library, as long as there is Gromacs built-in functionality  
> (potentially much slower) to make sure Gromacs always works. For  
> instance, we have FFTPACK built-in although we prefer to use FFTW  
> externally.
>
> > Well even if all of TNG would only be available with HDF5, I don't  
> see why we would need to have it 100% portable. The XTC compression  
> is not orders of magnitudes worse.
>
> 1) Because we want to avoid producing XTC for new trajectories. We  
> want a small library that other programs (VMD, PyMol, Babel, etc.)  
> can include to be able to read/write these trajectory formats. Case  
> in point: I believe there are at least  two orders of magnitude more  
> users that want Gromacs file formats they can directly feed into  
> other programs compared to the number of users for which it is  
> important with more efficient parallel IO.
> Parallel IO is not something which would benefit from HDF5. The  
> advantages I see (as I said a preliminary list because we are in the  
> middle of looking into it):
> - User can use HDF5 tools to look at files
> - User can write easy scripts (simple binding to scripting  
> languages) to analyze data
> - No large extra code base for container format in Gromacs  
> (maintainability)
> - Less developer time (we can focus on MD specific code and not how  
> to implement a container format)
> - Easy to implement convenience features (like adding structure and  
> topology to trajectory; combining trajectory, energies, and extra  
> output (e.g. pull) and thus having (optionally) only one output  
> file; having build in history; making files user extendable)
> - Trajectory easy to index and seek (e.g. for parallel analysis)
>
> Obviously one could either keep the own container format very  
> simple, and thus would miss most convince features, tools, and  
> language bindings, or one could make it very complex, and thus  
> require a lot of developer time and have the maintainability issues.  
> But without an external library one cannot have both.
>
> 2) Because it would introduce internal incompatibilities in Gromacs.  
> If the default file format isn't available on all platforms we would  
> be back in the situation before we had XTC available everywhere, and  
> Gromacs compiled on host B would not always be able to read the  
> files produced on host A with the very same Gromacs version.
> But what is the problem if all non exotic systems would support the  
> new file format. Then one can still guarantee that the analysis can  
> be done in the new format. And the only thing, the (very few users)  
> of exotic platforms have to do, is to convert the XTC format to the  
> new format (post production) to avoid any problems.
>
> Roland
>
>
>
>
>
>
>
>
>
> -- 
> ORNL/UT Center for Molecular Biophysics cmb.ornl.gov
> 865-241-1537, ORNL PO BOX 2008 MS6309
> -- 
> gmx-developers mailing list
> gmx-developers at gromacs.org
> http://lists.gromacs.org/mailman/listinfo/gmx-developers
> Please don't post (un)subscribe requests to the list. Use the
> www interface or send it to gmx-developers-request at gromacs.org.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://maillist.sys.kth.se/pipermail/gromacs.org_gmx-developers/attachments/20120417/1d747b89/attachment.html>