[gmx-developers] Re: TNG format in Gromacs

[Szilárd Páll]

On Tue, Apr 17, 2012 at 6:55 PM, Erik Lindahl <erik at kth.se> wrote:
> Hi,
>
> On Apr 17, 2012, at 6:14 PM, Szilárd Páll wrote:
>
>> a) The need for a new format which will *immediately* replace XTC as
>> the default in Gromacs and seems to have requirements that pretty much
>> exclude any external library (that is not as widespread as libc :).
>>
>> b) It's not in line with the EU deliverable which requires a new
>> library with certain specs to be written. Could it be that this is the
>> classic case where the specification was created before the actual
>> requirement engineering?
>
> No, I don't think so, but the goals are different: I very much see the point for a file format that enables you to do fancy stuff, but if you want a file format that is supported by as many applications as possible you have to settle for the greatest common divisor.

Assuming that the file format in question will become a standard, who
will provide the long-term support and development? If GROMACS comes
up with a new container format, it will be on the GROMACS community to
maintaining the code and provide some level of support for other
projects that adopt it. With a brand new format written from scratch
this might turn out to be a headache.

The burden of maintaining a standard is not negligible and offloading
this has serious benefits.

>>
>> c) The apparent need for ultimate portability to extremely rare and
>> exotic architectures without accepting XTC as a fallback on these
>> platforms (with conversion for post-processing and analysis). I might
>> be wrong, but to me it seems that these extremely rare architectures
>> are often more showcase platforms rather than the iron on which 99% of
>> the science is carried out.
>
> Well, windows was exactly such a very exotic architecture until we started using Gromacs in Folding at Home. In hindsight, the decision to rely on XDR libraries for all our IO was a bad one.
>
> Just counting core-hours, I actually think windows is one of our most common Gromacs architectures today. Similarly, I think there is a clear possibility nativeclient could become a dominant platform quite soon if Google decides to push it for cloud services. It is very difficult to make predictions, in particular about the future.

If Google nativeclient becomes dominant overnight do you think that
Google and/or other parties familiar with a well known standard would
not pitch in and port the code in no time? Same goes for F at H, I'd say.
If a project is as large and important as F at H, it surely does have
plenty of resources and can surely afford to contribute some code.

> *Of course* does not mean a new default file format must be 100% portable without modifying the code.

Sure, that is anyway impossible...

>> Wouldn't it be beneficial to struck a balance
>> between portability and effort/time required by accepting a short-term
>> "compromise" (which isn't really a compromise if we don't consider
>> ultimate portability a strict requirement :). XTC will have to be
>> anyway maintained anyway so it could as well be kept as the
>> alternative for platforms where the new format is not supported in
>> early versions. So a file format that works on all reference platforms
>> (that we can and should simply list and track) with XTC  as a fallback
>> for the exotic iron should be an acceptable compromise, I'd say.
>
> I think that's a perfectly valid short- and long-term balance *as long as there is a way to gradually expand support to other platforms as we need it*.
>
> The problem isn't the unimportant platforms, but when a new platform gets important for us, while it might not yet be important for the external library. What do we do when 10MB of source code fails to work on one of our reference platforms?

That can't happen with adequate software engineering practices. If a
platform is important, the library should work on it at the first
release and continuous regression testing can ensure that problems
don't pile up over time. When it comes to new platforms, these won't
pop up overnight, but even if they do, the same way as the plain C
kernels provide a viable option -- well, at the admittedly large cost
of running at a fraction of peak perf. --, at the early porting phase
the legacy XTC format can perfectly bridge the gap and enable GROMACS
to run on these platforms in "24h".

--
Szilárd

>
> Cheers,
>
> Erik
>
>