[gmx-developers] Errors in tng_io reading box vectors

[Peter Kenneth Eastman]

I found another error in tng_io.  When compiling on Windows with Visual Studio, trying to load tng_example.tng fails.  The data gets completely corrupted, and many functions return errors.

The problem is in the routines for converting between byte orders (tng_swap_byte_order_little_endian_64 etc.).  Those routines take a signed argument (for example int64_t) and then apply bit shift operators to it, which results in undefined behavior.  See https://www.securecoding.cert.org/confluence/display/c/INT13-C.+Use+bitwise+operators+only+on+unsigned+operands.  It appears that clang and gcc always use zero padding, while Visual Studio uses sign bit extension.  So if the argument happens to be negative, they produce different results.

The fix is to change the argument to uint64_t.  Then you always get zero padding, as intended.

Peter