[gmx-users] Can't run v. 4.6.6 on worker nodes if they are compiled with SIMD

[Seyyed Mohtadin Hashemi]

Hi everyone,

I'm having a very weird problem with GROMACS 4.6.6:

I am currently testing out GPU capabilities and was trying to compile
GROMACS with CUDA (v6.0). I can not make this work if I compile GROMACS
with SIMD, no matter what kernel I choose - I have tried everything from
SSE2 to AVX_256.

The log-in node, where I compile, has AMD Interlagos CPUs (worker nodes use
Xeon E5-2630 and are equipped with Tesla K20), but I do not think this is
the problem - I have compiled GROMACS, using the log-in node, without CUDA
but with AVX_256 SIMD and everything works. As soon as CUDA is added to the
mix, I get "Illegal Instruction" every time I try to run on the worker
nodes.

Compiling on worker nodes gives the same result. However, as soon as I set
SIMD=None everything works and I am able to run simulation using GPUs, this
is regardless of if I use log-in node or worker node to compile.


The cmake string used to configure is:
ccmake .. -DCMAKE_INSTALL_PREFIX=/work/gromacs4gpu -DGMX_DOUBLE=OFF
-DGMX_DEFAULT_SUFFIX=OFF -DGMX_BINARY_SUFFIX=_4gpu -DGMX_LIBS_SUFFIX=_4gpu
-DGMX_GPU=ON -DBUILD_SHARED_LIBS=OFF -DGMX_PREFER_STATIC_LIBS=ON
-GMX_MPI=OFF -DGMX_CPU_ACCELERATION=AVX_256

CUDA v6.0 and FFTW v3.3.4 (single precision) libs are set globally and
correctly identified by GROMACS. To remove OpenMPI as a problem I am
compiling without it (compiling with OpenMPI produced the same behavior as
without), once I have found the error I will compile with OpenMPI v1.6.5.

I get these warnings during the configuration, nothing important:

 A BLAS library was not found by CMake in the paths available to it.
Falling back on the GROMACS internal version of the BLAS library instead.
This is fine for normal usage.

 A LAPACK library was not found by CMake in the paths available to it.
Falling back on the GROMACS internal version of the LAPACK library instead.
This is fine for normal usage.

I am currently trying to compile and test GROMACS 5.0 to see if it also
exhibits the same behavior.

I hope that someone can point me in the direction of a possible solution,
if not then I will file a bug report.

Regards,
Mohtadin