[gmx-users] configuration/installation of gromacs-4.6.1 on heterogeneous cluster

[Martin Siegert]

Hi,

On Sat, May 04, 2013 at 09:50:37AM +0200, Mark Abraham wrote:
> On Sat, May 4, 2013 at 1:41 AM, Martin Siegert <siegert at sfu.ca> wrote:
> 
> > Hi,
> >
> > I am struggling with the configuration and compilation/installation
> > of gromacs-4.6.1. Our cluster has 2 different processors: the older
> > generation supports sse4.1, the newer sse4.2. Configuration and
> > compilation must be done on the headnode of the cluster, which
> > supports sse4.2. I am using the following command to configure
> > gromacs-4.6.1:
> >
> > CFLAGS='-fpic -O3 -axSSE4.2,SSE4.1 -xSSSE3 -ip -opt-prefetch' \
> > CXXFLAGS='-fpic -O3 -axSSE4.2,SSE4.1 -xSSSE3 -ip -opt-prefetch' \
> > FFLAGS='-fpic -O3 -axSSE4.2,SSE4.1 -xSSSE3 -ip -opt-prefetch' \
> >
> 
> If the above is successful at generating instructions for SSE>4.1...

-xSSSE3 ensures that the resulting code will run on processors that
support ssse3 or later and is basically equivalent to -mssse3.

-axSSE4.2,SSE4.1 tells the compiler to generate multiple code paths
for sse4.1 and sse4.2 while still supporting the default architecture
ssse3 i.e., when run on a processor that supports sse4.2 then the
code will use the special instructions for that architecture).

We compile all our codes with -axSSE4.2,SSE4.1 -xSSSE3 and they run on
ssse3 and better.

> CC=mpicc \
> > CXX=mpicxx \
> > FC=mpif90 \
> > LDFLAGS="-lfftw3f -lgoto2 -Wl,-rpath,/usr/local/gromacs-4.6.1/lib" \
> > cmake -DCMAKE_VERBOSE_MAKEFILE=ON -DGMX_MPI=ON \
> >       -DGMX_CPU_ACCELERATION=SSE4.1 -DGMX_OPENMP=OFF -DGMX_GPU=OFF \
> >       -DCMAKE_INSTALL_PREFIX=/usr/local/gromacs-4.6.1 \
> >       -DCMAKE_SKIP_RPATH=YES \
> >       ../gromacs-4.6.1
> >
> > However, after compilation/installation mdrun_mpi fails on nodes that
> > only support sse4.1 with "Illegal instruction".
> >
> 
> ... then this is not a surprise. Your compiler has been allowed to generate
> SSE 4.2 instructions, I suspect.

Not really ... -xssse3 specifies the default architecture that the code
supports.

> > The CMakeCache.txt file contains the line:
> >
> > BUILD_CPU_FEATURES:INTERNAL=aes apic clfsh cmov cx8 cx16 htt lahf_lm mmx
> > msr nonstop_tsc pcid pclmuldq pdcm pdpe1gb popcnt pse rdtscp sse2 sse3
> > sse4.1 sse4.2 ssse3
> >
> > Since this line contains "sse4.2" it appears that the flag
> > -DGMX_CPU_ACCELERATION=SSE4.1
> > is ignored.
> >
> 
> That list is what features are available on the CPU (mostly for helping
> detect what acceleration to use, and to solve problems). The target
> acceleration scheme (i.e. code with heavy use of compiler instrinsics) is
> SSE4.1, which is what GROMACS will compile for if you leave it alone :-)
> SSE4.2 is roughly useless for mdrun.
> 
> What is the correct way of specifying the cpu architecture within the
> > cmake build system? (I never had problems with this with the pre 4.6
> > versions).
> >
> 
> Back then, GROMACS was nearly insensitive to compilation settings, because
> of the use of assembly kernels. Now GROMACS is sensitive to compiler
> version (in that compilation of SIMD instrinsics needs to work well, and
> OpenMP needs to be supported, etc.) but one still doesn't generally want to
> mess with the compiler flags. We have some internal disgreement about
> whether we should be permitting/encouraging/facilitiating setting/checking
> compiler flags. So far nobody has demonstrated a use case that suggests we
> need to support more than "shut up and let GROMACS do its thing!"

For curiosity I tried this, i.e., I did not set CFLAGS, CXXFLAGS and FFLAGS
and use the gromacs supplied values which turn out to be

-O3 -ip -fPIC -msse4.1

The result is absolutely the same: on the nodes that only support sse4.1 the
code fails with "Illegal instruction".

Any suggestions what else I should try?

I am mostly concerned about the following setting in CMakeCache.txt:

//Build CPU brand
BUILD_CPU_BRAND:INTERNAL=Intel(R) Xeon(R) CPU           X5650  @ 2.67GHz
//Build CPU family
BUILD_CPU_FAMILY:INTERNAL=6
//Build CPU features
BUILD_CPU_FEATURES:INTERNAL=aes apic clfsh cmov cx8 cx16 htt lahf_lm mmx msr nonstop_tsc pcid pclmuldq pdcm pdpe1gb popcnt pse rdtscp sse2 sse3 sse4.1 sse4.2 ssse3
//Build CPU model
BUILD_CPU_MODEL:INTERNAL=44
//Build CPU stepping
BUILD_CPU_STEPPING:INTERNAL=2

As the names of the variables say, these are for the BUILD cpu, not necessarily for
the cpu the code needs to run on. Thus, if any of these are used to generate
processor specific code, then this would explain the illegal instruction.
Consequently I would need to change these settings, but to what?

Cheers,
Martin