[gmx-users] building for Knights Landing

[Dave Love]

I've tried to build version 2016 for Knights Landing
(i.e. GMX_SIMD=AVX_512_KNL) without success and could use advice; it
looks painful to debug and I don't know initially whether it's really a
gromacs or gcc problem.

This is on CentOS 7 with the devtoolset-4 compiler/binutils, i.e. GCC
5.2 (and 5.3 on RHEL 6 is the same).  Depending on variations in the
build, I get different errors with SIMD intrinsics.  Here's one example:

    /opt/rh/devtoolset-4/root/usr/bin/c++   -DGMX_DOUBLE=1 -DHAVE_CONFIG_H -mavx512er -mfma    -std=c++0x   -DNDEBUG -funroll-all-loops -fexcess-precision=fast   -I/builddir/build/BUILD/gromacs-2016/openmpi_d/src -I/builddir/build/BUILD/gromacs-2016/src/external/thread_mpi/include -I/builddir/build/BUILD/gromacs-2016/src -isystem /usr/include/openmpi-x86_64    -fopenmp -o CMakeFiles/libgromacs.dir/mdlib/nbnxn_kernels/simd_4xn/nbnxn_kernel_ElecQSTab_VdwLJ_VF_4xn.cpp.o -c /builddir/build/BUILD/gromacs-2016/src/gromacs/mdlib/nbnxn_kernels/simd_4xn/nbnxn_kernel_ElecQSTab_VdwLJ_VF_4xn.cpp
    In file included from /builddir/build/BUILD/gromacs-2016/src/gromacs/simd/impl_x86_avx_512_knl/impl_x86_avx_512_knl_util_double.h:45:0,
                     from /builddir/build/BUILD/gromacs-2016/src/gromacs/simd/impl_x86_avx_512_knl/impl_x86_avx_512_knl.h:45,
                     from /builddir/build/BUILD/gromacs-2016/src/gromacs/simd/simd.h:113,
                     from /builddir/build/BUILD/gromacs-2016/src/gromacs/mdlib/nbnxn_simd.h:40,
                     from /builddir/build/BUILD/gromacs-2016/src/gromacs/mdlib/nbnxn_kernels/simd_4xn/nbnxn_kernel_ElecQSTab_VdwLJ_VF_4xn.cpp:48:
    /builddir/build/BUILD/gromacs-2016/src/gromacs/simd/impl_x86_avx_512/impl_x86_avx_512_util_double.h: In function 'double gmx::reduceIncr4ReturnSum(double*, gmx::SimdDouble, gmx::SimdDouble, gmx::SimdDouble, gmx::SimdDouble)':
    /builddir/build/BUILD/gromacs-2016/src/gromacs/simd/impl_x86_avx_512/impl_x86_avx_512_util_double.h:384:23: error: '__builtin_ia32_permdf256_mask' needs isa option -m32 -mavx512vl
         t3 = _mm256_add_pd(t3, _mm256_permutex_pd(t3, 0x4E));
                           ^
    /builddir/build/BUILD/gromacs-2016/src/gromacs/simd/impl_x86_avx_512/impl_x86_avx_512_util_double.h:385:23: error: '__builtin_ia32_permdf256_mask' needs isa option -m32 -mavx512vl
         t3 = _mm256_add_pd(t3, _mm256_permutex_pd(t3, 0xB1));
  
And another with the sort of options I tried initially:

    /opt/rh/devtoolset-4/root/usr/bin/c++   -DGMX_DOUBLE=1 -DHAVE_CONFIG_H -mavx512er -mfma   -O3 -g -march=knl  -std=c++0x   -DNDEBUG -funroll-all-loops -fexcess-precision=fast   -I/builddir/build/BUILD/gromacs-2016/openmpi_d/src -I/builddir/build/BUILD/gromacs-2016/src/external/thread_mpi/include -I/builddir/build/BUILD/gromacs-2016/src -isystem /usr/include/openmpi-x86_64    -fopenmp -o CMakeFiles/libgromacs.dir/mdlib/nbnxn_kernels/simd_4xn/nbnxn_kernel_ElecRF_VdwLJEwCombGeom_VF_4xn.cpp.o -c /builddir/build/BUILD/gromacs-2016/src/gromacs/mdlib/nbnxn_kernels/simd_4xn/nbnxn_kernel_ElecRF_VdwLJEwCombGeom_VF_4xn.cpp
    In file included from /opt/rh/devtoolset-4/root/usr/lib/gcc/x86_64-redhat-linux/5.2.1/include/immintrin.h:53:0,
                     from /builddir/build/BUILD/gromacs-2016/src/gromacs/simd/impl_x86_avx_512/impl_x86_avx_512_general.h:39,
                     from /builddir/build/BUILD/gromacs-2016/src/gromacs/simd/impl_x86_avx_512_knl/impl_x86_avx_512_knl_general.h:39,
                     from /builddir/build/BUILD/gromacs-2016/src/gromacs/simd/impl_x86_avx_512_knl/impl_x86_avx_512_knl.h:40,
                     from /builddir/build/BUILD/gromacs-2016/src/gromacs/simd/simd.h:113,
                     from /builddir/build/BUILD/gromacs-2016/src/gromacs/mdlib/nbnxn_simd.h:40,
                     from /builddir/build/BUILD/gromacs-2016/src/gromacs/mdlib/nbnxn_kernels/simd_4xn/nbnxn_kernel_ElecRF_VdwLJPSw_F_4xn.cpp:48:
    /opt/rh/devtoolset-4/root/usr/lib/gcc/x86_64-redhat-linux/5.2.1/include/avx512vlintrin.h: In function 'double gmx::reduceIncr4ReturnSum(double*, gmx::SimdDouble, gmx::SimdDouble, gmx::SimdDouble, gmx::SimdDouble)':
    /opt/rh/devtoolset-4/root/usr/lib/gcc/x86_64-redhat-linux/5.2.1/include/avx512vlintrin.h:11777:1: error: inlining failed in call to always_inline '__m256d _mm256_permutex_pd(__m256d, int)': target specific option mismatch
     _mm256_permutex_pd (__m256d __X, const int __M)
     ^

Suggestions gratefully received.