[gmx-users] Gromacs 4.6.7 with MPI and OpenMP

[Malcolm Tobias]

All,

I'm attempting to build gromacs on a new cluster and following the same recipies that I've used in the past, but encountering a strange behavior: It claims to be using both MPI and OpenMP, but I can see by 'top' and the reported core/walltime that it's really only generating the MPI processes and no threads.

We're running a hetergenous environment, so I tend to build with MPI/OpenMP/CUDA and the Intel compilers, but I'm seeing this same sort of behavior with the GNU compilers.  Here's how I'm configuring things:

[root at login01 build2]# cmake -DGMX_FFT_LIBRARY=mkl -DGMX_MPI=ON -DGMX_GPU=ON -DCUDA_TOOLKIT_ROOT_DIR=/opt/cuda -DGMX_OPENMP=ON -DCMAKE_INSTALL_PREFIX=/act/gromacs-4.6.7_take2 .. | tee cmake.out

or with GNU:

[root at login01 build2]#  cmake -DGMX_BUILD_OWN_FFTW=ON -DGMX_GPU=ON -DCUDA_TOOLKIT_ROOT_DIR=/opt/cuda -DGMX_MPI=ON -DGMX_OPENMP=ON -DCMAKE_INSTALL_PREFIX=/act/gromacs-4.6.7_take2 .. | tee cmake.out

I'm explicitly setting GMX_GPU=ON, but I don't recall having to do so in the past.  I see in the output where it's testing for OpenMP:

[root at login01 build]# grep OpenMP  cmake.out 
-- Try OpenMP C flag = [-openmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Success
-- Try OpenMP CXX flag = [-openmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Success
-- Found OpenMP: -openmp  

When I go to run, I use 'mpirun -np 2' to trigger it to use 2 GPUs, then use the '-ntomp 4' flag to trigger 4 OpenMP threads/MPI proceess.   The funny part is that GROMACS reports that it's doing all this:

Using 2 MPI processes
Using 4 OpenMP threads per MPI process

although I do see this warning:

Number of CPUs detected (16) does not match the number reported by OpenMP (1).

I'm not sure how to proceed with debugging this, so any suggestions would be helpful.

Thanks in advance,
Malcolm

-- 
Malcolm Tobias
314.362.1594