[gmx-users] NVML library in CUDA 9 & Gromacs 2018

[Jernej Zidar]

Hi,

I have an interesting problem with the  NVML library from CUDA 9 (and 9.1)
and Gromacs 2018. Namely, the NVML is reported as detected during cmake but
Gromacs 2018 is linked without it.

This is the cmake command I would issue:

m -rf * && CMAKE_PREFIX_PATH=/home/zidar/utils/fftw-3.3.6-gcc/ \

cmake ../gromacs-2018-rc1 -DGMX_GPU=on -DGMX_MPI=off
-DGMX_FFT_LIBRARY=fftw3 \

-DCMAKE_INSTALL_PREFIX=/home/zidar/utils/gromacs-2018-rc1-gcc
-DGMX_SIMD=AVX2_256

make install

Relevant part from cmake:

-- Looking for NVIDIA GPUs present in the system

-- Number of NVIDIA GPUs detected: 3

-- Found CUDA: /usr/local/cuda (found suitable version "9.1", minimum
required is "6.5")

-- Found NVML: /usr/lib/x86_64-linux-gnu/libnvidia-ml.so

But when I run mdrun:

NOTE: GROMACS was configured without NVML support hence it can not exploit

      application clocks of the detected Tesla K40c GPU to improve
performance.

      Recompile with the NVML library (compatible with the driver used) or
set application clocks manually.

CUDA is installed in /usr/local/cuda-9.1 with a symlink to /usr/local/cuda.
The library file is physically present in /usr/lib/x86_64-linux-gnu/

Gromacs 2016.4 can be compiled without issues. The NVML library is detected
and properly linked:

ldd /home/zidar/utils/gromacs-2016.4-gcc/bin/gmx

linux-vdso.so.1 (0x00007ffc333ad000)

libgromacs.so.2 => /home/zidar/utils/gromacs-2016.4-gcc/lib/libgromacs.so.2
(0x00002ac6ada45000)

libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00002ac6b06c2000)

libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00002ac6b08df000)

librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00002ac6b0ae3000)

libnvidia-ml.so.1 => /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1
(0x00002ac6b0ceb000)

libhwloc.so.5 => /usr/lib/x86_64-linux-gnu/libhwloc.so.5
(0x00002ac6b12c5000)

libfftw3f.so.3 => /home/zidar/utils/fftw-3.3.6-gcc/lib/libfftw3f.so.3
(0x00002ac6b1501000)

libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00002ac6b1a2f000)

libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6
(0x00002ac6b1d33000)

libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00002ac6b20b5000)

libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00002ac6b22e2000)

libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00002ac6b24f9000)

/lib64/ld-linux-x86-64.so.2 (0x00002ac6ad5df000)

libnuma.so.1 => /usr/lib/x86_64-linux-gnu/libnuma.so.1 (0x00002ac6b2898000)

libltdl.so.7 => /usr/lib/x86_64-linux-gnu/libltdl.so.7 (0x00002ac6b2aa3000)

But not with neither Gromacs 2018 nor Gromacs 2018 rc1:

ldd /home/zidar/utils/gromacs-2018-rc1-gcc/bin/gmx

linux-vdso.so.1 (0x00007ffd2d5fd000)

libgromacs.so.3 =>
/home/zidar/utils/gromacs-2018-rc1-gcc/lib/libgromacs.so.3
(0x00002aad09c56000)

libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6
(0x00002aad0ca83000)

libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00002aad0ce05000)

libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00002aad0d109000)

libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00002aad0d336000)

libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00002aad0d54d000)

libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00002aad0d76a000)

librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00002aad0db09000)

libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00002aad0dd11000)

libcufft.so.9.1 => /usr/local/cuda/lib64/libcufft.so.9.1
(0x00002aad0df15000)

libhwloc.so.5 => /usr/lib/x86_64-linux-gnu/libhwloc.so.5
(0x00002aad15402000)

libfftw3f.so.3 => /home/zidar/utils/fftw-3.3.6-gcc/lib/libfftw3f.so.3
(0x00002aad1563e000)

/lib64/ld-linux-x86-64.so.2 (0x00002aad097ed000)

libnuma.so.1 => /usr/lib/x86_64-linux-gnu/libnuma.so.1 (0x00002aad15b6c000)

libltdl.so.7 => /usr/lib/x86_64-linux-gnu/libltdl.so.7 (0x00002aad15d77000)

Any idea where to start looking ?

Thanks,
Jernej