[gmx-users] NVML library in CUDA 9 & Gromacs 2018

[Mark Abraham]

Hi,

The default changed in GROMACS 2018 - even if NVML is found, we do not link
with it by default. Use cmake -DGMX_USE_NVML=on. Be advised that you may
need to take care that the nvml library found at run time is compatible
with the CUDA version used at compilation time. That can be non-trivial to
automate, so we have left that issue to users who have the desire and
ability to change the clocks (generally requires root privilege / sysadmin
approval).

Mark

On Fri, Jan 12, 2018 at 9:19 AM Jernej Zidar <jernej.zidar at gmail.com> wrote:

> Hi,
>
> I have an interesting problem with the  NVML library from CUDA 9 (and 9.1)
> and Gromacs 2018. Namely, the NVML is reported as detected during cmake but
> Gromacs 2018 is linked without it.
>
> This is the cmake command I would issue:
>
> m -rf * && CMAKE_PREFIX_PATH=/home/zidar/utils/fftw-3.3.6-gcc/ \
>
> cmake ../gromacs-2018-rc1 -DGMX_GPU=on -DGMX_MPI=off
> -DGMX_FFT_LIBRARY=fftw3 \
>
> -DCMAKE_INSTALL_PREFIX=/home/zidar/utils/gromacs-2018-rc1-gcc
> -DGMX_SIMD=AVX2_256
>
> make install
>
> Relevant part from cmake:
>
> -- Looking for NVIDIA GPUs present in the system
>
> -- Number of NVIDIA GPUs detected: 3
>
> -- Found CUDA: /usr/local/cuda (found suitable version "9.1", minimum
> required is "6.5")
>
> -- Found NVML: /usr/lib/x86_64-linux-gnu/libnvidia-ml.so
>
> But when I run mdrun:
>
> NOTE: GROMACS was configured without NVML support hence it can not exploit
>
>       application clocks of the detected Tesla K40c GPU to improve
> performance.
>
>       Recompile with the NVML library (compatible with the driver used) or
> set application clocks manually.
>
> CUDA is installed in /usr/local/cuda-9.1 with a symlink to /usr/local/cuda.
> The library file is physically present in /usr/lib/x86_64-linux-gnu/
>
> Gromacs 2016.4 can be compiled without issues. The NVML library is detected
> and properly linked:
>
> ldd /home/zidar/utils/gromacs-2016.4-gcc/bin/gmx
>
> linux-vdso.so.1 (0x00007ffc333ad000)
>
> libgromacs.so.2 => /home/zidar/utils/gromacs-2016.4-gcc/lib/libgromacs.so.2
> (0x00002ac6ada45000)
>
> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
> (0x00002ac6b06c2000)
>
> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00002ac6b08df000)
>
> librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00002ac6b0ae3000)
>
> libnvidia-ml.so.1 => /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1
> (0x00002ac6b0ceb000)
>
> libhwloc.so.5 => /usr/lib/x86_64-linux-gnu/libhwloc.so.5
> (0x00002ac6b12c5000)
>
> libfftw3f.so.3 => /home/zidar/utils/fftw-3.3.6-gcc/lib/libfftw3f.so.3
> (0x00002ac6b1501000)
>
> libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00002ac6b1a2f000)
>
> libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6
> (0x00002ac6b1d33000)
>
> libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00002ac6b20b5000)
>
> libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00002ac6b22e2000)
>
> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00002ac6b24f9000)
>
> /lib64/ld-linux-x86-64.so.2 (0x00002ac6ad5df000)
>
> libnuma.so.1 => /usr/lib/x86_64-linux-gnu/libnuma.so.1 (0x00002ac6b2898000)
>
> libltdl.so.7 => /usr/lib/x86_64-linux-gnu/libltdl.so.7 (0x00002ac6b2aa3000)
>
> But not with neither Gromacs 2018 nor Gromacs 2018 rc1:
>
> ldd /home/zidar/utils/gromacs-2018-rc1-gcc/bin/gmx
>
> linux-vdso.so.1 (0x00007ffd2d5fd000)
>
> libgromacs.so.3 =>
> /home/zidar/utils/gromacs-2018-rc1-gcc/lib/libgromacs.so.3
> (0x00002aad09c56000)
>
> libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6
> (0x00002aad0ca83000)
>
> libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00002aad0ce05000)
>
> libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00002aad0d109000)
>
> libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00002aad0d336000)
>
> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
> (0x00002aad0d54d000)
>
> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00002aad0d76a000)
>
> librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00002aad0db09000)
>
> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00002aad0dd11000)
>
> libcufft.so.9.1 => /usr/local/cuda/lib64/libcufft.so.9.1
> (0x00002aad0df15000)
>
> libhwloc.so.5 => /usr/lib/x86_64-linux-gnu/libhwloc.so.5
> (0x00002aad15402000)
>
> libfftw3f.so.3 => /home/zidar/utils/fftw-3.3.6-gcc/lib/libfftw3f.so.3
> (0x00002aad1563e000)
>
> /lib64/ld-linux-x86-64.so.2 (0x00002aad097ed000)
>
> libnuma.so.1 => /usr/lib/x86_64-linux-gnu/libnuma.so.1 (0x00002aad15b6c000)
>
> libltdl.so.7 => /usr/lib/x86_64-linux-gnu/libltdl.so.7 (0x00002aad15d77000)
>
> Any idea where to start looking ?
>
> Thanks,
> Jernej
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
>