[gmx-users] gromacs 5.1.2 mdrun can't detect GPU

[Szilárd Páll]

On Thu, Apr 21, 2016 at 12:22 AM, treinz <treinz at 163.com> wrote:
> Hi all,
>
>
> I recently built 5.1.2 with GPU support and the config options are:
>
>
> module load cuda/7.5.18
> cmake .. -DCMAKE_C_COMPILER=gcc-4.9 \
>          -DCMAKE_CXX_COMPILER=g++-4.9 \
>          -DGMX_MPI=OFF \
>          -DGMX_THREAD_MPI=ON \
>          -DGMX_GPU=ON \
>          -DCMAKE_PREFIX_PATH=$HOME/local \
>          -DCMAKE_INSTALL_PREFIX=$HOME/local/gromacs/grid_frontend \
>          -DGMX_BUILD_OWN_FFTW=ON \
>          -DGMX_DEFAULT_SUFFIX=OFF \
>          -DGMX_BINARY_SUFFIX=_gpu \
>          -DGMX_LIBS_SUFFIX=_gpu
>
>
> and the installation was successful. But when I tried running mdrun, it wasn't able to detect the GPU:
>
>
> Build OS/arch:      Linux 2.6.32-573.1.1.el6.x86_64 x86_64
> Build CPU vendor:   GenuineIntel
> Build CPU brand:    Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
> Build CPU family:   6   Model: 63   Stepping: 2
> Build CPU features: aes apic avx avx2 clfsh cmov cx8 cx16 f16c fma htt lahf_lm mmx msr nonstop_tsc pcid pclmuldq pdcm pdpe1gb popcnt pse rdrnd rdtscp sse2 sse3 sse4.1 sse4.2 ssse3 tdt x2apic
> C compiler:         /net/noble/vol1/home/dejunlin/local/bin/mpicc GNU 4.9.3
> C compiler flags:    -march=core-avx2    -Wextra -Wno-missing-field-initializers -Wno-sign-compare -Wpointer-arith -Wall -Wno-unused -Wunused-value -Wunused-parameter  -O3 -DNDEBUG -funroll-all-loops -fexcess-precision=fast  -Wno-array-bounds
> C++ compiler:       /net/noble/vol1/home/dejunlin/local/bin/mpicxx GNU 4.9.3
> C++ compiler flags:  -march=core-avx2    -Wextra -Wno-missing-field-initializers -Wpointer-arith -Wall -Wno-unused-function  -O3 -DNDEBUG -funroll-all-loops -fexcess-precision=fast  -Wno-array-bounds
> Boost version:      1.59.0 (external)
> CUDA compiler:      /net/gs/vol3/software/modules-sw/cuda/7.5.18/Linux/RHEL6/x86_64/bin/nvcc nvcc: NVIDIA (R) Cuda compiler driver;Copyright (c) 2005-2015 NVIDIA Corporation;Built on Tue_Aug_11_14:27:32_CDT_2015;Cuda compilation tools, release 7.5, V7.5.17
> CUDA compiler flags:-gencode;arch=compute_20,code=sm_20;-gencode;arch=compute_30,code=sm_30;-gencode;arch=compute_35,code=sm_35;-gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_52,code=sm_52;-gencode;arch=compute_52,code=compute_52;-use_fast_math;; ;-march=core-avx2;-Wextra;-Wno-missing-field-initializers;-Wpointer-arith;-Wall;-Wno-unused-function;-O3;-DNDEBUG;-funroll-all-loops;-fexcess-precision=fast;-Wno-array-bounds;
> CUDA driver:        0.0
> CUDA runtime:       0.0
>
>
> NOTE: Error occurred during GPU detection:
>       CUDA driver version is insufficient for CUDA runtime version
>       Can not use GPU acceleration, will fall back to CPU kernels.
>

^^^ This is the message you should be focusing on. For some reason,
the runtime mdrun was compiled with is incompatible with the driver.
This should not be the case if the driver is indeed 352.xx (which is
driver API v 7.5) as shown below and the runtime API v 7.5.

> The command lines for this run is:
>
>
> module load cuda/7.5.18
> source $HOME/local/gromacs/grid_frontend/bin/GMXRC
> env
> nvidia-smi -a
> gmx_gpu mdrun -ntmpi 24 -ntomp 1 -gpu_id 1 -deffnm

FYI: That's going to be inefficient, probably the worst possible
option in fact. Try 1-6 ranks/GPU.

>
> I look at the stdout from nvidia-smi, it looks like the nvidia driver was installed (there are 4 GPUs but I'm only showing one of them):
>
>
> ==============NVSMI LOG==============
>
>
> Timestamp                           : Wed Apr 20 15:05:52 2016
> Driver Version                      : 352.39
>
>
> Attached GPUs                       : 4
> GPU 0000:02:00.0
>     Product Name                    : Tesla K40c
>     Product Brand                   : Tesla
>     Display Mode                    : Disabled
>     Display Active                  : Disabled
>     Persistence Mode                : Disabled
>     Accounting Mode                 : Disabled
>     Accounting Mode Buffer Size     : 1920
>     Driver Model
>         Current                     : N/A
>         Pending                     : N/A
>     Serial Number                   : 0321715040048
>     GPU UUID                        : GPU-647b8474-e09f-7f98-3ac5-4604e02f1c75
>     Minor Number                    : 0
>     VBIOS Version                   : 80.80.3E.00.02
>     MultiGPU Board                  : No
>     Board ID                        : 0x200
>     Inforom Version
>         Image Version               : 2081.0206.01.04
>         OEM Object                  : 1.1
>         ECC Object                  : 3.0
>         Power Management Object     : N/A
>     GPU Operation Mode
>
>
> One catch is that I didn't have the CUDA GPU Deployment Kit or the NVML library installed -- would that matter in detecting GPUs? Another catch is that I was running the job in a SGE cluster. I believe I have requested the GPU resources correctly because I could compile GROMACS with GPU support on the GPU nodes, although the resulting mdrun still wasn't able to detect any GPU if I ran it on the same GPU node where it was compiled.

No! NVML allows mdrun to check/adjust the application clock. It's optional.

Cheers,
--
Szilárd

>
> Can anyone tell me if there's any flag I have to turn on during compilation or running mdrun in order to make it see the GPUs? Also, is NVML or the dev kit a must for this?
>
>
> Thanks,
> Tim
> --
> Gromacs Users mailing list
>
> * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a mail to gmx-users-request at gromacs.org.