[gmx-users] Problem with CUDA

[Szilárd Páll]

Hi,

What is the reason for using the custom CMake options? What's the
-rdc=true for -- I don't think it's needed and it can very well be
causing the issue. Have you tried to actually do an
as-vanilla-as-possible build?

--
Szilárd


On Thu, Apr 5, 2018 at 6:52 PM, Borchert, Christopher B
ERDC-RDE-ITL-MS Contractor <Christopher.B.Borchert at erdc.dren.mil>
wrote:
> Hello. I'm taking a working build from a co-worker and trying to add GPU support on a Cray XC. CMake works but make fails. Both 2016 and 2018 die at the same point -- can't find gromac's own routines.
>
> 2016.5:
> /opt/cray/pe/craype/2.5.13/bin/CC    -march=core-avx2   -O2 -fPIC -dynamic -std=c++0x   -O3 -DNDEBUG -funroll-all-loops -fexcess-precision=fast      -dynamic CMakeFiles/template.dir/template.cpp.o  -o ../../bin/template -Wl,-rpath,/p/work/cots/gromacs-2016.5/build/lib:/opt/nvidia/cudatoolkit8.0/8.0.54_2.3.12_g180d272-2.2/lib64/stubs -dynamic ../../lib/libgromacs_mpi.so.2.5.0 -fopenmp -lcudart /opt/nvidia/cudatoolkit8.0/8.0.54_2.3.12_g180d272-2.2/lib64/stubs/libnvidia-ml.so -lhwloc -lz -ldl -lrt -lm -lfftw3f
> ../../lib/libgromacs_mpi.so.2.5.0: undefined reference to `__cudaRegisterLinkedBinary_59_tmpxft_0001bc78_00000000_21_pmalloc_cuda_compute_61_cpp1_ii_63d60154'
> ../../lib/libgromacs_mpi.so.2.5.0: undefined reference to `__cudaRegisterLinkedBinary_56_tmpxft_0001bac2_00000000_21_gpu_utils_compute_61_cpp1_ii_d70ebee0'
> ../../lib/libgromacs_mpi.so.2.5.0: undefined reference to `__cudaRegisterLinkedBinary_56_tmpxft_0001b90b_00000000_21_cudautils_compute_61_cpp1_ii_24d20763'
> ../../lib/libgromacs_mpi.so.2.5.0: undefined reference to `__cudaRegisterLinkedBinary_71_tmpxft_0001c016_00000000_21_cuda_version_information_compute_61_cpp1_ii_e35285be'
> ../../lib/libgromacs_mpi.so.2.5.0: undefined reference to `__cudaRegisterLinkedBinary_57_tmpxft_0001b592_00000000_21_nbnxn_cuda_compute_61_cpp1_ii_6e47f057'
> ../../lib/libgromacs_mpi.so.2.5.0: undefined reference to `__cudaRegisterLinkedBinary_67_tmpxft_0001b754_00000000_21_nbnxn_cuda_data_mgmt_compute_61_cpp1_ii_a1eafeba'
> collect2: error: ld returned 1 exit status
>
> 2018.1:
> /opt/cray/pe/craype/2.5.13/bin/CC    -march=core-avx2   -O2 -fPIC -dynamic -std=c++11   -O3 -DNDEBUG -funroll-all-loops -fexcess-precision=fast      -dynamic CMakeFiles/template.dir/template.cpp.o  -o ../../bin/template -Wl,-rpath,/p/work/cots/gromacs-2018.1/build/lib ../../lib/libgromacs_mpi.so.3.1.0 -fopenmp -lm
> ../../lib/libgromacs_mpi.so.3.1.0: undefined reference to `__cudaRegisterLinkedBinary_56_tmpxft_000068a5_00000000_21_pme_3dfft_compute_61_cpp1_ii_79dff388'
> ../../lib/libgromacs_mpi.so.3.1.0: undefined reference to `__cudaRegisterLinkedBinary_67_tmpxft_00006621_00000000_21_nbnxn_cuda_data_mgmt_compute_61_cpp1_ii_a1eafeba'
> ../../lib/libgromacs_mpi.so.3.1.0: undefined reference to `__cudaRegisterLinkedBinary_57_tmpxft_00006f47_00000000_21_pme_spread_compute_61_cpp1_ii_d982d3ad'
> ../../lib/libgromacs_mpi.so.3.1.0: undefined reference to `__cudaRegisterLinkedBinary_56_tmpxft_00006d70_00000000_21_pme_solve_compute_61_cpp1_ii_06051a94'
> ../../lib/libgromacs_mpi.so.3.1.0: undefined reference to `__cudaRegisterLinkedBinary_59_tmpxft_00008da7_00000000_21_pmalloc_cuda_compute_61_cpp1_ii_63d60154'
> ../../lib/libgromacs_mpi.so.3.1.0: undefined reference to `__cudaRegisterLinkedBinary_50_tmpxft_00007930_00000000_21_pme_compute_61_cpp1_ii_6dbf966c'
> ../../lib/libgromacs_mpi.so.3.1.0: undefined reference to `__cudaRegisterLinkedBinary_58_tmpxft_00007382_00000000_21_pme_timings_compute_61_cpp1_ii_75ae0e44'
> ../../lib/libgromacs_mpi.so.3.1.0: undefined reference to `__cudaRegisterLinkedBinary_57_tmpxft_00006b11_00000000_21_pme_gather_compute_61_cpp1_ii_a7a2f9c7'
> ../../lib/libgromacs_mpi.so.3.1.0: undefined reference to `__cudaRegisterLinkedBinary_56_tmpxft_00007f9f_00000000_21_cudautils_compute_61_cpp1_ii_25933dd5'
> ../../lib/libgromacs_mpi.so.3.1.0: undefined reference to `__cudaRegisterLinkedBinary_54_tmpxft_000088f9_00000000_21_pinning_compute_61_cpp1_ii_5d0f4aae'
> ../../lib/libgromacs_mpi.so.3.1.0: undefined reference to `__cudaRegisterLinkedBinary_57_tmpxft_000039b7_00000000_21_nbnxn_cuda_compute_61_cpp1_ii_f147f02c'
> ../../lib/libgromacs_mpi.so.3.1.0: undefined reference to `__cudaRegisterLinkedBinary_71_tmpxft_000091d4_00000000_21_cuda_version_information_compute_61_cpp1_ii_8ab8dc1d'
> ../../lib/libgromacs_mpi.so.3.1.0: undefined reference to `__cudaRegisterLinkedBinary_56_tmpxft_00008407_00000000_21_gpu_utils_compute_61_cpp1_ii_70828085'
> collect2: error: ld returned 1 exit status
>
> BUILD INSTRUCTIONS:
> module swap PrgEnv-cray PrgEnv-gnu
> module swap gcc gcc/5.3.0
> export CRAYPE_LINK_TYPE=dynamic
>
> module load cudatoolkit/8.0.54_2.3.12_g180d272-2.2
> module load cmake/gcc-6.3.0/3.7.2
> module load fftw/3.3.4.11
> export BOOST_DIR=/app/unsupported/boost/1.64.0-gcc-6.3.0
>
> export PREFIX=/app/unsupported/gromacs/201x.x
> mkdir $PREFIX
>
> cmake ..  \
>  -DCMAKE_VERBOSE_MAKEFILE:BOOL=TRUE \
>  -DCMAKE_C_COMPILER:FILEPATH=`which cc` \
>  -DCMAKE_C_FLAGS:STRING="-O2 -fPIC -dynamic" \
>  -DCMAKE_CXX_COMPILER:FILEPATH=`which CC` \
>  -DCMAKE_CXX_FLAGS:STRING="-O2 -fPIC -dynamic" \
>  -DCMAKE_INSTALL_PREFIX:PATH=$PREFIX \
>  -DGMX_FFT_LIBRARY=fftw3 \
>  -DCMAKE_EXE_LINKER_FLAGS:STRING=-dynamic \
>  -DGMX_SIMD=AVX2_256 \
>  -DFFTWF_LIBRARY:FILEPATH=${FFTW_DIR}/libfftw3f.so \
>  -DFFTWF_INCLUDE_DIR:PATH=$FFTW_INC \
>  -DBoost_DIR:PATH=$BOOST_DIR \
>  -DBoost_INCLUDE_DIR:PATH=${BOOST_DIR}/include \
>  -DBoost_LIBRARY_DIR:PATH=${BOOST_DIR}/lib \
>  -DGMX_MPI=ON \
>  -DGMX_GPU=ON \
>  -DCUDA_NVCC_FLAGS:STRING="-rdc=true" \
>  -DCUDA_USE_STATIC_CUDA_RUNTIME:BOOL=OFF
>
> make 2>&1 | tee make.log
>
> Thanks,
> Chris
>
> --
> Gromacs Users mailing list
>
> * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a mail to gmx-users-request at gromacs.org.