[gmx-users] Install GROMACS 2018 with CUDA features in dynamic linking way

Tue Dec 11 10:44:38 CET 2018

I'm trying to rewrite the RPATH because shared libraries paths used by
GROMACS are hardcoded in the binary.

ldd /nfs2/opt/APPS/x86_64/APPS/GROMACS/2016/CUDA/8.0/bin/gmx
linux-vdso.so.1 =>  (0x00007ffddf1d3000)
libgromacs.so.2 =>
/nfs2/opt/APPS/x86_64/APPS/GROMACS/2016/CUDA/8.0/bin/../lib64/libgromacs.so.2
(0x00007f0094b25000)
libcudart.so.8.0 => not found
libnvidia-ml.so.1 => /lib64/libnvidia-ml.so.1 (0x00007f0094500000)
libz.so.1 => /lib64/libz.so.1 (0x00007f00942ea000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f00940e5000)
librt.so.1 => /lib64/librt.so.1 (0x00007f0093edd000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f0093cc1000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f00939b7000)
libm.so.6 => /lib64/libm.so.6 (0x00007f00936b5000)
libgomp.so.1 => /lib64/libgomp.so.1 (0x00007f009348f000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f0093278000)
libc.so.6 => /lib64/libc.so.6 (0x00007f0092eb7000)
libcudart.so.8.0 => /nfs2/LIBS/x86_64/LIBS/CUDA/8.0/lib64/libcudart.so.8.0
(0x00007f0092c50000)
/lib64/ld-linux-x86-64.so.2 (0x00007f0097ad2000)

ldd /nfs2/opt/APPS/x86_64/APPS/GROMACS/2018/CUDA/8.0/bin/gmx
linux-vdso.so.1 =>  (0x00007fff27b8d000)
libgromacs.so.3 =>
/nfs2/opt/APPS/x86_64/APPS/GROMACS/2018/CUDA/8.0/bin/../lib64/libgromacs.so.3
(0x00007fcb4aa3e000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fcb4a71f000)
libm.so.6 => /lib64/libm.so.6 (0x00007fcb4a41d000)
libgomp.so.1 => /lib64/libgomp.so.1 (0x00007fcb4a1f7000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fcb49fe0000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fcb49dc4000)
libc.so.6 => /lib64/libc.so.6 (0x00007fcb49a03000)
libcudart.so.8.0 => /nfs2/LIBS/x86_64/LIBS/CUDA/8.0/lib64/libcudart.so.8.0
(0x00007fcb4979c000)
libcufft.so.8.0 => /nfs2/LIBS/x86_64/LIBS/CUDA/8.0/lib64/libcufft.so.8.0
(0x00007fcb4094e000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007fcb40749000)
librt.so.1 => /lib64/librt.so.1 (0x00007fcb40541000)
libfftw3f.so.3 =>
/nfs2/LIBS/x86_64/LIBS/FFTW/3.3.3/SINGLE/lib/libfftw3f.so.3
(0x00007fcb401c8000)
libmkl_intel_lp64.so =>
/nfs2/LIBS/x86_64/LIBS/mkl/l_mkl_11.1.0.080/composer_xe_2013_sp1.0.080/mkl/lib/intel64/libmkl_intel_lp64.so
(0x00007fcb3faa4000)
libmkl_intel_thread.so =>
/nfs2/LIBS/x86_64/LIBS/mkl/l_mkl_11.1.0.080/composer_xe_2013_sp1.0.080/mkl/lib/intel64/libmkl_intel_thread.so
(0x00007fcb3ea0a000)
libmkl_core.so =>
/nfs2/LIBS/x86_64/LIBS/mkl/l_mkl_11.1.0.080/composer_xe_2013_sp1.0.080/mkl/lib/intel64/libmkl_core.so
(0x00007fcb3d4dc000)
libiomp5.so =>
/nfs2/LIBS/x86_64/LIBS/mkl/l_mkl_11.1.0.080/lib/intel64/libiomp5.so
(0x00007fcb3d1c2000)
libmkl_gf_lp64.so =>
/nfs2/LIBS/x86_64/LIBS/mkl/l_mkl_11.1.0.080/composer_xe_2013_sp1.0.080/mkl/lib/intel64/libmkl_gf_lp64.so
(0x00007fcb3caa0000)
/lib64/ld-linux-x86-64.so.2 (0x00007fcb4d785000)

See the differences between the 2016 & 2018 version.

I'm using Cmake 3.13.1.

~/cmake-3.13.1-Linux-x86_64/bin/cmake .. -DGMX_BUILD_OWN_FFTW=ON
-DREGRESSIONTEST_DOWNLOAD=ON -DGMX_GPU=ON
-DCUDA_TOOLKIT_ROOT_DIR=/nfs2/LIBS/x86_64/LIBS/CUDA/8.0
-DCMAKE_INSTALL_PREFIX=/nfs2/opt/APPS/x86_64/APPS/GROMACS/2018/CUDA/8.0
-DCUDA_USE_STATIC_CUDA_RUNTIME=OFF -DBUILD_SHARED_LIBS=ON
-DCUDA_NVCC_FLAGS=--cudart=shared -DGMX_PREFER_STATIC_LIBS=OFF
-DEXTRA_NVCCFLAGS=--cudart=shared

I think I've tried almost everything.

Regards.

El lun., 10 dic. 2018 a las 16:09, Szilárd Páll (<pall.szilard at gmail.com>)
escribió:

> On Sat, Dec 8, 2018 at 10:00 PM Gmail <jaisiero at gmail.com> wrote:
> >
> > My mistake! It was a typo. Anyway, this is the result before executing
> > the chrpath command:
> >
> > chrpath -l $APPS/GROMACS/2018/CUDA/8.0/bin/gmx
> > $APPS/GROMACS/2018/CUDA/8.0/bin/gmx: RPATH=$ORIGIN/../lib64
> >
> > I'm suspicious that GROMACS 2018 is not being compiled using shared
> > libraries, at least, for CUDA.
>
> First of all, what is the goal, why are you trying to manually rewrite
> the binary RPATH?
>
> Well, if the binaries not linked against libcudart.so than it clearly
> isn't (and the ldd output is a better way to confirm that -- a library
> can be linked against gmx even without an RPATH being set).
>
> I have a vague memory that this may have been the default in CMake or
> perhaps it changed at some point. What's your CMake version, perhaps
> you're using an old CMake?
>
> >
> > Jaime.
> >
> >
> > On 8/12/18 21:39, Mark Abraham wrote:
> > > Hi,
> > >
> > > Your final line doesn't match your CMAKE_INSTALL_PREFIX
> > >
> > > Mark
> > >
> > > On Sun., 9 Dec. 2018, 07:00 Jaime Sierra <jaisiero at gmail.com wrote:
> > >
> > >> Hi pall,
> > >>
> > >> thanks for your answer,
> > >> I have my own "HOW_TO_INSTALL" guide like:
> > >>
> > >> $ wget ftp://ftp.gromacs.org/pub/gromacs/gromacs-5.1.4.tar.gz
> > >> $ tar xzf gromacs-5.1.4.tar.gz
> > >> $ cd gromacs-5.1.4.tar.gz
> > >> $ mkdir build
> > >> $ cd build
> > >> $ export EXTRA_NVCCFLAGS=--cudart=shared
> > >> $ export PATH=$APPS/CMAKE/2.8.12.2/bin/:$PATH
> > >> $ cmake .. -DCMAKE_INSTALL_PREFIX=$APPS/GROMACS/5.1.4/CUDA8.0/GPU
> > >> -DGMX_FFT_LIBRARY=fftw3 -DCMAKE_PREFIX_PATH=$LIBS/FFTW/3.3.3/SINGLE/
> > >> -DGMX_GPU=ON -DCUDA_TOOLKIT_ROOT_DIR=$LIBS/CUDA/8.0/
> > >> $ make -j $(nproc)
> > >> $ make install
> > >> $ chrpath -r '$ORIGIN/../lib64' $APPS/GROMACS/5.1.4/GPU/bin/gmx
> > >>
> > >> that works until GROMACS 2016, I couldn't make it work for GROMACS
> 2018.
> > >>
> > >> Regards,
> > >>
> > >> Jaime.
> > >>
> > >> El vie., 7 dic. 2018 a las 15:49, Szilárd Páll (<
> pall.szilard at gmail.com>)
> > >> escribió:
> > >>
> > >>> Hi Jaime,
> > >>>
> > >>> Have you tried passing that variable to nvcc? Does it not work?
> > >>>
> > >>> Note that GROMACS makes up to a dozen of CUDA runtime calls (kernels
> > >>> and transfers) per iteration with iteration times in the range of
> > >>> milliseconds at longest and peak in the hundreds of nanoseconds and
> > >>> the CPU needs to sync up every iteration with the GPU. Hence, I
> > >>> suspect GROMACS may be a challenging use-case for rCUDA, but I'm very
> > >>> interested in your observations and benchmarks results when you have
> > >>> some.
> > >>>
> > >>> Cheers,
> > >>> On Fri, Dec 7, 2018 at 10:39 AM Jaime Sierra <jaisiero at gmail.com>
> wrote:
> > >>>> Hi,
> > >>>>
> > >>>> my name is Jaime Sierra, a researcher from Polytechnic University of
> > >>>> Valencia, Spain. I would like to know how to compile & install
> GROMACS
> > >>>> 2018 with CUDA features with the "--cudart=shared" compilation
> option
> > >> to
> > >>>> use it with our rCUDA software.
> > >>>>
> > >>>>
> > >>>> We haven't had this problem in previous releases of GROMACS like
> 2016,
> > >>>> 5.1.4 and so on.
> > >>>>
> > >>>>
> > >>>> Regards,
> > >>>>
> > >>>> Jaime.
> > >>>> --
> > >>>> Gromacs Users mailing list
> > >>>>
> > >>>> * Please search the archive at
> > >>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> > >>> posting!
> > >>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> > >>>>
> > >>>> * For (un)subscribe requests visit
> > >>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users
> or
> > >>> send a mail to gmx-users-request at gromacs.org.
> > >>> --
> > >>> Gromacs Users mailing list
> > >>>
> > >>> * Please search the archive at
> > >>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> > >>> posting!
> > >>>
> > >>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> > >>>
> > >>> * For (un)subscribe requests visit
> > >>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users
> or
> > >>> send a mail to gmx-users-request at gromacs.org.
> > >>>
> > >> --
> > >> Gromacs Users mailing list
> > >>
> > >> * Please search the archive at
> > >> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> > >> posting!
> > >>
> > >> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> > >>
> > >> * For (un)subscribe requests visit
> > >> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> > >> send a mail to gmx-users-request at gromacs.org.
> > --
> > Gromacs Users mailing list
> >
> > * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
> >
> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >
> > * For (un)subscribe requests visit
> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.