[gmx-users] Compile Gromacs with OpenCL for MacBook Pro with AMD Radeon Pro 560 GPU

[Michael Williams]

Hi again Szilárd, I just wanted to follow up on my last message. I found no difference after modifying the file you suggested in Gromacs 2018.5. However, I did find that if I compiled the unmodified 2018.5 code and ran “make install” then (after sourcing the GMXRC file) I could run "./gmxtest.pl all" in the regression test folder and all of those tests passed. (It doesn’t seem like that runs all of the same tests as “make check”, though). In any case, even the tests that fail if doing “make check” directly after compiling only fail because a header file couldn’t be found; none of the tests failed due to returning a value outside of tolerance. Also, mdrun seems to be utilizing the GPU as expected. For the record, here is the cmake command I used:

cmake .. -DCMAKE_INSTALL_PREFIX=/Users/michael/.local/apps/gromacs-2018.5-apple-clang-omp-ocl -DCMAKE_LIBRARY_PATH=/Users/michael/.local/lib -DCMAKE_INCLUDE_PATH=/Users/michael/.local/include -DCMAKE_C_COMPILER=/usr/bin/clang -DCMAKE_CXX_COMPILER=/usr/bin/clang++ -DCMAKE_C_FLAGS="-Xpreprocessor -fopenmp -lomp -L/Users/michael/.local/lib -I/Users/michael/.local/include" -DCMAKE_CXX_FLAGS="-Xpreprocessor -fopenmp -lomp -L/Users/michael/.local/lib -I/Users/michael/.local/include" -DGMX_FFT_LIBRARY=fftw3 -DGMX_GPU=ON -DGMX_USE_OPENCL=ON

My $HOME/.local directory is the prefix I used to install appropriate versions of hwloc (1.11.12), libomp (7.0.1), and fftw3 (3.3.8) that I compiled with the system’s default clang (in OSX 10.14.3, Mojave). Thanks again,


Mike


> On Feb 21, 2019, at 11:35 PM, Szilárd Páll <pall.szilard at gmail.com> wrote:
> 
> Michael,
> 
> That was my copy-paste mistake, I meant to suggest editing the following
> file:
> src/gromacs/gpu_utils/ocl_compiler.cpp
> 
> Please try this and run make check to see if it can now find included files.
> 
> --
> Szilárd
> 
> 
> On Tue, Feb 19, 2019 at 10:59 PM Michael Williams <
> michael.r.c.williams at gmail.com <mailto:michael.r.c.williams at gmail.com>> wrote:
> 
>> Hi Szilárd, thank you for the suggestion. I modified the file
>> "src/gromacs/gpu_utils/gpu_utils_ocl.cpp” and replaced both instances of
>> “#ifdef __APPLE__” with “#if 0”. I then checked that the build and
>> regression tests worked without enabling the GPU support. All of the tests
>> passed. Then, in a new build folder, I added -DGMX_GPU=ON and
>> -DGMX_USE_OPENCL=ON to the Cmake instructions (see below). I did this
>> separately for the unmodified version of GROMACS 2018.5 as well as inside a
>> separate copy where I’d made the above changes. When I ran “make check” for
>> both the unmodified and modified versions, I got the same error as before
>> (logs pasted below). However, it seems to happen at different parts of the
>> test suite: the unmodified version passed the MdrunTests, then failed on
>> one of the MdrunMpiTests. The modified version of GROMACS failed during
>> MdrunTests and then passed the MdrunMpiTests.
>> 
>> During both runs of the tests I got a system popup asking if I wanted to
>> allow the test executable that was running to accept incoming network
>> connections. The window was only visible for a few seconds and I didn’t
>> click either option (allow / deny) before the window went away again.
>> 
>> Thanks again for your help, and if you have any other ideas I’d be quite
>> willing to try them out and let you know the results.
>> 
>> 
>> Mike
>> 
>> 
>> Build settings on MacBook Pro (OSX 10.14.3, Mojave) using system clang and
>> OpenMP library (from LLVM 7.0.1) built in custom path (no other parts of
>> LLVM 7.0.1 are installed in this path):
>> 
>> cmake ..
>> -DCMAKE_INSTALL_PREFIX=/Users/michael/.local/apps/gromacs-2018.5-apple-clang-omp
>> \
>>                -DCMAKE_PREFIX_PATH=/Users/michael/.local \
>>                -DCMAKE_C_COMPILER=/usr/bin/clang \
>>                -DCMAKE_CXX_COMPILER=/usr/bin/clang++ \
>>                -DCMAKE_C_FLAGS="-Wno-deprecated-declarations
>> -Wno-unused-command-line-argument -Xpreprocessor -fopenmp -lomp
>> -I/Users/michael/.local/include -L/Users/michael/.local/lib” \
>>                -DCMAKE_CXX_FLAGS="-Wno-deprecated-declarations
>> -Wno-unused-command-line-argument -Xpreprocessor -fopenmp -lomp
>> -I/Users/michael/.local/include -L/Users/michael/.local/lib” \
>>                -DGMX_FFT_LIBRARY=fftw3 \
>>                -DGMX_GPU=ON \
>>                -DGMX_USE_OPENCL=ON \
>> 
>> -DREGRESSIONTEST_PATH=/Users/michael/.local/source/regressiontests-2018.5
>> 
>> (Note: tabs added to make the above command easier to read)
>> 
>> 
>> ***ERROR LOG WITH UNMODIFIED GPU_UTILS_OCL.CPP:
>> 
>> Can not increase nstlist because an NVE ensemble is used
>> Using 2 MPI threads
>> Using 1 OpenMP thread per tMPI thread
>> 
>> On host veronika.local 1 GPU auto-selected for this run.
>> Mapping of GPU IDs to the 2 GPU tasks in the 2 ranks on this node:
>>  PP:0,PP:0
>> 
>> NOTE: Your choice of number of MPI ranks and amount of resources results
>> in using 1 OpenMP threads per rank, which is most likely inefficient. The
>> optimum is usually between 2 and 6 threads per rank.
>> Compilation of source file
>> /Users/michael/.local/source/gromacs-2018.5/src/gromacs/mdlib/nbnxn_ocl/
>> nbnxn_ocl_kernels.cl failed!
>> -- Used build options: -DWARP_SIZE_TEST=64 -D_AMD_SOURCE_
>> -DGMX_OCL_FASTGEN -DEL_CUTOFF -DEELNAME=_ElecCut -DLJ_COMB_GEOM
>> -DVDWNAME=_VdwLJCombGeom -DCENTRAL=22
>> -DNBNXN_GPU_NCLUSTER_PER_SUPERCLUSTER=8 -DNBNXN_GPU_CLUSTER_SIZE=8
>> -DNBNXN_GPU_JGROUP_SIZE=4 -DGMX_NBNXN_PRUNE_KERNEL_J4_CONCURRENCY=4
>> -DNBNXN_MIN_RSQ=3.82e-07f -DIATYPE_SHMEM -cl-fast-relaxed-math
>> -cl-denorms-are-zero
>> -I/Users/michael/.local/source/gromacs-2018.5/src/gromacs/mdlib/nbnxn_ocl
>> --------------LOG START---------------
>> <program source>:80:10: fatal error: 'nbnxn_ocl_kernel_pruneonly.clh' file
>> not found
>> #include "nbnxn_ocl_kernel_pruneonly.clh"
>>         ^
>> ---------------LOG END----------------
>> 
>> -------------------------------------------------------
>> Program:     mdrun-mpi-test, version 2018.5
>> Source file: src/gromacs/gpu_utils/ocl_compiler.cpp (line 507)
>> Function:    cl_program gmx::ocl::compileProgram(FILE *, const std::string
>> &, const std::string &, cl_context, cl_device_id, ocl_vendor_id_t)
>> MPI rank:    1 (out of 2)
>> 
>> Internal error (bug):
>> Failed to compile NBNXN kernels for GPU #AMD Radeon Pro 560 Compute Engine
>>  Could not build OpenCL program, error was CL_BUILD_PROGRAM_FAILURE
>> 
>> ***ERROR LOG WITH MODIFIED GPU_UTILS_OCL.CPP:
>> 
>> Can not increase nstlist because an NVE ensemble is used
>> Using 1 MPI thread
>> Using 1 OpenMP thread
>> 
>> 1 GPU auto-selected for this run.
>> Mapping of GPU IDs to the 1 GPU task in the 1 rank on this node:
>>  PP:0
>> Compilation of source file
>> /Users/michael/.local/source/gromacs-2018.5-mod/src/gromacs/mdlib/nbnxn_ocl/
>> nbnxn_ocl_kernels.cl failed!
>> -- Used build options: -DWARP_SIZE_TEST=64 -D_AMD_SOURCE_
>> -DGMX_OCL_FASTGEN -DEL_CUTOFF -DEELNAME=_ElecCut -DLJ_COMB_GEOM
>> -DVDWNAME=_VdwLJCombGeom -DCENTRAL=22
>> -DNBNXN_GPU_NCLUSTER_PER_SUPERCLUSTER=8 -DNBNXN_GPU_CLUSTER_SIZE=8
>> -DNBNXN_GPU_JGROUP_SIZE=4 -DGMX_NBNXN_PRUNE_KERNEL_J4_CONCURRENCY=4
>> -DNBNXN_MIN_RSQ=3.82e-07f -DIATYPE_SHMEM -cl-fast-relaxed-math
>> -cl-denorms-are-zero
>> -I/Users/michael/.local/source/gromacs-2018.5-mod/src/gromacs/mdlib/nbnxn_ocl
>> --------------LOG START---------------
>> <program source>:80:10: fatal error: 'nbnxn_ocl_kernel_pruneonly.clh' file
>> not found
>> #include "nbnxn_ocl_kernel_pruneonly.clh"
>>         ^
>> ---------------LOG END----------------
>> 
>> -------------------------------------------------------
>> Program:     mdrun-test, version 2018.5
>> Source file: src/gromacs/gpu_utils/ocl_compiler.cpp (line 507)
>> Function:    cl_program gmx::ocl::compileProgram(FILE *, const std::string
>> &, const std::string &, cl_context, cl_device_id, ocl_vendor_id_t)
>> 
>> Internal error (bug):
>> Failed to compile NBNXN kernels for GPU #AMD Radeon Pro 560 Compute Engine
>>  Could not build OpenCL program, error was CL_BUILD_PROGRAM_FAILURE
>>