[gmx-developers] Compiling with GPU/mkl on MSVC 2019

[Mark Abraham]

Hi,

On Tue, 23 Jul 2019 at 20:47, Adriaan Riet <aar93 at case.edu> wrote:

>
>
> Adriaan Riet
> 801-602-0172
> adriaan.riet at case.edu
> Hello,
>
> I have been working on building gromacs with GMX_GPU=ON in MSVC 2019 (and
> cuda 10.1)  I've used the Ninja and Visual studio 2019 Win64 generators,
> and I'm using the x64 cl.exe as the host compiler, cxx compiler, etc.
>

Great. We haven't prioritized trying to make such a setup work, as it's not
commonly used for computation, and we don't have a machine with the MS
compiler and a NVIDA GPU. But perhaps we can get things in order with your
help :-)


> With GMX_GPU=OFF, I only needed to make sure there were no spaces in some
> of the cmake flags (I changed CUDA_TOOLKIT_ROOT_DIR, MKL_LIBRARIES,
> MKL_INCLUDE_DIR), and everything builds and runs fine.
>
> With GMX_GPU=ON, I've seen a couple of issues. First, and I've gotten
> around this by changing CUDA_HOST_COMPILER, the location ${VCInstallDir}bin
> no longer points to a valid compiler. MS has opted to change the location
> and provide separate x86 and x64 compilers and cross compilers. I've
> pointed the CUDA_HOST_COMPILER variable to the same compiler as the other
> CMAKE_C**_COMPILER variables,
> C:\Program Files (x86)\Microsoft Visual
> Studio\2019\Community\VC\Tools\MSVC\14.21.27702\bin\Hostx64\x64\cl.exe
> I'm not sure that this is the compiler I should be using, as GPU should be
> single precision, but I wanted to be consistent.
>

That looks fine.

I'm running into an issue in the CUDA code. Specifically, there are a few
> lines in nbnxm_cuda.cu that set up for launching the kernel (lines 834,
> 879, etc):
>
> const auto kernelArgs          = prepareGpuKernelArguments(kernelFn,
> config,
>                                                                &numColumns,
>                                                                &xqPtr,
>
>  &setFillerCoords,
>                                                                &d_x,
>
>  &d_atomIndices,
>                                                                &d_cxy_na,
>                                                                &d_cxy_ind,
>                                                                &cellOffset,
>
>  &numAtomsPerCell);
>     launchGpuKernel(kernelFn, config, nullptr, "XbufferOps", kernelArgs);
>
> the relevant template generating prepareGpuKernelArguments is (I think,
> from cudautils.cuh):
>
> template <typename ... Args>
> std::array<void *, sizeof ... (Args)> prepareGpuKernelArguments(void
>               (*kernel)(Args...),
>                                                                 const
> KernelLaunchConfig & /*config */,
>                                                                 const Args
> *...          argsPtrs)
> {
>     std::array<void *, sizeof ... (Args)> kernelArgs;
>     prepareGpuKernelArgument(kernel, &kernelArgs, 0, argsPtrs ...);
>     return kernelArgs;
> }
>
> the compiler throws a fit with this, stating that (I've changed the order
> to what I understand to be cause/effect):
>
> 1. "Error C2782 'std::array<void*,sizeof...(Args)>
> prepareGpuKernelArguments(void (__cdecl *)(Args...),const
> KernelLaunchConfig &,const Args *...)': template parameter 'Args' is
> ambiguous "
> 2. " 'prepareGpuKernelArguments': no matching overloaded function found
> 3. " 'kernelArgs': cannot be used before it is initialized
> 4. " 'kernelArgs': an object of const-qualified type must be initialized
> 5. " 'void launchGpuKernel<const float3*__restrict ,rvec(*),const
> int*__restrict ,int,int>(void (__cdecl *)(const float3 *__restrict ,rvec
> (*),const int *__restrict ,int,int),const KernelLaunchConfig &,CommandEvent
> *,const char *,const std::array<void *,5> &)': cannot convert argument 5
> from 'int' to 'const std::array<void *,5> &' nbnxm_cuda.cu 886
>
> Seems to me that VS isn't recognizing some feature here. Is this something
> that might be easy to fix?
>

Googling around, it seems that MSVC does struggle more with variadic
templates than the other major compilers. I experimented a bit with
Compiler Explorer, and I think the problem arises because the three kinds
of arguments to prepareGpuKernelArguments aren't clear enough to the
compiler. If so, then I think the proposed change at
https://gerrit.gromacs.org/c/gromacs/+/12183 will fix it. Other compilers
still seem to be happy, and it can't affect the quality of the generated
code (touch wood). If you're able to use the download link there, you can
get a tarball of source code that you can build in the same way you have
done with the official tarball so far, and let us know how you get on. (By
the way, 2019 has had several patch-level releases so far, which you'd
probably want to take advantage of...)

Mark


> Thanks,
> Adriaan Riet
> --
> Gromacs Developers mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers
> or send a mail to gmx-developers-request at gromacs.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://maillist.sys.kth.se/pipermail/gromacs.org_gmx-developers/attachments/20190723/20e622a7/attachment.html>