[gmx-developers] Fatal error: cudaStreamSynchronize failed in cu_blockwait_nb

[AOWI (Anders Ossowicki)]

Hello,

We are testing out Gromacs 4.6.5 with an Nvidia K20 card. We keep running into the error message below, no matter which setup we're trying. In the included case, it was the RNAse example from http://www.gromacs.org/GPU_acceleration. Furthermore, we get the following line in dmesg as well:

   NVRM: GPU at 0000:42:00: GPU-d0b07804-027a-5a02-43bc-fd7dc9064637
   NVRM: Xid (0000:42:00): 31, Ch 00000003, engmask 00000101, intr 10000000

Are we just completely out of luck with this card, or have we done something wrong?

We've built Gromacs from source against the cuda 5.5 libraries straight from Nvidia. The system is Ubuntu 12.04. Gromacs works fine when it's not using the GPU.
The card identifies itself as NVIDIA Corporation GK110GL [Tesla K20m] (rev a1)

This is what we've done to trigger the error:
$ grompp -f pme_verlet.mdp -c conf.gro -p topol.top
$ mdrun

Here is the output from mdrun. The error message tells me absolutely nothing, so any advice on how to proceed with debugging this would be much appreciated.

Reading file topol.tpr, VERSION 4.6.5 (single precision)
Changing nstlist from 10 to 40, rlist from 0.9 to 0.996

Using 1 MPI thread
Using 48 OpenMP threads

1 GPU detected:
  #0: NVIDIA Tesla K20m, compute cap.: 3.5, ECC: yes, stat: compatible

1 GPU auto-selected for this run.
Mapping of GPU to the 1 PP rank in this node: #0


Back Off! I just backed up ener.edr to ./#ener.edr.1#
starting mdrun 'RNASE ZF-1A in water'
10000 steps,     20.0 ps.

-------------------------------------------------------
Program mdrun, VERSION 4.6.5
Source code file: /home/nztest/src/gromacs-4.6.5/src/mdlib/nbnxn_cuda/nbnxn_cuda.cu, line: 591

Fatal error:
cudaStreamSynchronize failed in cu_blockwait_nb: unspecified launch failure

For more information and tips for troubleshooting, please check the GROMACS
website at http://www.gromacs.org/Documentation/Errors
-------------------------------------------------------

Thanks in advance!

-- 
Best Regards
Anders Ossowicki