[gmx-users] Unexpected cudaStreamQuery failure: unspecified launch failure

[Mariem Ghoula]

Hi,

I had this error while I was performing a short (1 ns) simulation. It seems
to be related to the Nvidia driver. Can anyone please help me understand
why I'm having this error?

My system is a Ubuntu 18.04.4 LTS with cuda-toolkit version 9.1.85, nvidia
driver version 410.78.

I had this error:

Command line:
  gmx mdrun -v -deffnm nvt

Reading file nvt.tpr, VERSION 2019.5 (single precision)
Changing nstlist from 20 to 100, rlist from 1.219 to 1.338

Using 1 MPI thread
Using 20 OpenMP threads

1 GPU selected for this run.
Mapping of GPU IDs to the 2 GPU tasks in the 1 rank on this node:
  PP:0,PME:0
PP tasks will do (non-perturbed) short-ranged interactions on the GPU
PME tasks will do all aspects on the GPU
starting mdrun 'Protein in water'
500000 steps,   1000.0 ps.
step  200: timed with pme grid 72 72 72, coulomb cutoff 1.200: 14224.2
M-cycles
step  400: timed with pme grid 60 60 60, coulomb cutoff 1.369: 19270.5
M-cycles
step  600: timed with pme grid 64 64 64, coulomb cutoff 1.283: 16957.3
M-cycles
step  800: timed with pme grid 72 72 72, coulomb cutoff 1.200: 14246.1
M-cycles
              optimal pme grid 72 72 72, coulomb cutoff 1.200
step 75400, will finish Mon Feb 17 22:17:08 2020
-------------------------------------------------------
Program:     gmx mdrun, version 2019.5
Source file: src/gromacs/gpu_utils/cudautils.cuh (line 251)

Fatal error:
Unexpected cudaStreamQuery failure: unspecified launch failure

For more information and tips for troubleshooting, please check the GROMACS
website at http://www.gromacs.org/Documentation/Errors
-------------------------------------------------------

I opened the file cudautils.cuh (line 251) to see what kind of error it can
be but I don't get it.

/*! \brief  Returns true if all tasks in \p s have completed.
 *
 * \param[in] s stream to check
 *
 *  \returns     True if all tasks enqueued in the stream \p s (at the time
of this call) have completed.
 */
static inline bool haveStreamTasksCompleted(cudaStream_t s)
{
    cudaError_t stat = cudaStreamQuery(s);

    if (stat == cudaErrorNotReady)
    {
        // work is still in progress in the stream
        return false;
    }

    GMX_ASSERT(stat !=  cudaErrorInvalidResourceHandle, "Stream idnetifier
not valid");

    // cudaSuccess and cudaErrorNotReady are the expected return values
    CU_RET_ERR(stat, "Unexpected cudaStreamQuery failure");

    GMX_ASSERT(stat == cudaSuccess, "Values other than cudaSuccess should
have been explicitly handled");

    return true;
}


my mdp file is the following and I'm using the charmm36m ff:


title                   = IDE NVT equilibration
define                  = -DPOSRES  ; position restrain the protein
; Run parameters
integrator              = md        ; leap-frog integrator
nsteps                  = 500000     ; 2 * 500000 = 1000 ps (1ns)
dt                      = 0.002     ; 2 fs
; Output control
nstenergy               = 500   ; save energies every 1.0 ps
nstlog                  = 500   ; update log file every 1.0 ps
nstxout-compressed      = 500   ; save coordinates every 1.0 ps
; Bond parameters
continuation            = no        ; first dynamics run
constraint_algorithm    = lincs     ; holonomic constraints
constraints             = h-bonds   ; bonds to H are constrained
lincs_iter              = 1         ; accuracy of LINCS
lincs_order             = 4         ; also related to accuracy
; Neighbor searching and vdW
cutoff-scheme           = Verlet
ns_type                 = grid      ; search neighboring grid cells
nstlist                 = 20        ; largely irrelevant with Verlet
rlist                   = 1.2
vdwtype                 = cutoff
vdw-modifier            = force-switch
rvdw-switch             = 1.0
rvdw                    = 1.2       ; short-range van der Waals cutoff (in
nm)
; Electrostatics
coulombtype             = PME       ; Particle Mesh Ewald for long-range
electrostatics
rcoulomb                = 1.2       ; short-range electrostatic cutoff (in
nm)
pme_order               = 4         ; cubic interpolation
fourierspacing          = 0.16      ; grid spacing for FFT
; Temperature coupling
tcoupl                  = V-rescale                     ; modified
Berendsen thermostat
tc-grps                 = Protein non-Protein    ; two coupling groups -
more accurate
tau_t                   = 0.1   0.1                     ; time constant, in
ps
ref_t                   = 300   300                     ; reference
temperature, one for each group, in K
; Pressure coupling
pcoupl                  = no        ; no pressure coupling in NVT
; Periodic boundary conditions
pbc                     = xyz       ; 3-D PBC
; Dispersion correction is not used for proteins with the C36 additive FF
DispCorr                = no
; Velocity generation
gen_vel                 = yes       ; assign velocities from Maxwell
distribution
gen_temp                = 300       ; temperature for Maxwell distribution
gen_seed                = -1        ; generate a random seed



Thank you in advance for your help,

- Myriam