[gmx-users] GPU job frequently failed

[Albert]

Hello:

I noticed that my GPU job under 5.1.2 often failed with messages:

step 52400, will finish Mon Jul 25 12:04:59 2016
-------------------------------------------------------
Program gmx mdrun, VERSION 5.1.2
Source code file: 
/home/albert/Downloads/gromacs/gromacs-5.1.2/src/gromacs/mdlib/nbnxn_cuda/nbnxn_cuda.cu, 
line: 688

Fatal error:
cudaStreamSynchronize failed in cu_blockwait_nb: unknown error

For more information and tips for troubleshooting, please check the GROMACS
website at http://www.gromacs.org/Documentation/Errors
-------------------------------------------------------

Halting program gmx mdrun
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------


Does anybody has any idea?

thx a lot