[gmx-users] Works well on CPU, not on GPU

[Alex]

Dear all,
A simulation of mine works well when I don't use the GPU for 20 ns after
three rounds of minimization and equilibrations;

Works:
aprun -n 176 gmx_mpi mdrun -deffnm an -s c.tpr -g c.log -v -dlb yes -rdd
1.5 -dds 0.9999 -gcom 1 -npme 44 -ntomp_pme 1 -ntomp 6 -tunepme yes -pin on

But when I continue the simulation having the GPU involved (below command),

Not working
aprun -n 88 gmx_mpi mdrun -deffnm an -s c.tpr -g c.log -v -dlb yes -rdd 1.6
-dds 0.9999 -gcom 1 -nb gpu -npme 40 -ntomp_pme 1 -ntomp 6 -tunepme yes -pin
on -cpi an.cpt -append

The simulation crashes after around 0.25 ns with an error that:

Not all bonded interactions have been properly assigned to the domain
decomposition cells

A list of missing interactions:
           G96Angle of 279715 missing      1

Molecule type 'group1'
the first 10 missing interactions, except for exclusions:
           G96Angle atoms 5248 5258 5250      global  5248  5258  5250

-------------------------------------------------------
Program:     gmx mdrun, version 2018.1
Source file: src/gromacs/domdec/domdec_topology.cpp (line 436)
MPI rank:    0 (out of 88)

Fatal error:
1 of the 489664 bonded interactions could not be calculated because some
atoms involved moved further apart than the multi-body cut-off distance
(1.6 nm) or the two-body cut-off distance (1.6 nm), see option -rdd, for
pairs and tabulated bonds also see option -ddcheck

As recommended, I played around with the -rdd, but the problem still
survives.
Any commend would be highly appreciated.

Below is just some information about the machine which I am using;

GROMACS version:    2018.1
Precision:          single
Memory model:       64 bit
MPI library:        MPI
OpenMP support:     enabled (GMX_OPENMP_MAX_THREADS = 64)
GPU support:        CUDA
SIMD instructions:  AVX2_256
FFT library:        commercial-fftw-3.3.6-pl1-fma-sse2-avx-avx2-avx2_128
RDTSCP usage:       enabled
TNG support:        enabled
Hwloc support:      hwloc-1.11.0
Tracing support:    disabled
Built on:           2018-09-12 20:34:33
Built by:           cots at onyx02 [CMAKE]
Build OS/arch:      Linux 3.12.61-52.111-default x86_64
Build CPU vendor:   Intel
Build CPU brand:    Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz
Build CPU family:   6   Model: 79   Stepping: 1
Build CPU features: aes apic avx avx2 clfsh cmov cx8 cx16 f16c fma hle htt
intel lahf mmx msr nonstop_tsc pcid pclmuldq pdcm pdpe1gb popcnt pse rdrnd
rdtscp rtm sse2 sse3 sse4.1 sse4.2 ssse3 tdt x2apic
C compiler:         /opt/cray/pe/craype/2.5.13/bin/cc GNU 5.3.0
C compiler flags:    -march=core-avx2     -O3 -DNDEBUG -funroll-all-loops
-fexcess-precision=fast
C++ compiler:       /opt/cray/pe/craype/2.5.13/bin/CC GNU 5.3.0
C++ compiler flags:  -march=core-avx2    -std=c++11   -O3 -DNDEBUG
-funroll-all-loops -fexcess-precision=fast
CUDA compiler:
     /opt/nvidia/cudatoolkit8.0/8.0.61_2.3.13_g32c34f9-2.1/bin/nvcc nvcc:
NVIDIA (R) Cuda compiler driver;Copyright (c) 2005-2016 NVIDIA
Corporation;Built on Tue_Jan_10_13:22:03_CST_2017;Cuda compilation tools,
release 8.0
, V8.0.61
CUDA compiler
flags:-gencode;arch=compute_60,code=sm_60;-use_fast_math;-Wno-deprecated-gpu-targets;;;
;-march=core-avx2;-std=c++11;-O3;-DNDEBUG;-funroll-all-loops;-fexcess-precision=fast;

CUDA driver:        9.20
CUDA runtime:       8.0

Thank you
Alex