[gmx-users] mdrun initialises, fails to run, no error message

[Justin Lemkul]

On 1/3/17 10:43 AM, Natalie Tatum wrote:
> Dear all,
>
> I'm hoping you can shed light on (a) what my mdrun problem is and (b) where
> to start fixing it.
>
> I'm simulating different mutants of a protein dimer on DNA, for 10 ns
> a-piece. I have successfully run this protocol on the wild-type protein, on
> two single residue mutants, and on a double mutant. I came to run the same
> on a fourth, single site mutant. I have followed the same protocols and
> utilised the same MDP settings throughout. All were subject to 5000 steps
> of steepest-descent energy minimisation, then 200 ps of equilibration in
> the NVT ensemble, then the same in the NPT. For this particular mutant
> there were no issues apparent going into production MD. Therefore, I don't
> think it's an issue of my MDP setup or system...
>
> So I have two compatible (OpenCL 1.2) AMD Radeon HD Firepro D300 GPUs, and
> I have one mutant (run/process) assigned to each.
>
> For this mutant I call mdrun with:
>
> gmx mdrun -deffnm md -gpu_id 1 &
>
> Whereas the other is on -gpu_id 0, and walk away. This worked successfully
> in the week prior for two other systems. It's New Year, then I come back to
> what should be completed simulations this morning to get my hands dirty in
> analysis.
>
> Run on gpu 0 has completed successfully, all is grand.
>
> Mutant on gpu 1 has not. Attempts to resume/restart fail (on either GPU, or
> both, or calling neither explicitly). All output looks like this:
>
> GROMACS:      gmx mdrun, VERSION 5.1.3
>
> Executable:   /usr/local/gromacs/bin/gmx
>
> Data prefix:  /usr/local/gromacs
>
> Command line:
>
>
>
>   gmx mdrun -deffnm md
>

 From the .log, it appears your command was not what you think it was.  Is it 
possible that the job failed because mdrun tried to consume all available 
hardware and got hung up?

>
>
> GROMACS version:    VERSION 5.1.3
>
> Precision:          single
>
> Memory model:       64 bit
>
> MPI library:        thread_mpi
>
> OpenMP support:     disabled
>
> GPU support:        enabled
>
> OpenCL support:     enabled
>
> invsqrt routine:    gmx_software_invsqrt(x)
>
> SIMD instructions:  AVX_256
>
> FFT library:        fftw-3.3.4-sse2
>
> RDTSCP usage:       enabled
>
> C++11 compilation:  disabled
>
> TNG support:        enabled
>
> Tracing support:    disabled
>
> Built on:           Mon  1 Aug 2016 17:20:18 BST
>
> Built by:           natalie at t <natalie at nicr00353.ncl.ac.uk>
> hemachineIuse.here.there [CMAKE]
>
> Build OS/arch:      Darwin 15.5.0 x86_64
>
> Build CPU vendor:   GenuineIntel
>
> Build CPU brand:    Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz
>
> Build CPU family:   6   Model: 62   Stepping: 4
>
> Build CPU features: aes apic avx clfsh cmov cx8 cx16 f16c htt lahf_lm mmx
> msr nonstop_tsc pcid pclmuldq pdcm pdpe1gb popcnt pse rdrnd rdtscp sse2
> sse3 sse4.1 sse4.2 ssse3 tdt x2apic
>
> C compiler:         /Applications/Xcode.app/Contents/Developer/Toolchains/
> XcodeDefault.xctoolchain/usr/bin/cc Clang 7.3.0.7030031
>
> C compiler flags:    -mavx    -Wall -Wno-unused -Wunused-value
> -Wunused-parameter -Wno-unknown-pragmas  -O3 -DNDEBUG
>
> C++ compiler:       /Applications/Xcode.app/Contents/Developer/Toolchains/
> XcodeDefault.xctoolchain/usr/bin/c++ Clang 7.3.0.7030031
>
> C++ compiler flags:  -mavx    -Wextra -Wno-missing-field-initializers
> -Wpointer-arith -Wall -Wno-unused-function -Wno-unknown-pragmas  -O3
> -DNDEBUG
>
> Boost version:      1.60.0 (external)
>
> OpenCL include dir: /System/Library/Frameworks/OpenCL.framework
>
> OpenCL library:     /System/Library/Frameworks/OPENCL.framework
>
> OpenCL version:     1.2
>
>
> And there it ends. No files except the log shown above - and though this
> initial output looks identical in content to the beginnings of logs for
> successful simulations, mdrun does not then seem to engage with the
> GPU/CPUs available.
>
> There are no error messages, no apparent indication as to where this has
> gone wrong... And now I can't run mdrun at all, for any system.
>

Test whether or not your GPU is still accessible and capable of running test 
programs.

-Justin

> I've checked my disk space (fine, >100 GB available), I'm able to call and
> execute other gmx commands, but mdrun does the above.
>
> The closest error I can find with my google-fu is three years ago where
> this user (
> http://gromacs.org_gmx-users.maillist.sys.kth.narkive.com/FEdWd6gC/mdrun-no-error-but-hangs-no-results
> ) got no error but a killed process, but I don't even get as far as
> detection of CPUs/GPUs or domain decomposition.
>
> Any suggestions much appreciated,
>
> Natalie
>

-- 
==================================================

Justin A. Lemkul, Ph.D.
Ruth L. Kirschstein NRSA Postdoctoral Fellow

Department of Pharmaceutical Sciences
School of Pharmacy
Health Sciences Facility II, Room 629
University of Maryland, Baltimore
20 Penn St.
Baltimore, MD 21201

jalemkul at outerbanks.umaryland.edu | (410) 706-7441
http://mackerell.umaryland.edu/~jalemkul

==================================================