[gmx-users] GXM 512 allocation issue

[Carlo Camilloni]

Dear GMX users,

the last week I began using a GPU cluster and I am now getting an error I have never seen before:

 85 Source code file: /homea/hmu24/hmu240/Codes/gromacs/src/gromacs/utility/smalloc.c, line: 227
 86 
 87 Fatal error:
 88 Not enough memory. Failed to realloc 323880 bytes for sort->sort2, sort->sort2=67733e20
 89 (called from file /homea/hmu24/hmu240/Codes/gromacs/src/gromacs/domdec/domdec.cpp, line 9215)
 90 For more information and tips for troubleshooting, please check the GROMACS

I am using GMX 5.1.2, these are the details of the architecture from the log file
(i am running the simulation using 4 MPI + 6 OpenMP threads using 4 GPUs) 
the errors is trigger randomly.

Is this also happening to others?

Precision:          single
   38 Memory model:       64 bit
   39 MPI library:        MPI
   40 OpenMP support:     enabled (GMX_OPENMP_MAX_THREADS = 32)
   41 GPU support:        enabled
   42 OpenCL support:     disabled
   43 invsqrt routine:    gmx_software_invsqrt(x)
   44 SIMD instructions:  AVX2_256
   45 FFT library:        fftw-3.3.4-sse2-avx
   46 RDTSCP usage:       enabled
   47 C++11 compilation:  disabled
   48 TNG support:        enabled
   49 Tracing support:    disabled
   50 Built on:           Tue 21 Jun 21:34:57 CEST 2016
   51 Built by:           hmu240 at jrl03 [CMAKE]
   52 Build OS/arch:      Linux 3.10.0-327.18.2.el7.x86_64 x86_64
   53 Build CPU vendor:   GenuineIntel
   54 Build CPU brand:    Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
   55 Build CPU family:   6   Model: 63   Stepping: 2
   56 Build CPU features: aes apic avx avx2 clfsh cmov cx8 cx16 f16c fma htt lahf_lm mmx msr nonstop_tsc pcid pclmuldq pdcm pdpe1gb popcnt pse rdrnd rdtscp sse2 sse3 sse4.1 sse4.2 ssse3 tdt x2apic
   57 C compiler:         /usr/bin/cc GNU 4.8.5
   58 C compiler flags:    -march=core-avx2    -Wextra -Wno-missing-field-initializers -Wno-sign-compare -Wpointer-arith -Wall -Wno-unused -Wunused-value -Wunused-parameter  -O3 -DNDEBUG -funroll-all-loop      s -fexcess-precision=fast  -Wno-array-bounds
   59 C++ compiler:       /usr/bin/c++ GNU 4.8.5
   60 C++ compiler flags:  -march=core-avx2    -Wextra -Wno-missing-field-initializers -Wpointer-arith -Wall -Wno-unused-function  -O3 -DNDEBUG -funroll-all-loops -fexcess-precision=fast  -Wno-array-bound      s
   61 Boost version:      1.55.0 (internal)
   62 CUDA compiler:      /usr/local/software/jureca/Stages/2016a/software/CUDA/7.5.18-iccifort-2015.3.187-GCC-4.9.3-2.25/bin/nvcc nvcc: NVIDIA (R) Cuda compiler driver;Copyright (c) 2005-2015 NVIDIA Corp      oration;Built on Tue_Aug_11_14:27:32_CDT_2015;Cuda compilation tools, release 7.5, V7.5.17
   63 CUDA compiler flags:-gencode;arch=compute_20,code=sm_20;-gencode;arch=compute_30,code=sm_30;-gencode;arch=compute_35,code=sm_35;-gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_5      0;-gencode;arch=compute_52,code=sm_52;-gencode;arch=compute_52,code=compute_52;-use_fast_math;; ;-march=core-avx2;-Wextra;-Wno-missing-field-initializers;-Wpointer-arith;-Wall;-Wno-unused-function;-      O3;-DNDEBUG;-funroll-all-loops;-fexcess-precision=fast;-Wno-array-bounds;
   64 CUDA driver:        7.50
   65 CUDA runtime:       7.50
   66 
   67 
   68 Number of logical cores detected (48) does not match the number reported by OpenMP (6).
   69 Consider setting the launch configuration manually!
   70 
   71 Running on 1 node with total 24 cores, 48 logical cores, 4 compatible GPUs
   72 Hardware detected on host jrc0489 (the node of MPI rank 0):
   73   CPU info:
   74     Vendor: GenuineIntel
   75     Brand:  Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
   76     Family:  6  model: 63  stepping:  2
   77     CPU features: aes apic avx avx2 clfsh cmov cx8 cx16 f16c fma htt lahf_lm mmx msr nonstop_tsc pcid pclmuldq pdcm pdpe1gb popcnt pse rdrnd rdtscp sse2 sse3 sse4.1 sse4.2 ssse3 tdt x2apic
   78     SIMD instructions most likely to fit this hardware: AVX2_256
   79     SIMD instructions selected at GROMACS compile time: AVX2_256
   80   GPU info:
   81     Number of GPUs detected: 4
   82     #0: NVIDIA Tesla K80, compute cap.: 3.7, ECC: yes, stat: compatible
   83     #1: NVIDIA Tesla K80, compute cap.: 3.7, ECC: yes, stat: compatible
   84     #2: NVIDIA Tesla K80, compute cap.: 3.7, ECC: yes, stat: compatible
   85     #3: NVIDIA Tesla K80, compute cap.: 3.7, ECC: yes, stat: compatible


Best regards,
Carlo