[gmx-users] GPU job failed

[Albert]

Here are more informations from log file:

mpirun -np 2 mdrun_mpi -v -s npt2.tpr -c npt2.gro -x npt2.xtc -g
npt2.log -gpu_id 01 -ntomp 0


Number of hardware threads detected (20) does not match the number
reported by OpenMP (10).
Consider setting the launch configuration manually!

Number of hardware threads detected (20) does not match the number
reported by OpenMP (10).
Consider setting the launch configuration manually!
Reading file npt2.tpr, VERSION 5.0.1 (single precision)
Reading file npt2.tpr, VERSION 5.0.1 (single precision)
Using 1 MPI process
Using 10 OpenMP threads

2 GPUs detected on host cudaB:
#0: NVIDIA GeForce GTX 780 Ti, compute cap.: 3.5, ECC: no, stat: compatible
#1: NVIDIA GeForce GTX 780 Ti, compute cap.: 3.5, ECC: no, stat: compatible

2 GPUs user-selected for this run.
Mapping of GPUs to the 1 PP rank in this node: #0, #1


-------------------------------------------------------
Program mdrun_mpi, VERSION 5.0.1
Source code file:
/soft2/plumed-2.2/gromacs-5.0.1/src/gromacs/gmxlib/gmx_detect_hardware.c, line:
359

Fatal error:
Incorrect launch configuration: mismatching number of PP MPI processes
and GPUs per node.
mdrun_mpi was started with 1 PP MPI process per node, but you provided 2
GPUs.
For more information and tips for troubleshooting, please check the GROMACS
website at http://www.gromacs.org/Documentation/Errors
-------------------------------------------------------

Halting program mdrun_mpi

gcq#314: "Do You Have Sex Maniacs or Schizophrenics or Astrophysicists
in Your Family?" (Gogol Bordello)

--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode -1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
Using 1 MPI process
Using 10 OpenMP threads

2 GPUs detected on host cudaB:
#0: NVIDIA GeForce GTX 780 Ti, compute cap.: 3.5, ECC: no, stat: compatible
#1: NVIDIA GeForce GTX 780 Ti, compute cap.: 3.5, ECC: no, stat: compatible

2 GPUs user-selected for this run.
Mapping of GPUs to the 1 PP rank in this node: #0, #1


-------------------------------------------------------
Program mdrun_mpi, VERSION 5.0.1
Source code file:
/soft2/plumed-2.2/gromacs-5.0.1/src/gromacs/gmxlib/gmx_detect_hardware.c, line:
359

Fatal error:
Incorrect launch configuration: mismatching number of PP MPI processes
and GPUs per node.
mdrun_mpi was started with 1 PP MPI process per node, but you provided 2
GPUs.
For more information and tips for troubleshooting, please check the GROMACS
website at http://www.gromacs.org/Documentation/Errors
-------------------------------------------------------

Halting program mdrun_mpi

gcq#56: "Lunatics On Pogo Sticks" (Red Hot Chili Peppers)

--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode -1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------









On 09/08/2014 11:59 PM, Yunlong Liu wrote:
> Same idea with Szilard.
>
> How many nodes are you using?
> On one nodes, how many MPI ranks do you have? The error is complaining about you assigned two GPUs to only one MPI process on one node. If you spread your two MPI ranks on two nodes, that means you only have one at each. Then you can't assign two GPU for only one MPI rank.
>
> How many GPU do you have on one node? If there are two, you can either launch two PPMPI processes on one node and assign two GPU for them. If you only want to launch one MPI rank on each node, you can assign only one GPU for each node ( by -gpu_id 0 )
>
> Yunlong