[gmx-users] MPI GPU job failed

[Justin Lemkul]

On 8/11/16 9:08 AM, Albert wrote:
> Hi, I used your suggested command line, but it failed with the following messages:
>
>
> -------------------------------------------------------
> Program gmx mdrun, VERSION 5.1.3
> Source code file:
> /home/albert/Downloads/gromacs/gromacs-5.1.3/src/gromacs/gmxlib/gmx_detect_hardware.cpp,
> line: 458
>
> Fatal error:
> Incorrect launch configuration: mismatching number of PP MPI processes and GPUs
> per node.

So you're trying to run on two nodes, each of which has one GPU?  I haven't done 
such a run, but perhaps mpirun -np 2 gmx_mpi mdrun -v -s 62.tpr -gpu_id 0 would 
do the trick, by finding the first GPU on each node?

-Justin

> gmx_mpi was started with 1 PP MPI process per node, but you provided 2 GPUs.
> For more information and tips for troubleshooting, please check the GROMACS
> website at http://www.gromacs.org/Documentation/Errors
> -------------------------------------------------------
>
> Halting program gmx mdrun
> --------------------------------------------------------------------------
> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
> with errorcode 1.
>
> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
> You may or may not see output from other processes, depending on
> exactly when Open MPI kills them.
> --------------------------------------------------------------------------
> Using 1 MPI process
> Using 20 OpenMP threads
>
> 2 GPUs user-selected for this run.
> Mapping of GPU IDs to the 1 PP rank in this node: 0,1
>
>
> -------------------------------------------------------
> Program gmx mdrun, VERSION 5.1.3
> Source code file:
> /home/albert/Downloads/gromacs/gromacs-5.1.3/src/gromacs/gmxlib/gmx_detect_hardware.cpp,
> line: 458
>
> Fatal error:
> Incorrect launch configuration: mismatching number of PP MPI processes and GPUs
> per node.
> gmx_mpi was started with 1 PP MPI process per node, but you provided 2 GPUs.
> For more information and tips for troubleshooting, please check the GROMACS
> website at http://www.gromacs.org/Documentation/Errors
> -------------------------------------------------------
>
>
>
>
> On 08/11/2016 01:18 PM, jkrieger at mrc-lmb.cam.ac.uk wrote:
>> The problem is you compiled gromacs with mpi (hence the default _mpi in
>> your command). You therefore need to set the number of mpi processes
>> rather than threads. The appropriate command would instead be the
>> following:
>>
>> mpirun -np 2 gmx_mpi mdrun -v -s 62.tpr -gpu_id 01
>>
>> Alternatively you could compile a different gromacs version without mpi.
>> This should have thread-mpi and openmp by default if you leave out
>> DGMX_MPI=ON from the cmake command.
>>
>> Best wishes
>> James
>

-- 
==================================================

Justin A. Lemkul, Ph.D.
Ruth L. Kirschstein NRSA Postdoctoral Fellow

Department of Pharmaceutical Sciences
School of Pharmacy
Health Sciences Facility II, Room 629
University of Maryland, Baltimore
20 Penn St.
Baltimore, MD 21201

jalemkul at outerbanks.umaryland.edu | (410) 706-7441
http://mackerell.umaryland.edu/~jalemkul

==================================================