[gmx-users] mdrun_mpi in HP_MPI LSF/SLURM setup

[Mark Abraham]

On 4/18/2011 6:44 PM, Larcombe, Lee wrote:
> Ok. Solved it. Nothing wrong with the LSF/SLURM/MPI stuff. GMXRC.bash
> hadn't executed properly to set the environment up (typo) and the shared
> libraries weren’t being found. Seems this makes mdrun_mpi run like serial
> mdrun!

OK, but failing to pick up shared libraries would cause a fatal error to
that effect. I think there is another mdrun_mpi in the PATH that is not
actually MPI-configured.

Mark

> Oops.
>
>
> Lee
>
> On 16/04/2011 22:51, "Larcombe, Lee" <l.larcombe at cranfield.ac.uk> wrote:
>
>> Thanks Mark,
>>
>> HP-MPI is configured correctly on the system - an HP XC 3000 >800 cores.
>> It works for all the other users (none gromacs) and no I've tested it, I
>> can launch an mpi job which runs fine on the login node (two quad core
>> xeons)
>>
>> It seems to be an issue with the number of processors passed from the BSUB
>> settings in the LSF script via srun to gromacs. I just wondered whether
>> anyone else had 4.5.3 working in a similar setup?
>>
>> Most of the LSF/SLURM stuff I can google (not much) seems to be pre 4.5
>> where mdrun_mpi will take the depreciated command line switch to tell it
>> the number of processors
>>
>> Lee
>>
>> ==================================
>> Dr Lee Larcombe
>> - Lecturer in Genetics & Computational Biology
>> - London Technology Network Business Fellow
>> - Course Director of MSc Applied Bioinformatics
>>
>> Course website: http://bit.ly/cM6SkT
>> Group Research: http://bit.ly/bfqtyo
>>
>> Bioinformatics Group, Cranfield University
>> Bedfordshire MK43 0AL.   T: +44 (0)1234 758320
>> ==================================
>>
>>
>>
>>
>>
>>
>> On 16/04/2011 22:34, "Mark Abraham" <Mark.Abraham at anu.edu.au> wrote:
>>
>>> On 16/04/2011 12:13 AM, Larcombe, Lee wrote:
>>>> Hi gmx-users
>>>>
>>>> We have an HPC setup running HP_MPI and LSF/SLURM. Gromacs 4.5.3 has
>>>> been compiled with mpi support
>>>> The compute nodes on the system contain 2 x dual core Xeons which the
>>>> system sees as 4 processors
>>>>
>>>> An LSF script called gromacs_run.lsf is as shown below
>>>>
>>>> #BSUB -N
>>>> #BSUB -J "gromacsTest5"
>>>> #BSUB -u l.larcombe at cranfield.ac.uk
>>>> #BSUB -n 4
>>>> #BSUB -q short
>>>> #BSUB -o %J.log
>>>> mpirun -srun mdrun_mpi -v -s xxx.tpr -o xxx.trr
>>>>
>>>> Queued with:
>>>>
>>>> Bsub<  gromacs_run.lsf
>>>>
>>>> This is intended to run 1 mdrun on a single node using all four cores
>>>> of the two xeons. The result is that although the job is only submitted
>>>> to one compute node, 4 mdruns are launched on each of the 4 cores = 16
>>>> jobs. These are all the same as if mdrun has not been compiled with mpi
>>>> support.
>>> mdrun_mpi will run one process on each core that the MPI configuration
>>> declares is available. Spawning four separate runs of four processes
>>> indicates that MPI is not configured to reflect the hardware (since each
>>> run thinks it can have four processes), or that the submission script is
>>> inappropriate (since four runs get spawned), or both. We can't help
>>> there. Trouble-shoot with a trivial MPI test program.
>>>
>>>> If I tell srun to start just one task with "mpirun -srun -n1 mdrun_mpi
>>>> -v -s xxx.tpr ­o xxx.trr" it starts one job on each core instead of 4:
>>>>
>>>> NNODES=1, MYRANK=0, HOSTNAME=comp195
>>>> NNODES=1, MYRANK=0, HOSTNAME=comp195
>>>> NNODES=1, MYRANK=0, HOSTNAME=comp195
>>>> NNODES=1, MYRANK=0, HOSTNAME=comp195
>>>>
>>>> Logs show 4 mdrun_mpi starts, 4 file read ins and I get 4 of all run
>>>> files in CWD. I am sure
>>>
>>>
>>>> that mdrun_mpi is indeed compiled with mpi support - although our
>>>> sysadmin did that, not me. For example, if I try and execute "mdrun_mpi
>>>> ­h" I get a message from HP­MPI and have to execute "mpirun mdrun_mpi
>>>> ­h" to see the help text.
>>>>
>>>> Does anyone have any experience of running with this setup  - any
>>>> ideas?
>>>>
>>>> Thanks
>>>> Lee
>>> -- 
>>> gmx-users mailing list    gmx-users at gromacs.org
>>> http://lists.gromacs.org/mailman/listinfo/gmx-users
>>> Please search the archive at
>>> http://www.gromacs.org/Support/Mailing_Lists/Search before posting!
>>> Please don't post (un)subscribe requests to the list. Use the
>>> www interface or send it to gmx-users-request at gromacs.org.
>>> Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>> -- 
>> gmx-users mailing list    gmx-users at gromacs.org
>> http://lists.gromacs.org/mailman/listinfo/gmx-users
>> Please search the archive at
>> http://www.gromacs.org/Support/Mailing_Lists/Search before posting!
>> Please don't post (un)subscribe requests to the list. Use the
>> www interface or send it to gmx-users-request at gromacs.org.
>> Can't post? Read http://www.gromacs.org/Support/Mailing_Lists