[gmx-users] Problems with simulation on multi-nodes cluster

James Starlight jmsstarlight at gmail.com
Tue Mar 20 12:35:14 CET 2012


Could someone tell me what tell the below error

Getting Loaded...
Reading file MD_100.tpr, VERSION 4.5.4 (single precision)
Loaded with Money


Will use 30 particle-particle and 18 PME only nodes
This is a guess, check the performance at the end of the log file
[ib02:22825] *** Process received signal ***
[ib02:22825] Signal: Segmentation fault (11)
[ib02:22825] Signal code: Address not mapped (1)
[ib02:22825] Failing at address: 0x10
[ib02:22825] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0xf030)
[0x7f535903e03$
[ib02:22825] [ 1] /usr/lib/openmpi/lib/openmpi/mca_pml_ob1.so(+0x7e23)
[0x7f535$
[ib02:22825] [ 2] /usr/lib/openmpi/lib/openmpi/mca_pml_ob1.so(+0x8601)
[0x7f535$
[ib02:22825] [ 3] /usr/lib/openmpi/lib/openmpi/mca_pml_ob1.so(+0x8bab)
[0x7f535$
[ib02:22825] [ 4] /usr/lib/openmpi/lib/openmpi/mca_btl_sm.so(+0x42af)
[0x7f5353$
[ib02:22825] [ 5] /usr/lib/libopen-pal.so.0(opal_progress+0x5b)
[0x7f535790506b]
[ib02:22825] [ 6] /usr/lib/libmpi.so.0(+0x37755) [0x7f5359282755]
[ib02:22825] [ 7] /usr/lib/openmpi/lib/openmpi/mca_coll_tuned.so(+0x1c3a)
[0x7f$
[ib02:22825] [ 8] /usr/lib/openmpi/lib/openmpi/mca_coll_tuned.so(+0x7fae)
[0x7f$
[ib02:22825] [ 9] /usr/lib/libmpi.so.0(ompi_comm_split+0xbf)
[0x7f535926de8f]
[ib02:22825] [10] /usr/lib/libmpi.so.0(MPI_Comm_split+0xdb) [0x7f535929dc2b]
[ib02:22825] [11]
/usr/lib/libgmx_mpi_d.openmpi.so.6(gmx_setup_nodecomm+0x19b) $
[ib02:22825] [12] mdrun_mpi_d.openmpi(mdrunner+0x46a) [0x40be7a]
[ib02:22825] [13] mdrun_mpi_d.openmpi(main+0x1256) [0x407206]
[ib02:22825] [14] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd)
[0x7f$
[ib02:22825] [15] mdrun_mpi_d.openmpi() [0x407479]
[ib02:22825] *** End of error message ***
--------------------------------------------------------------------------
mpiexec noticed that process rank 36 with PID 22825 on node ib02 exited on
sign$
--------------------------------------------------------------------------


I've obtained it when I've tried to use my system on multi-node station (
there is no problem on single node). Does this problem with the cluster
system or something wrong with parameters of my simulation?


JAmes

15 марта 2012 г. 15:25 пользователь James Starlight
<jmsstarlight at gmail.com>написал:

> Mark, Peter,
>
>
> I've tried to do .tpr file on my local CPU and launch only
>
> mpiexec -np 24 mdrun_mpi_d.openmpi -v -deffnm MD_100
>
> on the cluster with 2 nodes.
>
> I see my job as working but when I've checking the MD_100.log (attached)
> file there are no any information about simulation steps in that file (
> when I use just one node I see in that file step-by-step progression of my
> simulation like below wich was find in the same log file for ONE NODE
> simulation ):
>
> Started mdrun on node 0 Thu Mar 15 11:22:35 2012
>
>            Step           Time         Lambda
>               0        0.00000        0.00000
>
> Grid: 12 x 9 x 12 cells
>    Energies (kJ/mol)
>        G96Angle    Proper Dih.  Improper Dih.          LJ-14     Coulomb-14
>     1.32179e+04    3.27485e+03    2.53267e+03    4.06443e+02    6.15315e+04
>         LJ (SR)        LJ (LR)  Disper. corr.   Coulomb (SR)   Coul. recip.
>     4.12152e+04   -5.51788e+03   -1.70930e+03   -4.54886e+05   -1.46292e+05
>      Dis. Rest. D.R.Viol. (nm)     Dih. Rest.      Potential    Kinetic En.
>     2.14240e-02    3.46794e+00    1.33793e+03   -4.84889e+05    9.88771e+04
>    Total Energy  Conserved En.    Temperature Pres. DC (bar) Pressure (bar)
>    -3.86012e+05   -3.86012e+05    3.11520e+02   -1.14114e+02    3.67861e+02
>    Constr. rmsd
>     3.75854e-05
>
>            Step           Time         Lambda
>            2000        4.00000        0.00000
>
>    Energies (kJ/mol)
>        G96Angle    Proper Dih.  Improper Dih.          LJ-14     Coulomb-14
>     1.31741e+04    3.25280e+03    2.58442e+03    3.51371e+02    6.15913e+04
>         LJ (SR)        LJ (LR)  Disper. corr.   Coulomb (SR)   Coul. recip.
>     4.16349e+04   -5.53474e+03   -1.70930e+03   -4.56561e+05   -1.46485e+05
>      Dis. Rest. D.R.Viol. (nm)     Dih. Rest.      Potential    Kinetic En.
>     4.78276e+01    3.38844e+00    9.82735e+00   -4.87644e+05    9.83280e+04
>    Total Energy  Conserved En.    Temperature Pres. DC (bar) Pressure (bar)
>    -3.89316e+05   -3.87063e+05    3.09790e+02   -1.14114e+02    7.25905e+02
>    Constr. rmsd
>     1.88008e-05
>
> end etc...
>
>
>
> What's wrong can be with multi-node computations?
>
>
> James
>
>
> 15 марта 2012 г. 11:25 пользователь Mark Abraham <Mark.Abraham at anu.edu.au>написал:
>
> On 15/03/2012 6:13 PM, Peter C. Lai wrote:
>>
>>> Try separating your grompp run from your mpirun:
>>> You should not really be having the scheduler execute the grompp. Run
>>> your grompp step to generate a .tpr either on the head node or on your
>>> local
>>> machine (then copy it over to the cluster).
>>>
>>
>> Good advice.
>>
>>
>>> (The -p that the scheduler is complaining about only appears in the
>>> grompp
>>> step, so don't have the scheduler run it).
>>>
>>
>> grompp is running successfully, as you can see from the output
>>
>> I think "mpiexec -np 12" is being interpreted as "mpiexec -n 12 -p", and
>> the process of separating the grompp stage from the mdrun stage would help
>> make that clear - read documentation first, however.
>>
>> Mark
>>
>>
>>
>>>
>>> On 2012-03-15 10:04:49AM +0300, James Starlight wrote:
>>>
>>>> Dear Gromacs Users!
>>>>
>>>>
>>>> I have some problems with running my simulation on multi-modes station
>>>> wich
>>>> use open_MPI
>>>>
>>>> I've launch my jobs by means of that script. The below example of
>>>> running
>>>> work on 1 node ( 12 cpu).
>>>>
>>>> #!/bin/sh
>>>> #PBS -N gromacs
>>>> #PBS -l nodes=1:red:ppn=12
>>>> #PBS -V
>>>> #PBS -o gromacs.out
>>>> #PBS -e gromacs.err
>>>>
>>>> cd /globaltmp/xz/job_name
>>>> grompp -f md.mdp -c nvtWprotonated.gro -p topol.top -n index.ndx -o
>>>> job.tpr
>>>> mpiexec -np 12 mdrun_mpi_d.openmpi -v -deffnm job
>>>>
>>>> All nodes of my cluster consist of 12 CPU. When I'm using just 1 node on
>>>> that cluster I have no problems with running of my jobs but when I try
>>>> to
>>>> use more than one nodes I've obtain error ( the example is attached in
>>>> the
>>>> gromacs.err file as well as mmd.mdp of that system). Another outcome of
>>>> such multi-node simulation is that my job has been started but no
>>>> calculation were done ( the name_of_my_job.log file was empty and no
>>>> update
>>>> of .trr file was seen ). Commonly this error occurs when I uses many
>>>> nodes
>>>> (8-10) Finally sometimes I've obtain some errors with the PME order (
>>>> this
>>>> time I've used 3 nodes). The exactly error differs when I varry the
>>>> number
>>>> of nodes.
>>>>
>>>>
>>>> Could you tell me whats wrong could be with my cluster?
>>>>
>>>> Thanks for help
>>>>
>>>> James
>>>>
>>>
>>>
>>>  --
>>>> gmx-users mailing list    gmx-users at gromacs.org
>>>> http://lists.gromacs.org/**mailman/listinfo/gmx-users<http://lists.gromacs.org/mailman/listinfo/gmx-users>
>>>> Please search the archive at http://www.gromacs.org/**
>>>> Support/Mailing_Lists/Search<http://www.gromacs.org/Support/Mailing_Lists/Search>before posting!
>>>> Please don't post (un)subscribe requests to the list. Use the
>>>> www interface or send it to gmx-users-request at gromacs.org.
>>>> Can't post? Read http://www.gromacs.org/**Support/Mailing_Lists<http://www.gromacs.org/Support/Mailing_Lists>
>>>>
>>>
>>>
>> --
>> gmx-users mailing list    gmx-users at gromacs.org
>> http://lists.gromacs.org/**mailman/listinfo/gmx-users<http://lists.gromacs.org/mailman/listinfo/gmx-users>
>> Please search the archive at http://www.gromacs.org/**
>> Support/Mailing_Lists/Search<http://www.gromacs.org/Support/Mailing_Lists/Search>before posting!
>> Please don't post (un)subscribe requests to the list. Use the www
>> interface or send it to gmx-users-request at gromacs.org.
>> Can't post? Read http://www.gromacs.org/**Support/Mailing_Lists<http://www.gromacs.org/Support/Mailing_Lists>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://maillist.sys.kth.se/pipermail/gromacs.org_gmx-users/attachments/20120320/b7fb487b/attachment.html>


More information about the gromacs.org_gmx-users mailing list