[gmx-users] Assistance needed running gromacs 4.6.3 on Blue Gene/P

[Christopher Neale]

I have run into similar error messages running gromacs 4.6.2 on some new Xeon's. That cluster is having file system problems sporadically and I end up with jobs that die about 1% of the time with message like this:

mdrun_mpi:21408 terminated with signal 6 at PC=38e02328a5 SP=7f534f467738.  Backtrace:

...

/lib64/libc.so.6(gsignal+0x35)[0x38e02328a5]
/lib64/libc.so.6(abort+0x175)[0x38e0234085]
/sb/software/CentOS-6/applications/gromacs/4.6.2-mvapich2-intel/bin/../lib/libgmx_mpi.so.8(+0x12bbfd)[0x7f5355e22bfd]
/lib64/libpthread.so.0[0x38e060f500]
/lib64/libc.so.6(__poll+0x53)[0x38e02dea63]
/usr/lib64/libpsm_infinipath.so.1(+0x215fd)[0x7f53557055fd]
/lib64/libpthread.so.0[0x38e0607851]
/lib64/libc.so.6(clone+0x6d)[0x38e02e811d]
[mpiexec at sw-4r02-n57] HYDU_sock_write (./utils/sock/sock.c:291): write error (Bad file descriptor)
[mpiexec at sw-4r02-n57] HYD_pmcd_pmiserv_send_signal (./pm/pmiserv/pmiserv_cb.c:170): unable to write data to proxy
[mpiexec at sw-4r02-n57] ui_cmd_cb (./pm/pmiserv/pmiserv_pmci.c:79): unable to send signal downstream
[mpiexec at sw-4r02-n57] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
[mpiexec at sw-4r02-n57] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:197): error waiting for event
[mpiexec at sw-4r02-n57] main (./ui/mpich/mpiexec.c:331): process manager error waiting for completion


Turns out that I have backtraces available, but the Sys admins at my local cluster say its a filesystem problem, not a gromacs problem.

I may be way off track since the systems differ so much, but if its a GPFS problem, then it might be related.

Chris. 
________________________________________
From: gromacs.org_gmx-users-bounces at maillist.sys.kth.se <gromacs.org_gmx-users-bounces at maillist.sys.kth.se> on behalf of Mark Abraham <mark.j.abraham at gmail.com>
Sent: 13 March 2014 01:43
To: Discussion list for GROMACS users
Subject: Re: [gmx-users] Assistance needed running gromacs 4.6.3 on Blue        Gene/P

Nope. Need a back trace.

Mark
On Mar 13, 2014 4:59 AM, "arrow50311" <linxingcheng50311 at gmail.com> wrote:

> Hi,
>
> Gromacs 4.0.7 is different story, and I did not test it with the same
> system
> though.
>
> The job script is as follows
>
> #@ job_name = job4.6
> #@ comment = "Hello World Job"
> #@ error = $(job_name).$(jobid).err
> #@ output = $(job_name).$(jobid).out
> #@ environment = COPY_ALL
> #@ wall_clock_limit = 10:00:00
> #@ notification = error
> #@ job_type = bluegene
> #@ class = compute
> #@ bg_size = 128
> # #@ bg_connection = torus
> #@ queue
> dir=/home/user/test
> GROM=4.6
> DDIR=/bgpscratch/user/test.$GROM
> mkdir -p $DDIR
> cd $DDIR
> MDRUN=/opt/apps/gromacs/$GROM/bin/mdrun_mpi
>
> ln -s $dir/run.tpr run.tpr
>
> /bgsys/drivers/ppcfloor/bin/mpirun -cwd $DDIR -mode VN -np 512 -exe $MDRUN
> -args "-s run.tpr -pd"
>
>
> With domain decomposition, it will be killed by signal 6, with particle
> decomposition, it will be killed by signal 11. I have no idea why it is
> like
> that.
>
> Do you have any suggestions?
>
> Thank you,
>
>
>
>
>
> --
> View this message in context:
> http://gromacs.5086.x6.nabble.com/Assistance-needed-running-gromacs-4-6-3-on-Blue-Gene-P-tp5010370p5015116.html
> Sent from the GROMACS Users Forum mailing list archive at Nabble.com.
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
>
--
Gromacs Users mailing list

* Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!

* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists

* For (un)subscribe requests visit
https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a mail to gmx-users-request at gromacs.org.