[gmx-users] Multi-node Replica Exchange Segfault

[Mark Abraham]

Hi,

On Fri, Oct 30, 2015 at 9:59 AM <jkrieger at mrc-lmb.cam.ac.uk> wrote:

> You could try using a mixture of openmpi and thread-mpi.


Unfortunately not, replica exchange only works with real external MPI.


> I have found when
> linked replicas in multi-walker metadynamics that it only works if the
> replicas have the same allocation of the cluster.


Yes, that's normal. The multiple processes of an MPI program find out who
they need to talk to by being children of the same mpirun process, which
means they must come from the same job when running on a cluster.

Mark


> In Sun Grid Engine, I'd
> have the following in my submit scripts:
>
> #$ -pe openmpi 2
> #$ -l dedicated=20
> export OMP_NUM_THREADS=20
> mpirun -np $NSLOTS mdrun_mpi -deffnm metad -cpi metad -plumed plumed.dat
> -multi 2
>
> It's probably slightly different with PBS but you could try the equivalent
> without the plumed and then with replex.
>
> Best wishes
> James
>
> > Hi,
> >
> > I've never heard of such. You could try a multisim without -replex, to
> > help
> > diagnose.
> >
> > Mark
> >
> > On Fri, 30 Oct 2015 03:33 Barnett, James W <jbarnet4 at tulane.edu> wrote:
> >
> >> Good evening here,
> >>
> >> I get a segmentation fault with my GROMACS 5.1 install only for replica
> >> exchange
> >> simulations right at the first successful exchange on a multi-node run.
> >> Normal
> >> simulations across multiple nodes work fine, and replica exchange
> >> simulations on
> >> one node work fine.
> >>
> >> I've reproduced the problem with just 2 replicas on 2 nodes with GPU's
> >> disabled
> >> (-nb cpu). Each node has 20 CPU's so I'm using 20 MPI ranks on each
> >> (OpenMPI).
> >>
> >> I get a segfault right when the first exchange is successful.
> >>
> >> The only other error I get sometimes is that the Infiniband connection
> >> timed out
> >> retrying the communication between nodes at the exact same moment as the
> >> segfault, but I don't get that every time, and it's usually with all
> >> replicas
> >> going (my goal is to do 30 replicas on 120 cpus). No other error logs,
> >> and
> >> mdrun's log does not indicate an error.
> >>
> >> PBS log: http://bit.ly/1P8Vs49
> >> mdrun log: http://bit.ly/1RD0ViQ
> >>
> >> I'm currently troubleshooting this some with the sysadmin, but I wanted
> >> to
> >> check
> >> to see if anyone has had a similar issue or any further steps to
> >> troubleshoot.
> >> I've also searched the mailing list and used my Google-fu, but it has
> >> failed me
> >> so far.
> >>
> >> Thanks for your help.
> >>
> >> --
> >> James "Wes" Barnett, Ph.D. Candidate
> >> Louisiana Board of Regents Fellow
> >>
> >> Chemical and Biomolecular Engineering
> >> Tulane University
> >> 341-B Lindy Boggs Center for Energy and Biotechnology
> >> 6823 St. Charles Ave
> >> New Orleans, Louisiana 70118-5674
> >> jbarnet4 at tulane.edu
> >> --
> >> Gromacs Users mailing list
> >>
> >> * Please search the archive at
> >> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> >> posting!
> >>
> >> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >>
> >> * For (un)subscribe requests visit
> >> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> >> send a mail to gmx-users-request at gromacs.org.
> > --
> > Gromacs Users mailing list
> >
> > * Please search the archive at
> > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> > posting!
> >
> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >
> > * For (un)subscribe requests visit
> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send
> > a mail to gmx-users-request at gromacs.org.
> >
>
>
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.
>