[gmx-users] Launch hybrid MPI/openMP run on multiple nodes

[Chunlei ZHANG]

Dear Sizilard and Mark,

Thank you very much for the explanations.
I found a line in the log file as:

OpenMP support:     enabled (GMX_OPENMP_MAX_THREADS = 32)

I wonder whether this setup limits the number of OpenMP threads and hence I
can not launch more than 32 threads on multiple nodes.

Best,
Chunlei

2016-01-15 22:01 GMT+08:00 Szilárd Páll <pall.szilard at gmail.com>:

> Hi,
>
> Your job scheduler and/or MPI launcher is most likely to blame. The fact
> that mdrun warns about the logical core (hardware thread) mismatch means
> that the OpenMP runtime thinks that you should be using one thread per
> rank. This typically means that the MPI launcher or job scheduler set an
> affinity mask for each rank and likely mdrun also skipped pinning threads
> because of this.
>
> This should however alone not cause nodes to be empty, but rather it would
> cause ranks to overlap and run on the same core.
>
> In any case, you should pass the correct ranks/node threads/rank settigns
> to your launchers; to ensure correct rank placement you'll have to either
> set up affinities through the scheduler or let mdrun do it.
>
> Cheers,
> --
> Szilárd
>
> On Fri, Jan 15, 2016 at 9:25 AM, Chunlei ZHANG <chunleizhang.pku at gmail.com
> >
> wrote:
>
> > Dear GMX developers and users,
> >
> > I have a cluster of 24 nodes, each having two 10-core intel CPUs.
> > Gromacs 5.1 is compiled by using intel mpi (version 5.1.1) and mkl.
> >
> > I can successfully run a simulation by using pure MPI (480 MPI
> processes).
> > But the performance is not good and the log file of mdrun suggests using
> > fewer MPI processes.
> > I try to launch 240 MPI processes, each using 2 openMP threads, by the
> > command:
> > mpirun -ppn 20 -np 240 gmx_mpi mdrun -ntomp 2
> >
> > But only a fraction of the nodes are running mdrun and the log file says:
> >
> > Number of logical cores detected (20) does not match the number reported
> by
> > OpenMP (1).
> > Consider setting the launch configuration manually!
> >
> > Does anyone know how to solve this problem?
> > Thanks in advance.
> >
> > Best,
> > Chunlei
> > --
> > Gromacs Users mailing list
> >
> > * Please search the archive at
> > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> > posting!
> >
> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >
> > * For (un)subscribe requests visit
> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> > send a mail to gmx-users-request at gromacs.org.
> >
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-request at gromacs.org.