[gmx-developers] Disabling Pthreads

Erik Lindahl erik.lindahl at gmail.com
Wed Nov 11 09:35:05 CET 2020 

Hi,

Gilles: If this happens, I *suspect* this might only be in the hardware
detection phase, right?

We might indeed have overlooked that, since ARM was originally just a
low-end platform (think 32-bit Tegra...) where we never even thought of
running anything multi-node.

We've already thought of adding cgroups awareness (for docker), so could
you possibly assist me by showing a concrete example of this on A64fx:

1) lscpu

2) cgroups information, ideally both for the node and specific MPI processes

... since otherwise we're programming a bit blind :-)

Cheers,

Erik









On Wed, Nov 11, 2020 at 8:47 AM Berk Hess <hess at kth.se> wrote:

> Hi,
>
> Then there is a strange bug somewhere.
>
> Are you using real MPI or thread-MPI. Do you know where this happens?
> Are you running with OpenMP support?
>
> Cheers,
>
> Berk
>
> On 2020-11-11 08:26, Gilles Gouaillardet wrote:
> > Berk,
> >
> >
> > There is a total of 52 cores, and my observation is that each MPI task
> > does spawn 52 threads.
> >
> > So the worst case scenario is 48 MPI tasks each spawning 52 threads,
> > so a total of 48 * 52 threads on a single node
> >
> >
> > Cheers,
> >
> >
> > Gilles
> >
> > On 11/11/2020 4:23 PM, Berk Hess wrote:
> >> On 2020-11-11 03:37, Gilles Gouaillardet wrote:
> >>> Erik and all,
> >>>
> >>>
> >>> I am kind of facing the exact opposite issue on an other ARM processor:
> >>>
> >>> High end A64fx (Fugaku/FX1000) have 48 cores plus 2 or 4 assistant
> >>> cores.
> >>>
> >>> A job is put in a cgroup of 48 cores (e.g. no assistant cores)
> >>>
> >>> Worst case scenario, a flat mpi run (48 tasks) will spawn 48 * 52
> >>> cores to spin up all the cores.
> >> What do you mean with 48 * 52 cores? There are only 52 cores. Gromacs
> >> will by default not spawn more threads in total then there are cores.
> >> If you ask for 48 MPI ranks with a real MPI library Gromacs will not
> >> spawn any additional threads. With thread-mpi it will spawn 48 or 52
> >> total.
> >>
> >> Cheers,
> >>
> >> Berk
> >>>
> >>> 1) GROMACS is not cgroup aware and hence consider there are 52 (or
> >>> 50) cores per node (this is a very minor issue)
> >>>
> >>> 2) spawning such a high number of threads caused some crashes (weird
> >>> stack traces, I did not spend much time investigating)
> >>>
> >>> 3) in the case of A64fx, all cores are up and running, ready to
> >>> crunch, and do not require any special tricks.
> >>>
> >>>
> >>> At this stage, I think the easiest path to address this on A64fx is
> >>> to add yet an other cmake option to
> >>>
> >>> unconditionally skip the spin up phase.
> >>>
> >>> This could be improved by adding a command line option, or an
> >>> environment variable, to change the default behavior
> >>>
> >>> (default behavior should be a cmake option imho)
> >>>
> >>>
> >>> Any thoughts on how to best move forward?
> >>>
> >>>
> >>> Cheers,
> >>>
> >>>
> >>> Gilles
> >>>
> >>> On 11/11/2020 6:08 AM, Erik Lindahl wrote:
> >>>> We might be able to work around the last aspect, but it will likely
> >>>> take a couple of weeks until I can lay hands on a new ARM-based
> >>>> Macbook.
> >>>>
> >>>> Long story made short: The advanced power-saving features on ARM
> >>>> mean some cores are not visible until they are used, so we created
> >>>> a small hack where we "spin up" the CPU by exercising all cores.
> >>>>
> >>>> We might anyway need to do something different with the new type of
> >>>> big.LITTLE cores where we have 4+4 or 8+4 cores, but I can't even
> >>>> start to work on that until I have suitable hardware. The good news
> >>>> is that such hardware was announced a couple of hours ago, with
> >>>> availability next week ;-)
> >>>>
> >>>> Cheers,
> >>>>
> >>>> Erik
> >>>>
> >>>> On Tue, Nov 10, 2020 at 9:45 PM Mark Abraham
> >>>> <mark.j.abraham at gmail.com <mailto:mark.j.abraham at gmail.com>> wrote:
> >>>>
> >>>>     No, the use of std::thread in eg hardware detection also requires
> >>>>     a lower level threading implementation.
> >>>>
> >>>>     Mark
> >>>>
> >>>>     On Tue, Nov 10, 2020, 20:41 Berk Hess <hess at kth.se
> >>>>     <mailto:hess at kth.se>> wrote:
> >>>>
> >>>>         Hi,
> >>>>
> >>>>         Turning off GMX_THREAD_MPI in cmake should remove the
> >>>>         dependency on pthreads.
> >>>>
> >>>>         Cheers,
> >>>>
> >>>>         Berk
> >>>>
> >>>>         On 2020-11-10 18:06, Guido Giuntoli wrote:
> >>>>>
> >>>>>         Hi,
> >>>>>
> >>>>>         Is there any way to disable the “Pthreads” dependency during
> >>>>>         the configuration/compilation of GROMACS?
> >>>>>
> >>>>>         *Best regards | Mit freundlichen Grüßen*
> >>>>>
> >>>>>         **
> >>>>>
> >>>>>         *Guido Giuntoli***
> >>>>>
> >>>>>         **
> >>>>>
> >>>>>         HUAWEI TECHNOLOGIES Duesseldorf GmbH
> >>>>>         Hansaallee 205, 40549 Dusseldorf, Germany, *www.huawei.com*
> >>>>>         <http://www.huawei.com/>
> >>>>>         Registered Office: Düsseldorf, Register Court Düsseldorf, HRB
> >>>>>         56063,
> >>>>>         Managing Director: Li Peng, Li Jian, Shi Yanli**
> >>>>>
> >>>>>         Sitz der Gesellschaft: Düsseldorf, Amtsgericht Düsseldorf,
> >>>>>         HRB 56063,
> >>>>>         Geschäftsführer: Li Peng, Li Jian, Shi Yanli
> >>>>>
> >>>>>
> *-----------------------------------------------------------------------------------------------*
>
> >>>>>
> >>>>>
> >>>>>         *This e-mail and its attachments contain confidential
> >>>>>         information from HUAWEI, which is intended only for the
> >>>>>         person or entity whose address is listed above. Any use of
> >>>>>         the information contained herein in any way (including, but
> >>>>>         not limited to, total or partial disclosure, reproduction, or
> >>>>>         dissemination) by persons other than the intended
> >>>>>         recipient(s) is prohibited. If you receive this e-mail in
> >>>>>         error, please notify the sender by phone or email immediately
> >>>>>         and delete it!*
> >>>>>
> >>>>>
> >>>>
> >>>>         --         Gromacs Developers mailing list
> >>>>
> >>>>         * Please search the archive at
> >>>> http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List
> >>>> <http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List>
> >>>>         before posting!
> >>>>
> >>>>         * Can't post? Read
> >>>>         http://www.gromacs.org/Support/Mailing_Lists
> >>>>         <http://www.gromacs.org/Support/Mailing_Lists>
> >>>>
> >>>>         * For (un)subscribe requests visit
> >>>>
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers
> >>>>
> >>>> <
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers>
> >>>>
> >>>>         or send a mail to gmx-developers-request at gromacs.org
> >>>>         <mailto:gmx-developers-request at gromacs.org>.
> >>>>
> >>>>     --     Gromacs Developers mailing list
> >>>>
> >>>>     * Please search the archive at
> >>>> http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List
> >>>> <http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List>
> >>>>     before posting!
> >>>>
> >>>>     * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >>>>     <http://www.gromacs.org/Support/Mailing_Lists>
> >>>>
> >>>>     * For (un)subscribe requests visit
> >>>>
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers
> >>>>
> >>>> <
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers>
> >>>>
> >>>>     or send a mail to gmx-developers-request at gromacs.org
> >>>>     <mailto:gmx-developers-request at gromacs.org>.
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Erik Lindahl <erik.lindahl at dbb.su.se <mailto:erik.lindahl at dbb.su.se>>
> >>>> Professor of Biophysics, Dept. Biochemistry & Biophysics, Stockholm
> >>>> University
> >>>> Science for Life Laboratory, Box 1031, 17121 Solna, Sweden
> >>>>
> >>
>
> --
> Gromacs Developers mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers
> or send a mail to gmx-developers-request at gromacs.org.



-- 
Erik Lindahl <erik.lindahl at dbb.su.se>
Professor of Biophysics, Dept. Biochemistry & Biophysics, Stockholm
University
Science for Life Laboratory, Box 1031, 17121 Solna, Sweden
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://maillist.sys.kth.se/pipermail/gromacs.org_gmx-developers/attachments/20201111/2c33b596/attachment.html>

More information about the gromacs.org_gmx-developers mailing list