[gmx-developers] parallel make problems

[Mark Abraham]

On Tue, Jun 18, 2013 at 8:16 PM, Marcus D. Hanwell <
marcus.hanwell at kitware.com> wrote:

> On Mon, Jun 17, 2013 at 1:59 PM, Roland Schulz <roland at utk.edu> wrote:
> > On Mon, Jun 17, 2013 at 1:10 PM, Mark Abraham <mark.j.abraham at gmail.com>
> wrote:
> >>
> >>
> >>
> >> On Mon, Jun 17, 2013 at 6:16 PM, Manuel Nuno Melo <m.n.melo at rug.nl>
> wrote:
> >>>
> >>> Hi,
> >>>
> >>> I have also had linking problems when making in parallel. In my case
> they
> >>> could be traced back to the option to let GMX download/build its own
> fftw
> >>> (-DGMX_BUILD_OWN_FFTW=ON).
> >>>
> >>> It seems that only one of make's threads starts building fftw, while
> the
> >>> others go ahead building/linking GMX. Since fftw compilation is not
> ready by
> >>> the time it is needed, GMX linking is botched.
> >>
> >>
> >> Yes, Rossen first showed this to me. I don't know if the underlying
> issue is
> >> that the dependency cannot be described properly, or that we're not
> doing it
> >> properly. If it's a problem, people are welcome to contribute a fix! :-)
> > It was working in https://gerrit.gromacs.org/#/c/1675/12. You then
> > changed how the dependency works in patch set 13. You never replied to
> > Christophs comment why this was changed (at least I can't find a
> > reply). Do you remember? Otherwise I can change it back as 12 did it
> > and it should work again.
> >
> One major issue is that external project was not designed to be used
> with standard targets, what we usually advise is the use of a
> superbuild that is all external projects - including GROMACS as the
> most dependent external project. That way when you get to GROMACS you
> know that all other dependencies were built (because that target
> depends on FFTW for example). This allows you to keep all of this
> logic outside of your project, and in a very small repository that
> coordinates the build of your project and its dependencies.
>

Interesting. So we might have a top-level CMakeLists.txt that does a lot of
detection stuff that could be re-used over packages (e.g. working out how
to build FFTW optimally). By the time we got to the GROMACS subdirectory we
would only have logic about how to build GROMACS given (perhaps partial)
context. Given that our src/CMakeLists.txt is basically a stub, that's not
a big deal. I have a bunch of draft patches from Teemu to review,
reorganizing our use of CMake for 5.0, so I should look at those in this
light.

A few examples of this include the Open Chemistry superbuild,
>
> https://github.com/OpenChemistry/openchemistry
>
> The ParaView superbuild,
>
> http://paraview.org/gitweb?p=ParaViewSuperbuild.git;a=summary
>
> These repositories provide additional convenience to get new
> developers up and running, but you can use the Avogadro, ParaView, VTK
> repositories alone if you build the dependencies. Once you introduce a
> single external project target, all others that want to use it need to
> be external projects if you would like to depend upon them and have
> reliable builds across all platforms, generators and compilers. This
> is in my experience at least (after working with several of the CMake
> developers on projects making use of external project).
>

Having such a structure would make the barrier to entry for people wanting
to add plugin functionality to (or from) GROMACS easier. Adding and calling
an external fast multipole solver or QM code is a much easier proposition
if done like this. Don't know if it might help PLUMED out. For 5.0, we
already have some imported fragments of Boost and Google Mock as well as
the historical fftpack and linear algebra stuff (and probably one or two
other single files). Reorganizing along the lines Marcus suggests could be
valuable. I'll take a look at those superbuilds and see how they work.

The other alternative that would work reliably is to use CMake to
> build FFTW (possibly creating the CMakeLists.txt files if they don't
> already exist) and using add_subdirectory to bring this in as a normal
> target. Something VTK does for many third party libraries for example,
> but the burden of maintaining a secondary build system for a project
> can be pretty high.
>

Yeah :-)

Mark


>
> Marcus
> --
> gmx-developers mailing list
> gmx-developers at gromacs.org
> http://lists.gromacs.org/mailman/listinfo/gmx-developers
> Please don't post (un)subscribe requests to the list. Use the
> www interface or send it to gmx-developers-request at gromacs.org.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://maillist.sys.kth.se/pipermail/gromacs.org_gmx-developers/attachments/20130619/339d70b2/attachment.html>