[gmx-developers] parallel make problems

[Marcus D. Hanwell]

On Mon, Jun 17, 2013 at 1:59 PM, Roland Schulz <roland at utk.edu> wrote:
> On Mon, Jun 17, 2013 at 1:10 PM, Mark Abraham <mark.j.abraham at gmail.com> wrote:
>>
>>
>>
>> On Mon, Jun 17, 2013 at 6:16 PM, Manuel Nuno Melo <m.n.melo at rug.nl> wrote:
>>>
>>> Hi,
>>>
>>> I have also had linking problems when making in parallel. In my case they
>>> could be traced back to the option to let GMX download/build its own fftw
>>> (-DGMX_BUILD_OWN_FFTW=ON).
>>>
>>> It seems that only one of make's threads starts building fftw, while the
>>> others go ahead building/linking GMX. Since fftw compilation is not ready by
>>> the time it is needed, GMX linking is botched.
>>
>>
>> Yes, Rossen first showed this to me. I don't know if the underlying issue is
>> that the dependency cannot be described properly, or that we're not doing it
>> properly. If it's a problem, people are welcome to contribute a fix! :-)
> It was working in https://gerrit.gromacs.org/#/c/1675/12. You then
> changed how the dependency works in patch set 13. You never replied to
> Christophs comment why this was changed (at least I can't find a
> reply). Do you remember? Otherwise I can change it back as 12 did it
> and it should work again.
>
One major issue is that external project was not designed to be used
with standard targets, what we usually advise is the use of a
superbuild that is all external projects - including GROMACS as the
most dependent external project. That way when you get to GROMACS you
know that all other dependencies were built (because that target
depends on FFTW for example). This allows you to keep all of this
logic outside of your project, and in a very small repository that
coordinates the build of your project and its dependencies.

A few examples of this include the Open Chemistry superbuild,

https://github.com/OpenChemistry/openchemistry

The ParaView superbuild,

http://paraview.org/gitweb?p=ParaViewSuperbuild.git;a=summary

These repositories provide additional convenience to get new
developers up and running, but you can use the Avogadro, ParaView, VTK
repositories alone if you build the dependencies. Once you introduce a
single external project target, all others that want to use it need to
be external projects if you would like to depend upon them and have
reliable builds across all platforms, generators and compilers. This
is in my experience at least (after working with several of the CMake
developers on projects making use of external project).

The other alternative that would work reliably is to use CMake to
build FFTW (possibly creating the CMakeLists.txt files if they don't
already exist) and using add_subdirectory to bring this in as a normal
target. Something VTK does for many third party libraries for example,
but the burden of maintaining a secondary build system for a project
can be pretty high.

Marcus