[gmx-developers] manpage generation brakes build process

[Szilárd Páll]

On Thu, Aug 30, 2012 at 4:59 PM, Szilárd Páll <szilard.pall at cbr.su.se> wrote:
> On Thu, Aug 30, 2012 at 4:38 PM, Roland Schulz <roland at utk.edu> wrote:
>> Hi,
>>
>>
>>
>> On Thu, Aug 30, 2012 at 9:03 AM, Szilárd Páll <szilard.pall at cbr.su.se>
>> wrote:
>>>
>>> Hi,
>>>
>>> Commit 26bd5f7 made the build process generate man pages and broke
>>> compilation for the most common cross-compile builds: when the build
>>> host doesn't support the instruction set of the target, e.g. a
>>> cluster's head node with "older" CPUs than the cluster nodes. To me it
>>> seems that the CMAKE_CROSSCOMPILING variable used to implement the
>>> detection of cross-compilation is pretty much useless because
>>> apparently it takes into account only the CMAKE_SYSTEM_NAME. This only
>>> detects when one builds e.g Windows binaries on Linux, but nothing
>>> else.
>>>
>>> The annoying part is that the build process fails a cryptic "error
>>> 132" after linking the binary and the user won't have a clue what's
>>> wrong.
>>>
>>> In my opinion by default automatically generating man pages is not
>>> very important because:
>>> - source releases can contain the man pages;
>>
>> This is already so. Cpack automatically adds the build man pages and the man
>> pages are only build if they aren't in the source package. So this is a
>> non-issue for users which download the source as a source package instead
>> from git.
>>
>>>
>>> - the development versions are either:
>>>   - in pre-release stage when one wouldn't rely on the man pages being
>>> very up-to-date anyway;
>>>   - post-release stage when there should not be many major changes
>>> compared to the latest patch release.
>>>
>>> I would suggest that we stick to installing static in-source man pages
>>> by default and we can provide a switch to generate them during
>>> building. It is fairly complicated to detect whether the binaries
>>> built will run on the host,
>>
>> why? It is as simple as running a binary and see whether it worked, isn't
>> it?
>
> A binary or the binary that would be required to execute successfully
> to generate the man page?
>
> If it's only the architecture, instruction set, or operating system
> incompatibility that can cause issues compiling and running a dummy
> program should indeed be fine. However, I'm not sure that's always the
> case. I'll think it through.
>
> Does anybody know of cross-compilation cases where a dummy program
> would execute, but e.g. mdrun wouldn't?

After giving this a second thought and discussing it with others, I
think that it is not unfeasible to come up with a dummy program that
tests weather the given CFLAGS will produce GROMACS binaries that can
run on the host machine. However, this solution will certainly not be
robust and will probably be a  maintenance headache.

The issue is that illegal instructions are not only the asm or
intrinsics used explicitly in the code but also what the compiler
generates. As we allow the compiler to generate e.g. AVX instructions
when setting GMX_ACCELERATION=AVX_256, the compiler will happily
generate as much AVX as it can. As a result, a cross-compiled binary
will in my case (compiled with AVX on a CPU with SSE2 support) fail
already inside CopyRight(), apparently in the bromacs() call.

Therefore, I think we either have to revert back to using in-source
man pages (at least by default) or come up with a robust mechanism
that at least will clearly state what the problem is instead of
leaving the user with an "error 123"-type message (and some .err files
deep in the build tree).

Additionally, it would also be useful to have an early runtime check
that warns the user if it looks like the binary will/could fail. This
can happen relatively often e.g. when using multiple clusters with
shared file system from which users can inevitably end up running a
binary incompatible with the target cluster. A solution for this is to
compile the CPU detection module with no optimization, run the
detection as the very first thing in every tool's main() function and
issue an error if it is sure that things will go wrong or at least a
warning if the build host and target are different.

--
Szilárd

>>
>>>
>>> so I think it is not worth writing such
>>> code that is error-prone and hard to maintain.
>>>
>>> Does anybody have a better idea to fix this issue?
>>
>> I prefer if it is done automatically because that makes sure it is
>> maintained. I think we should have as little as possible of things we do in
>> the last second before a release (and thus delaying the release) to be able
>> to release more frequently.
>
> Maintaining the documentation has not much to with whether it is built
> at run-time or not and I can't imagine that there have been so many
> bugs in the man page generation itself (which is the only thing that's
> ensured to be working through the auto-builds).
>
>> I think it is OK as it is. Because disabling the build is as easy as
>> GMX_BUILD_MANPAGES=no, and only effects developers and not users. But if you
>> want me to add a check whether executing binaries is possible, I can do
>> that.
>
> Actually, many advanced users use the git version of the releases --
> partly because of the super-sluggish release cycle. I'm not saying
> that the feature is not useful, but users shouldn't have to first
> figure out how to make compilation not fail with a cryptic error that
> doesn't even suggest a solution.
>
> Please do implement it If you (plural intended) think that the feature
> is robust enough.
>
>> admin/.isreposource is not specific for github. It is for any download which
>> is not using git to download source which didn't go through cpack.
>
> That is? Still sounds quite exotic, 2-3 official means of getting
> Gromacs should be more than enough (git.gromacs.org, nightly source,
> github as backup).
>
> Again, I don't mind a single hidden file, but IMO we shouldn't add
> code, not even a single line just to support some exotic way to obtain
> the source.
>
> --
> Szilárd
>
>>  Roland
>>
>>>
>>> --
>>> Szilárd
>>> --
>>> gmx-developers mailing list
>>> gmx-developers at gromacs.org
>>> http://lists.gromacs.org/mailman/listinfo/gmx-developers
>>> Please don't post (un)subscribe requests to the list. Use the
>>> www interface or send it to gmx-developers-request at gromacs.org.
>>>
>>>
>>>
>>>
>>
>>
>>
>> --
>> ORNL/UT Center for Molecular Biophysics cmb.ornl.gov
>> 865-241-1537, ORNL PO BOX 2008 MS6309
>>
>> --
>> gmx-developers mailing list
>> gmx-developers at gromacs.org
>> http://lists.gromacs.org/mailman/listinfo/gmx-developers
>> Please don't post (un)subscribe requests to the list. Use the
>> www interface or send it to gmx-developers-request at gromacs.org.