[gmx-developers] Python package build

[Eric Irrgang]

Thank you for the thorough comments. Some clarifications inline below.

On Tue, Jan 21, 2020 at 1:59 PM Stian Soiland-Reyes <
soiland-reyes at cs.manchester.ac.uk> wrote:

> First case avoid Gromacs ABI mismatch as you point out, but is then more
> sensitive to Python's ABIs, e.g. you update your Python from 3.7 to 3.8
> and you might need a whole new Gromacs compile.
>

They are equally sensitive to Python version, but the cost to rebuild is
different.


> It is also possible to avoid this by
> sticking to <https://docs.python.org/3/c-api/stable.html>
>

This does not provide ABI stability, and recompilation is still required.
Python ABI stability is improving, but I don't think we can ever count on a
single module build for all Python installations.


> IMHO think first option is preferable if the Python API is co-evolving
> with the C++ API.
>
> However this means Python users are not able to depend on an older
> Python API with a newer Gromacs - they are stuck to whatever Python API
> comes with their chosen Gromacs.
>

Python API stability is a goal in any case, but we aren't there yet. In
practice, we are not currently attempting to support multiple GROMACS
releases for gmxapi 0.x versions, and this build system question does not
affect that.

As the Python API stabilizes, we can start to provide a compatibility
guarantee by updating the Python package implementation in terms of the
evolving GROMACS. New versions of GROMACS will require Python package
updates until there is a sufficient and stable GROMACS C++ API (if ever).
There is currently no expectation of a stable GROMACS ABI.

When the Python-side API is stable this is less of an issue - but it
> means it's harder to have separate semantic versioning for the Python
> API.
>

This is an open question, but somewhat divorced from the present issue,
since the GROMACS repository is where the Python package sources live for
all packaging scenarios.

In other words:
* We won't tag gmxapi 1.0 until we are sure that Python scripts written for
it will also run on future gmxapi 1.x.
* To use a new version of GROMACS the Python bindings package will need to
be updated for the foreseeable future, and the Python package will likely
always need to be recompiled.
* It is unclear whether/when we will try to support older versions of
GROMACS in updated Python packages, but it is clear that this is not a
priority for the current development cycle.


> Having "pip install" do the compilation of the binding on install is
> possible (scikit-build), although slightly awkward as the client may
> need to have the right compiler paths etc, not to mention find the right
> gromacs install and headers. If they have installed Gromacs from a
> binary package (Conda, Debian, modules), then this may not be straight
> forward, although GMXRC helps.  <https://scikit-build.readthedocs.io/>
>

This is the situation we are in right now, and the reason I raised the
question.
Refs:
*
https://redmine.gromacs.org/projects/gromacs/repository/revisions/master/entry/python_packaging/src/setup.py
*
https://redmine.gromacs.org/projects/gromacs/repository/revisions/master/entry/python_packaging/src/CMakeLists.txt


> Again this would be sensitive to updates - would the pip-version need to
> match the gromacs version or would the older Python code seemlessly work
> if Gromacs got updated later, or if for some reason they wanted to
> compare Gromacs 2019 and 2020 using the same Python calls?
>

The long-term plan is that the pip-installed version will be tied to a
specific GROMACS installation, and different GROMACS versions would be
comparable through separate virtual environments in which gmxapi uses
different GROMACS installations. Under the proposed change, these GROMACS
installations would be self-contained within the Python package
installation instead of managed separately.


> As Eric point out this means the gmxapi would be more restricted in its
> changes - as the Python lib is installed separately it is not priviliged
> compared to other API users - "eat our own dogfood" could flag up
> any unintentional API/ABI changes.
>

Exactly. And I think this is still a plausible long-term goal. I don't know
whether it is worth trying to maintain at this stage, though.


> Alternatively you can include the prebuilt shared library in the
> published wheel package, which you would then need to build for multiple
> architectures and OSes - you are then sensitive to the installation path
> say in virtualenv (RPath magic to the rescue?).
>

There has been talk for the last few development cycles about making such
updates to libgromacs, but I don't think it is on a concrete roadmap or
likely to happen any time soon. But there's really no need to use a
separate shared library in this case, since the idea would be to use the
CMake infrastructure to give us a static target build with
position-independent code that we can link into the Python bindings module
shared object.


> I doubt this is right for Gromacs given the diverse hardware usage base.
> <https://packaging.python.org/guides/packaging-binary-extensions/>
>

Again, I am skeptical that it will ever be feasible to distribute the
Python package as a precompiled binary or that there would be a compelling
reason to do so.

Thank you again for your comments. I think this has been a good review of
the considerations and state of development.

Best,
Eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://maillist.sys.kth.se/pipermail/gromacs.org_gmx-developers/attachments/20200121/c7a01bbe/attachment-0001.html>