[gmx-developers] Python package build

[Stian Soiland-Reyes]

On Thu, 16 Jan 2020 21:12:51 +0300, Eric Irrgang <ericirrgang at gmail.com> wrote:
> In the teleconference yesterday, I asked you to think about issue
> https://redmine.gromacs.org/issues/2896
> 
> Specifically, I am asking you to weigh in on whether the gmxapi Python
> package should continue to be a client package of a GROMACS installation,
> or whether it should _be_ a GROMACS installation.
> 
> ...
> In summary:
> 
> # Reasons to turn gmxapi Python package build into a main GROMACS build
> system entry point
> ..
> # Reasons to keep gmxapi Python package as a client of the public C++
> interface
> ..

Coming from BioConda perspective the first case would be simpler (single
recipe provide bin & py), although second case in theory is cleaner
(separate conda package for py binding).  

Debian packaging is used to both variants - and frequently do a
single build of single src to produce multiple packages of the different
parts (e.g. gromacs libgromacs libgromacs-dev python-gromacs)


First case avoid Gromacs ABI mismatch as you point out, but is then more
sensitive to Python's ABIs, e.g. you update your Python from 3.7 to 3.8
and you might need a whole new Gromacs compile. 

For Conda this is OK because a conda environment always comes with a
particular Python version and so should not be as sensitive to OS
distribution's Python upgrades).  It is also possible to avoid this by
sticking to <https://docs.python.org/3/c-api/stable.html>

IMHO think first option is preferable if the Python API is co-evolving
with the C++ API.

However this means Python users are not able to depend on an older
Python API with a newer Gromacs - they are stuck to whatever Python API
comes with their chosen Gromacs.

When the Python-side API is stable this is less of an issue - but it
means it's harder to have separate semantic versioning for the Python
API.


Having "pip install" do the compilation of the binding on install is
possible (scikit-build), although slightly awkward as the client may
need to have the right compiler paths etc, not to mention find the right
gromacs install and headers. If they have installed Gromacs from a
binary package (Conda, Debian, modules), then this may not be straight
forward, although GMXRC helps.  <https://scikit-build.readthedocs.io/>

Again this would be sensitive to updates - would the pip-version need to
match the gromacs version or would the older Python code seemlessly work
if Gromacs got updated later, or if for some reason they wanted to
compare Gromacs 2019 and 2020 using the same Python calls?

As Eric point out this means the gmxapi would be more restricted in its
changes - as the Python lib is installed separately it is not priviliged
compared to other API users - "eat our own dogfood" could flag up
any unintentional API/ABI changes.


Alternatively you can include the prebuilt shared library in the
published wheel package, which you would then need to build for multiple
architectures and OSes - you are then sensitive to the installation path
say in virtualenv (RPath magic to the rescue?).
I doubt this is right for Gromacs given the diverse hardware usage base.
<https://packaging.python.org/guides/packaging-binary-extensions/>

-- 
Stian Soiland-Reyes
The University of Manchester 🐝
https://www.esciencelab.org.uk/
https://orcid.org/0000-0001-9842-9718