Management of Python dependencies sources

Management of dependencies sources

generated from rpm-build-pyproject/docs/deps.md

Overview

This feature allows to manage runtime and/or buildtime dependencies of Python RPM package in Python-ecosystem manner for the purpose of automation (e.g. such dependencies can be synced, verified or evaluated). In particular, %pyproject_deps_resync-based macros collect PEP508 requirements from different sources and store them in %_sourcedir/pyproject_deps.json while %pyproject_builddeps and %pyproject_runtimedeps evaluate dependencies specified by this config file in current Python environment and generate BuildRequires and Requires RPM specfile’s entries respectively. Sources of dependencies can be either well known standard formats like core metadata, PEP518, PEP517 or tool-specific formats like pip reqfile, tox deps, etc.

Typical RPM specfile for Python package using this feature looks like:

...
Source1: %pyproject_deps_config_name
%pyproject_runtimedeps_metadata
BuildRequires(pre): rpm-build-pyproject
%pyproject_builddeps_build
%if_with check
%pyproject_builddeps_metadata
%endif
...
%prep
%setup
%pyproject_deps_resync_build
%pyproject_deps_resync_metadata

%pyproject_deps-based RPM macros are shipped in rpm-build-pyproject RPM package, that can be used as drop-in replacement for rpm-build-python3 in RPM specfile.

Bootstrap

How to start using this automation.

• add build dependency(pre) on rpm-build-pyproject

  BuildRequires(pre): rpm-build-pyproject
  

• add empty configuration for dependencies sources

  • configuration

    echo '{"sources":{}}' > .gear/pyproject_deps.json
    

  • gear rules

    echo 'copy: .gear/pyproject_deps.json' >> .gear/rules
    

  • RPM spec

    SourceN: %pyproject_deps_config_name
    

• add %pyproject_builddeps_build and %pyproject_deps_resync_build (requires unpacked sources) RPM macros for configuring and syncing build dependencies

• add %pyproject_runtimedeps_metadata and %pyproject_deps_resync_metadata (requires unpacked sources) RPM macros for configuring and syncing runtime dependencies

• add %pyproject_builddeps_metadata and specific check RPM macros for configuring and syncing tests dependencies

BUILD

Build dependencies format is standardized by PEP518 and PEP517.

• add %pyproject_builddeps_build macro into RPM specfile that:

  • eval dependencies of PEP518 and PEP517 sources (they will be defined later) in Python environment
  • map dependencies to distro names
  • generate RPM BuildRequires tags

• append %pyproject_deps_resync_build macro to RPM %prep section. This macro:

  • configure PEP518 source of dependencies if it doesn’t exist yet
  • verify PEP518 source and resync it if needed
  • configure PEP517 source of dependencies if it doesn’t exist yet
  • verify PEP517 source and resync it if needed

• run build. The RPM build may fail on verification of PEP518 dependencies (if any). Update stored config with produced one if needed. For example,

  hsh-run -- /bin/bash -ec 'cat "$(rpm --eval %pyproject_deps_config)"' > .gear/pyproject_deps.json
  

• %pyproject_deps-based RPM macros expect to find its configuration in RPM sources directory by default. Hence,

  • add copy: .gear/pyproject_deps.json to .gear/rules
  • add Source1: %pyproject_deps_config_name to RPM specfile (the source number doesn’t matter)

• exclude filters can be applied on evaluation of dependencies. For example, filter out dependencies having normalized names that match types-:

  %set_pyproject_deps_build_filter types-

  or

  %add_pyproject_deps_build_filter types-

• rerun build. The RPM build may fail on verification of PEP517 dependencies (if any). Update stored config with produced one if needed. For example,

  hsh-run -- /bin/bash -ec 'cat "$(rpm --eval %pyproject_deps_config)"' > .gear/pyproject_deps.json
  

This is enough to manage Python build dependencies of vast of majority of Python projects.

RUNTIME

By default, runtime dependencies of Python projects in ALTLinux are generated automatically by parsing source code and fetching of all of external imports (this feature is known as autoreq). This may or may not work well. For example:

• conditional or optional dependencies can’t be automatically accounted
• installation tools like pip may expect all installed dependencies of projects

Downstream maintainers have to manually handle such cases.

Runtime dependencies format is standardized by core metadata and it’s usually generated by build backend from some tool-specific format. Produced list of dependencies is not necessary and sufficient. But it may complement auto-generated dependencies to cover unhandled cases.

• add %pyproject_runtimedeps_metadata macro into RPM specfile that:
  • eval dependencies of metadata source (it will be defined later) in Python environment
  • map dependencies to distro names
  • generate RPM Requires tags

Note: if some specific extras is required the %pyproject_runtimedeps_metadata_extra macro can be used instead (in this case all the runtime dependencies of current package and dependencies that are marked with the given extra of current package will be installed)

• append %pyproject_deps_resync_metadata macro to RPM %prep section. This macro:

  • configure core metadata source of dependencies if it doesn’t exist yet
  • verify core metadata source and resync it if needed

• rerun build. The RPM build may fail on verification of metadata dependencies (if any). Update stored config with produced one if needed. For example,

  hsh-run -- /bin/bash -ec 'cat "$(rpm --eval %pyproject_deps_config)"' > .gear/pyproject_deps.json
  

• exclude filters can be applied on evaluation of dependencies: For example, filter out dependencies having normalized names that match types-:

  %set_pyproject_deps_runtime_filter types-

  or

  %add_pyproject_deps_runtime_filter types-

CHECK

Tests dependencies format is not standardized, but pyproject_installer supports most common of them like:

• core metadata
• pip’s reqfile
• poetry group
• tox env
• hatch env
• pdm group
• pipenv category

Usually tests require all runtime dependencies of project and additional packages for test environment (e.g. pytest). Thus, configuration of several sources may be required (e.g. metadata + pip_reqfile).

check filter

There can be many of redundant dependencies like:

• code style checkers that are really helpful in development but they often report different results from version to version.
• coverage measurement, it’s pointless overhead to do this for RPM packaging.
• publishing tools
• documentation tools

Hence, there is a predefined filter of check dependencies %pyproject_deps_check_filter. This filter is applied automatically on %pyproject_builddeps_check or %pyproject_builddeps_metadata macro usage.

Today’s default filter includes:

• %{?pypi_name:‘%{pep503_name %pypi_name}$’}
• flake8 and its plugins
• black and its plugins
• coverage and its plugins
• codecov and its plugins
• coveralls and its plugins
• mypy and its plugins
• isort and its plugins
• pre-commit and its plugins
• pytest-checkdocs
• twine and its plugins
• ruff and its plugins
• typing stubs
• pyright and its plugins
• Sphinx and its plugins

This filter can be changed like:

%set_pyproject_deps_check_filter foo bar

or

%add_pyproject_deps_check_filter foo bar

Note: this is exposed as deps –exclude option that accepts regex patterns and excludes requirement having PEP503-normalized name which matches one of these patterns. Thus, if only the name should be excluded then this name should be normalized first:

%add_pyproject_deps_check_filter '%{pep503_name Foo-Project.my}$'

core metadata

Specification:

• https://packaging.python.org/en/latest/specifications/core-metadata/
• https://packaging.python.org/en/latest/specifications/core-metadata/#requires-dist-multiple-use

For example, upstream use core metadata as source of tests dependencies and mark them with testing extra (this means that all runtime dependencies of current package and any dependencies that are listed in the testing extra of current package will be installed, see extras). Usually this is exposed as extras = testing in tox.ini or pip install .[testing] in CI config.

• add macro to RPM specfile

  %if_with check
  %pyproject_builddeps_metadata_extra testing
  %endif

  This macro:

  • eval dependencies of metadata source (it’s already defined as source of runtime dependencies, otherwise it should be defined manually) in Python environment with testing extra
  • filter out dependencies according to %pyproject_deps_check_filter
  • map dependencies to distro names
  • generate RPM BuildRequires tags

• filter out extra dependencies with check-filter

pip reqfile

Specification:

• https://pip.pypa.io/en/stable/reference/requirements-file-format/#requirements-file-format

Limitations:

• supported only PEP508 requirements
• line continuations are not supported
• pip’s inline options are not supported

For example, upstream use pip’s reqfile as source of tests dependencies and it’s saved as test-requirements.txt.

• add macro to RPM specfile

  %if_with check
  %pyproject_builddeps_check
  %endif

  This macro:

  • eval dependencies of check source (it will be defined later) in Python environment
  • filter out dependencies according to %pyproject_deps_check_filter
  • map dependencies to distro names
  • generate RPM BuildRequires tags

• configure check source of dependencies by appending to RPM %prep section:

  %if_with check
  %pyproject_deps_resync_check_pipreqfile test-requirements.txt
  %endif

  or source having arbitrary name

  %if_with check
  %pyproject_deps_resync check pip_reqfile test-requirements.txt
  %endif

• rerun build. The RPM build may fail on verification of check dependencies (if any). Update stored config with produced one if needed. For example,

  hsh-run -- /bin/bash -ec 'cat "$(rpm --eval %pyproject_deps_config)"' > .gear/pyproject_deps.json
  

• filter out extra dependencies with check-filter

poetry group

Specification:

• https://python-poetry.org/docs/dependency-specification/
• https://python-poetry.org/docs/managing-dependencies/

Limitations:

• versions are not supported (poetry version format is not compatible with PEP440)
• python field is not supported

For example, upstream uses poetry dependencies group dev as source of tests dependencies. By default, poetry installs dependencies of all non-optional groups including dependencies of project. Hence, runtime should be configured first.

• add macro to RPM specfile

  %if_with check
  %pyproject_builddeps_metadata
  %pyproject_builddeps_check
  %endif

  These macros:

  • eval dependencies of metadata and check sources (metadata is already defined as source of runtime dependencies, otherwise it should be defined manually) in Python environment
  • filter out dependencies according to %pyproject_deps_check_filter
  • map dependencies to distro names
  • generate RPM BuildRequires tags

• configure check source of dependencies by appending to RPM %prep section:

  %if_with check
  %pyproject_deps_resync_check_poetry dev
  %endif

  or source having arbitrary name

  %if_with check
  %pyproject_deps_resync check poetry dev
  %endif

• rerun build. The RPM build may fail on verification of check dependencies (if any). Update stored config with produced one if needed. For example,

  hsh-run -- /bin/bash -ec 'cat "$(rpm --eval %pyproject_deps_config)"' > .gear/pyproject_deps.json
  

• filter out extra dependencies with check-filter

tox env

Specification:

• https://tox.wiki/en/latest/config.html#deps
• config formats: https://tox.wiki/en/latest/config.html#discovery-and-file-types

Limitations:

• supported only PEP508 requirements
• config substitutions are not supported
• testenv inheritance is not supported
• pip’s req files or options are not supported

Note: if deps includes pip’s requirements file then use pip’s reqfile instead. If testenv has extras then use core metadata instead.

For example, upstream uses tox’ deps of testenv section of tox.ini config as source of tests dependencies. By default, tox installs these dependencies prior to package installation and then installs package and its runtime dependencies. Hence, runtime should be configured first.

• add macro to RPM specfile

  %if_with check
  %pyproject_builddeps_metadata
  %pyproject_builddeps_check
  %endif

  These macros:

  • eval dependencies of metadata and check sources (metadata is already defined as source of runtime dependencies, otherwise it should be defined manually) in Python environment
  • filter out dependencies according to %pyproject_deps_check_filter
  • map dependencies to distro names
  • generate RPM BuildRequires tags

• configure check source of dependencies by appending to RPM %prep section:

  %if_with check
  %pyproject_deps_resync_check_tox tox.ini testenv
  %endif

  or source having arbitrary name

  %if_with check
  %pyproject_deps_resync check tox tox.ini testenv
  %endif

• rerun build. The RPM build may fail on verification of check dependencies (if any). Update stored config with produced one if needed. For example,

  hsh-run -- /bin/bash -ec 'cat "$(rpm --eval %pyproject_deps_config)"' > .gear/pyproject_deps.json
  

• filter out extra dependencies with check-filter

hatch env

Specification:

• https://hatch.pypa.io/latest/config/environment/overview/#dependencies
• https://hatch.pypa.io/latest/intro/#configuration

Limitations:

• supported only PEP508 requirements
• env inheritance is not supported
• context formatting is not supported
• features(extra) is not supported

Note: if env has features then use core metadata with extra instead.

For example, upstream uses hatch’s dependencies of test env in hatch.toml config as source of tests dependencies. By default, hatch installs these dependencies in addition to the runtime ones. Hence, runtime should be configured first.

• add macro to RPM specfile

  %if_with check
  %pyproject_builddeps_metadata
  %pyproject_builddeps_check
  %endif

  These macros:

  • eval dependencies of metadata and check sources (metadata is already defined as source of runtime dependencies, otherwise it should be defined manually) in Python environment
  • filter out dependencies according to %pyproject_deps_check_filter
  • map dependencies to distro names
  • generate RPM BuildRequires tags

• configure check source of dependencies by appending to RPM %prep section:

  %if_with check
  %pyproject_deps_resync_check_hatch hatch.toml test
  %endif

  or source having arbitrary name

  %if_with check
  %pyproject_deps_resync YOURSOURCENAME hatch hatch.toml test
  %endif

• rerun build. The RPM build may fail on verification of check dependencies (if any). Update stored config with produced one if needed. For example,

  hsh-run -- /bin/bash -ec 'cat "$(rpm --eval %pyproject_deps_config)"' > .gear/pyproject_deps.json
  

• filter out extra dependencies with check-filter

pdm group

Specification:

• https://pdm.fming.dev/latest/usage/dependency/#add-development-only-dependencies

Limitations:

• supported only PEP508 requirements

For example, upstream uses pdm dev dependencies’ group test as source of tests dependencies. By default, pdm installs dependencies of all non-optional groups including runtime dependencies of project. Hence, runtime should be configured first.

• add macro to RPM specfile

  %if_with check
  %pyproject_builddeps_metadata
  %pyproject_builddeps_check
  %endif

  These macros:

  • eval dependencies of metadata and check sources (metadata is already defined as source of runtime dependencies, otherwise it should be defined manually) in Python environment
  • filter out dependencies according to %pyproject_deps_check_filter
  • map dependencies to distro names
  • generate RPM BuildRequires tags

• configure check source of dependencies by appending to RPM %prep section:

  %if_with check
  %pyproject_deps_resync_check_pdm test
  %endif

  or source having arbitrary name

  %if_with check
  %pyproject_deps_resync YOURSOURCENAME pdm test
  %endif

• rerun build. The RPM build may fail on verification of check dependencies (if any). Update stored config with produced one if needed. For example,

  hsh-run -- /bin/bash -ec 'cat "$(rpm --eval %pyproject_deps_config)"' > .gear/pyproject_deps.json
  

• filter out extra dependencies with check-filter

pipenv category

Specification:

• https://pipenv.pypa.io/en/latest/pipfile.html
• https://pipenv.pypa.io/en/latest/specifiers.html

Limitations:

• standalone PEP508 specifiers as keys are not supported

For example, upstream uses pipenv dev dependencies’ category dev-packages as a source of tests dependencies. By default, pipenv installs dependencies of packages category and runtime dependencies should be configured first.

• add macro to RPM specfile

  %if_with check
  %pyproject_builddeps_metadata
  %pyproject_builddeps_check
  %endif

  These macros:

  • eval dependencies of metadata and check sources (metadata is already defined as source of runtime dependencies, otherwise it should be defined manually) in Python environment
  • filter out dependencies according to %pyproject_deps_check_filter
  • map dependencies to distro names
  • generate RPM BuildRequires tags

• configure check source of dependencies by appending to RPM %prep section:

  %if_with check
  %pyproject_deps_resync_check_pipenv Pipfile dev-packages
  %endif

  or source having arbitrary name

  %if_with check
  %pyproject_deps_resync YOURSOURCENAME pipenv Pipfile dev-packages
  %endif

• rerun build. The RPM build may fail on verification of check dependencies (if any). Update stored config with produced one if needed. For example,

  hsh-run -- /bin/bash -ec 'cat "$(rpm --eval %pyproject_deps_config)"' > .gear/pyproject_deps.json
  

• filter out extra dependencies with check-filter

Mapping project names to distro names

Python ecosystem uses PEP503 normalization rules for names comparison:

• https://packaging.python.org/en/latest/specifications/name-normalization/#package-name-normalization
• https://peps.python.org/pep-0503/#normalized-names

Mapping rule in ALTLinux:

project_name => python3-module-%{pep503_name project_name}

This almost always works out of the box, but distro packages having unnormalized names must provide such the name:

Provides: python3-module-%{pep503_name %pypi_name} = %EVR

The following naming rule for new Python distro packages is strongly encouraged:

Name: python3-module-%{pypi_nname}

where %pypi_nname must be equal to the result of evaluation of %{pep503_name %pypi_name} macro.

Disable verify mode on syncing deps

By default %pyproject_deps_resync_* macros sync dependencies, save updated configuration, but print diff and exit with code 4 if deps were unsynced.

It can be helpful for automation or bootstrap to make it possible to disable verify mode (to not fail on unsynced deps).

One can do this by defining pyproject_deps_verify_skip macro name, e.g. in RPM specfile:

%define pyproject_deps_verify_skip 1

or via rpmbuild’s cli:

rpmbuild --define='pyproject_deps_verify_skip 1' ...