Management of Python dependencies sources

Management of dependencies sources

Collect PEP508 requirements from different sources, store and evaluate them in Python environment.

See RFE for details.

%pyproject_deps-based RPM macros are shipped in rpm-build-pyproject RPM package, that in turn depends on rpm-build-python3. Hence, rpm-build-python3 should be replaced with rpm-build-pyproject in RPM specfile.

BUILD

Build dependencies format is standardized by PEP518 and PEP517.

• add %pyproject_builddeps_build macro into RPM specfile that:

  • eval dependencies of PEP518 and PEP517 sources (they will be defined later) in Python environment
  • map dependencies to distro names
  • generate RPM BuildRequires tags

• append %pyproject_deps_resync_build macro to RPM %prep section. This macro:

  • configure PEP518 source of dependencies if it doesn’t exist yet
  • verify PEP518 source and resync it if needed
  • configure PEP517 source of dependencies if it doesn’t exist yet
  • verify PEP517 source and resync it if needed

• run build. The RPM build may fail on verification of PEP518 dependencies (if any). Update stored config with produced one if needed. For example,

  cp ~/hasher/chroot/usr/src/RPM/SOURCES/pyproject_deps.json .gear/
  

• %pyproject_deps-based RPM macros expect to find its configuration in RPM sources directory by default. Hence,

  • add copy: .gear/pyproject_deps.json to .gear/rules
  • add SourceXXXX: pyproject_deps.json to RPM specfile (the source number doesn’t matter)

• exclude filters can be applied on evaluation of dependencies. For example, filter out dependencies having normalized names that match types-:

  %set_pyproject_deps_build_filter types-

  or

  %add_pyproject_deps_build_filter types-

• rerun build. The RPM build may fail on verification of PEP517 dependencies (if any). Update stored config with produced one if needed. For example,

  cp ~/hasher/chroot/usr/src/RPM/SOURCES/pyproject_deps.json .gear/
  

This is enough to manage Python build dependencies of vast of majority of Python projects.

RUNTIME

By default, runtime dependencies of Python projects in ALTLinux are generated automatically by parsing source code and fetching of all of external imports (this feature is known as autoreq). This may or may not work well. For example:

• conditional or optional dependencies can’t be automatically accounted
• installation tools like pip may expect all installed dependencies of projects

Downstream maintainers have to manually handle such cases.

Runtime dependencies format is standardized by core metadata and it’s usually generated by build backend from some tool-specific format. Produced list of dependencies is not necessary and sufficient. But it may complement auto-generated dependencies to cover unhandled cases.

• add %pyproject_runtimedeps_metadata macro into RPM specfile that:

  • eval dependencies of metadata source (it will be defined later) in Python environment
  • map dependencies to distro names
  • generate RPM Requires tags

• append %pyproject_deps_resync_metadata macro to RPM %prep section. This macro:

  • configure core metadata source of dependencies if it doesn’t exist yet
  • verify core metadata source and resync it if needed

• rerun build. The RPM build may fail on verification of metadata dependencies (if any). Update stored config with produced one if needed. For example,

  cp ~/hasher/chroot/usr/src/RPM/SOURCES/pyproject_deps.json .gear/
  

• exclude filters can be applied on evaluation of dependencies: For example, filter out dependencies having normalized names that match types-:

  %set_pyproject_deps_runtime_filter types-

  or

  %add_pyproject_deps_runtime_filter types-

CHECK

Tests dependencies format is not standardized, but pyproject_installer supports most common of them like:

• core metadata
• pip’s reqfile
• poetry group
• tox env

Usually tests require all runtime dependencies of project and additional packages for test environment (e.g. pytest). Thus, configuration of several sources may be required (e.g. metadata + pip_reqfile).

check filter

There can be many of redundant dependencies like:

• code style checkers that are really helpful in development but they often report different results from version to version.
• coverage measurement, it’s pointless overhead to do this for RPM packaging.

Hence, there is a predefined filter of check dependencies %pyproject_deps_check_filter. This filter is applied automatically on %pyproject_builddeps_check or %pyproject_builddeps_metadata macro usage.

Today’s default filter includes:

• {?pypi_name:%{pep503_name %pypi_name}}
• flake8
• pytest-flake8
• isort
• black
• pytest-black
• coverage
• pytest-cov
• covdefaults
• mypy
• pytest-mypy
• pytest-checkdocs
• pre-commit

This filter can be changed like:

%set_pyproject_deps_check_filter foo bar

or

%add_pyproject_deps_check_filter foo bar

Note: this is exposed as deps –exclude option that accepts regex patterns and excludes requirement having PEP503-normalized name which matches one of these patterns. Thus, if only the name should be excluded then this name should be normalized first:

%add_pyproject_deps_check_filter %{pep503_name Foo-Project.my}

core metadata

Specification:

• https://packaging.python.org/en/latest/specifications/core-metadata/
• https://packaging.python.org/en/latest/specifications/core-metadata/#requires-dist-multiple-use

For example, upstream use core metadata as source of tests dependencies and mark them with testing extra (this means that all runtime dependencies of current package and any dependencies that are listed in the testing extra of current package will be installed, see extras). Usually this is exposed as extras = testing in tox.ini or pip install .[testing] in CI config.

• add macro to RPM specfile

  %if_with check
  %pyproject_builddeps_metadata -- --extra testing
  %endif

  This macro:

  • eval dependencies of metadata source (it’s already defined as source of runtime dependencies, otherwise it should be defined manually) in Python environment with testing extra
  • filter out dependencies according to %pyproject_deps_check_filter
  • map dependencies to distro names
  • generate RPM BuildRequires tags

• filter out extra dependencies with check-filter

pip reqfile

Specification:

• https://pip.pypa.io/en/stable/reference/requirements-file-format/#requirements-file-format

Limitations:

• supported only PEP508 requirements
• line continuations are not supported
• pip’s inline options are not supported

For example, upstream use pip’s reqfile as source of tests dependencies and it’s saved as test-requirements.txt.

• add macro to RPM specfile

  %if_with check
  %pyproject_builddeps_check
  %endif

  This macro:

  • eval dependencies of check source (it will be defined later) in Python environment
  • filter out dependencies according to %pyproject_deps_check_filter
  • map dependencies to distro names
  • generate RPM BuildRequires tags

• configure check source of dependencies by appending to RPM %prep section:

  %if_with check
  %pyproject_deps_resync check pip_reqfile test-requirements.txt
  %endif

• rerun build. The RPM build may fail on verification of check dependencies (if any). Update stored config with produced one if needed. For example,

  cp ~/hasher/chroot/usr/src/RPM/SOURCES/pyproject_deps.json .gear/
  

• filter out extra dependencies with check-filter

poetry group

Specification:

• https://python-poetry.org/docs/dependency-specification/
• https://python-poetry.org/docs/managing-dependencies/

Limitations:

• versions are not supported (poetry version format is not compatible with PEP440)
• python field is not supported

For example, upstream uses poetry dependencies group dev as source of tests dependencies. By default, poetry installs dependencies of all non-optional groups including dependencies of project. Hence, runtime should be configured first.

• add macro to RPM specfile

  %if_with check
  %pyproject_builddeps_metadata
  %pyproject_builddeps_check
  %endif

  These macros:

  • eval dependencies of metadata and check sources (metadata is already defined as source of runtime dependencies, otherwise it should be defined manually) in Python environment
  • filter out dependencies according to %pyproject_deps_check_filter
  • map dependencies to distro names
  • generate RPM BuildRequires tags

• configure check source of dependencies by appending to RPM %prep section:

  %if_with check
  %pyproject_deps_resync check poetry dev
  %endif

• rerun build. The RPM build may fail on verification of check dependencies (if any). Update stored config with produced one if needed. For example,

  cp ~/hasher/chroot/usr/src/RPM/SOURCES/pyproject_deps.json .gear/
  

• filter out extra dependencies with check-filter

tox env

Specification:

• https://tox.wiki/en/latest/config.html#deps
• config formats: https://tox.wiki/en/latest/config.html#discovery-and-file-types

Limitations:

• supported only PEP508 requirements
• config substitutions are not supported
• testenv inheritance is not supported
• pip’s req files or options are not supported

Note: if deps includes pip’s requirements file then use pip’s reqfile instead. If testenv has extras then use core metadata instead.

For example, upstream uses tox’ deps of testenv section of tox.ini config as source of tests dependencies. By default, tox installs these dependencies prior to package installation and then installs package and its runtime dependencies. Hence, runtime should be configured first.

• add macro to RPM specfile

  %if_with check
  %pyproject_builddeps_metadata
  %pyproject_builddeps_check
  %endif

  These macros:

  • eval dependencies of metadata and check sources (metadata is already defined as source of runtime dependencies, otherwise it should be defined manually) in Python environment
  • filter out dependencies according to %pyproject_deps_check_filter
  • map dependencies to distro names
  • generate RPM BuildRequires tags

• configure check source of dependencies by appending to RPM %prep section:

  %if_with check
  %pyproject_deps_resync check tox tox.ini testenv
  %endif

• rerun build. The RPM build may fail on verification of check dependencies (if any). Update stored config with produced one if needed. For example,

  cp ~/hasher/chroot/usr/src/RPM/SOURCES/pyproject_deps.json .gear/
  

• filter out extra dependencies with check-filter

Mapping project names to distro names

Python ecosystem uses PEP503 normalization rules for names comparison:

• https://packaging.python.org/en/latest/specifications/name-normalization/#package-name-normalization
• https://peps.python.org/pep-0503/#normalized-names

Mapping rule in ALTLinux:

project_name => python3-module-%{pep503_name project_name}

This almost always works out of the box, but distro packages having unnormalized names must provide such the name:

Provides: python3-module-%{pep503_name %pypi_name} = %EVR

The following naming rule for new Python distro packages is strongly encouraged:

Name: python3-module-%{pep503_name %pypi_name}