Management of Python dependencies sources: различия между версиями
Нет описания правки |
Нет описания правки |
||
(не показаны 2 промежуточные версии этого же участника) | |||
Строка 1: | Строка 1: | ||
<span id="management-of-dependencies-sources"></span> | <span id="management-of-dependencies-sources"></span> | ||
= Management of dependencies sources = | = Management of dependencies sources = | ||
''generated from rpm-build-pyproject/docs/deps.md'' | |||
<span id="overview"></span> | <span id="overview"></span> | ||
== Overview == | == Overview == | ||
This feature allows to manage runtime and/or buildtime dependencies of Python RPM package in Python-ecosystem manner for the purpose of automation (e.g. | This feature allows to manage runtime and/or buildtime dependencies of Python RPM package in Python-ecosystem manner for the purpose of automation (e.g. such dependencies can be synced, verified or evaluated). In particular, <tt><span style="color:green">%pyproject_deps_resync</span></tt>-based macros collect [https://peps.python.org/pep-0508/ PEP508] requirements from different sources and store them in <tt><span style="color:green">%_sourcedir/pyproject_deps.json</span></tt> while <tt><span style="color:green">%pyproject_builddeps</span></tt> and <tt><span style="color:green">%pyproject_runtimedeps</span></tt> evaluate dependencies specified by this config file in current Python environment and generate <tt><span style="color:green">BuildRequires</span></tt> and <tt><span style="color:green">Requires</span></tt> RPM specfile’s entries respectively. Sources of dependencies can be either well known standard formats like core metadata, PEP518, PEP517 or tool-specific formats like pip reqfile, tox deps, etc. | ||
Typical RPM specfile for Python package using this feature looks like: | Typical RPM specfile for Python package using this feature looks like: | ||
Строка 23: | Строка 25: | ||
%pyproject_deps_resync_metadata</pre> | %pyproject_deps_resync_metadata</pre> | ||
<tt><span style="color:green">%pyproject_deps</span></tt>-based RPM macros are shipped in [https://packages.altlinux.org/en/sisyphus/srpms/rpm-build-pyproject/ rpm-build-pyproject] RPM package, that can be used as drop-in replacement for [https://packages.altlinux.org/en/sisyphus/srpms/rpm-build-python3/ rpm-build-python3] in RPM specfile. | <tt><span style="color:green">%pyproject_deps</span></tt>-based RPM macros are shipped in [https://packages.altlinux.org/en/sisyphus/srpms/rpm-build-pyproject/ rpm-build-pyproject] RPM package, that can be used as drop-in replacement for [https://packages.altlinux.org/en/sisyphus/srpms/rpm-build-python3/ rpm-build-python3] in RPM specfile. | ||
<span id="bootstrap"></span> | |||
== Bootstrap == | |||
How to start using this automation. | |||
<ul> | |||
<li><p>add build dependency(pre) on <tt><span style="color:green">rpm-build-pyproject</span></tt></p> | |||
<syntaxhighlight lang="spec">BuildRequires(pre): rpm-build-pyproject</syntaxhighlight></li> | |||
<li><p>add empty configuration for dependencies sources</p> | |||
<ul> | |||
<li><p>configuration</p> | |||
<syntaxhighlight lang="sh">echo '{"sources":{}}' > .gear/pyproject_deps.json</syntaxhighlight></li> | |||
<li><p>gear rules</p> | |||
<syntaxhighlight lang="sh">echo 'copy: .gear/pyproject_deps.json' >> .gear/rules</syntaxhighlight></li> | |||
<li><p>RPM spec</p> | |||
<syntaxhighlight lang="spec">SourceN: %pyproject_deps_config_name</syntaxhighlight></li></ul> | |||
</li> | |||
<li><p>add <tt><span style="color:green">%pyproject_builddeps_build</span></tt> and <tt><span style="color:green">%pyproject_deps_resync_build</span></tt> (requires unpacked sources) RPM macros for configuring and syncing [[#build|build]] dependencies</p></li> | |||
<li><p>add <tt><span style="color:green">%pyproject_runtimedeps_metadata</span></tt> and <tt><span style="color:green">%pyproject_deps_resync_metadata</span></tt> (requires unpacked sources) RPM macros for configuring and syncing [[#runtime|runtime]] dependencies</p></li> | |||
<li><p>add <tt><span style="color:green">%pyproject_builddeps_metadata</span></tt> and specific check RPM macros for configuring and syncing [[#check|tests]] dependencies</p></li></ul> | |||
<span id="build"></span> | <span id="build"></span> | ||
Строка 101: | Строка 124: | ||
* [[#hatch-env|hatch env]] | * [[#hatch-env|hatch env]] | ||
* [[#pdm-group|pdm group]] | * [[#pdm-group|pdm group]] | ||
* [[#pipenv-category|pipenv category]] | |||
Usually tests require all [[#core-metadata|runtime]] dependencies of project ''and'' additional packages for test environment (e.g. | Usually tests require all [[#core-metadata|runtime]] dependencies of project ''and'' additional packages for test environment (e.g. pytest). Thus, configuration of several sources may be required (e.g. metadata + pip_reqfile). | ||
<span id="check-filter"></span> | <span id="check-filter"></span> | ||
Строка 112: | Строка 136: | ||
* <tt><span style="color:red">coverage</span></tt> measurement, it’s pointless overhead to do this for RPM packaging. | * <tt><span style="color:red">coverage</span></tt> measurement, it’s pointless overhead to do this for RPM packaging. | ||
* <tt><span style="color:red">publishing tools</span></tt> | * <tt><span style="color:red">publishing tools</span></tt> | ||
* <tt><span style="color:red">documentation tools</span></tt> | |||
Hence, there is a predefined filter of <tt><span style="color:green">check</span></tt> dependencies <tt><span style="color:green">%pyproject_deps_check_filter</span></tt>. This filter is applied automatically on <tt><span style="color:green">%pyproject_builddeps_check</span></tt> or <tt><span style="color:green">%pyproject_builddeps_metadata</span></tt> macro usage. | Hence, there is a predefined filter of <tt><span style="color:green">check</span></tt> dependencies <tt><span style="color:green">%pyproject_deps_check_filter</span></tt>. This filter is applied automatically on <tt><span style="color:green">%pyproject_builddeps_check</span></tt> or <tt><span style="color:green">%pyproject_builddeps_metadata</span></tt> macro usage. | ||
Строка 130: | Строка 155: | ||
* ruff and its plugins | * ruff and its plugins | ||
* typing stubs | * typing stubs | ||
* pyright and its plugins | |||
* Sphinx and its plugins | |||
This filter can be changed like: | This filter can be changed like: | ||
Строка 364: | Строка 391: | ||
<pre class="rpmspec">%if_with check | <pre class="rpmspec">%if_with check | ||
%pyproject_deps_resync YOURSOURCENAME pdm test | %pyproject_deps_resync YOURSOURCENAME pdm test | ||
%endif</pre></li> | |||
<li><p>rerun build. The RPM build may fail on verification of <tt><span style="color:green">check</span></tt> dependencies (if any). Update stored config with produced one if needed. For example,</p> | |||
<syntaxhighlight lang="sh">cp ~/hasher/chroot/usr/src/RPM/SOURCES/pyproject_deps.json .gear/</syntaxhighlight></li> | |||
<li><p>filter out extra dependencies with [[#check-filter|check-filter]]</p></li></ul> | |||
<span id="pipenv-category"></span> | |||
=== pipenv category === | |||
Specification: | |||
* https://pipenv.pypa.io/en/latest/pipfile.html | |||
* https://pipenv.pypa.io/en/latest/specifiers.html | |||
Limitations: | |||
* standalone [https://peps.python.org/pep-0508/#environment-markers PEP508 specifiers] as keys are not supported | |||
For example, upstream uses pipenv dev dependencies’ category <tt><span style="color:green">dev-packages</span></tt> as a source of tests dependencies. By default, pipenv installs dependencies of <tt><span style="color:green">packages</span></tt> category and [[#runtime|runtime dependencies]] should be configured first. | |||
<ul> | |||
<li><p>add macro to RPM specfile</p> | |||
<pre class="rpmspec">%if_with check | |||
%pyproject_builddeps_metadata | |||
%pyproject_builddeps_check | |||
%endif</pre> | |||
<p>These macros:</p> | |||
<ul> | |||
<li>eval dependencies of <tt><span style="color:green">metadata</span></tt> and <tt><span style="color:green">check</span></tt> sources (<tt><span style="color:green">metadata</span></tt> is already defined as source of [[#runtime|runtime]] dependencies, otherwise it should be defined manually) in Python environment</li> | |||
<li>filter out dependencies according to <tt><span style="color:green">%pyproject_deps_check_filter</span></tt></li> | |||
<li>map dependencies to distro names</li> | |||
<li>generate RPM <tt><span style="color:green">BuildRequires</span></tt> tags</li></ul> | |||
</li> | |||
<li><p>configure <tt><span style="color:green">check</span></tt> source of dependencies by appending to RPM <tt><span style="color:green">%prep</span></tt> section:</p> | |||
<pre class="rpmspec">%if_with check | |||
%pyproject_deps_resync_check_pipenv Pipfile dev-packages | |||
%endif</pre> | |||
<p>or source having arbitrary name</p> | |||
<pre class="rpmspec">%if_with check | |||
%pyproject_deps_resync YOURSOURCENAME pipenv Pipfile dev-packages | |||
%endif</pre></li> | %endif</pre></li> | ||
<li><p>rerun build. The RPM build may fail on verification of <tt><span style="color:green">check</span></tt> dependencies (if any). Update stored config with produced one if needed. For example,</p> | <li><p>rerun build. The RPM build may fail on verification of <tt><span style="color:green">check</span></tt> dependencies (if any). Update stored config with produced one if needed. For example,</p> |
Текущая версия от 09:43, 9 июля 2024
Management of dependencies sources
generated from rpm-build-pyproject/docs/deps.md
Overview
This feature allows to manage runtime and/or buildtime dependencies of Python RPM package in Python-ecosystem manner for the purpose of automation (e.g. such dependencies can be synced, verified or evaluated). In particular, %pyproject_deps_resync-based macros collect PEP508 requirements from different sources and store them in %_sourcedir/pyproject_deps.json while %pyproject_builddeps and %pyproject_runtimedeps evaluate dependencies specified by this config file in current Python environment and generate BuildRequires and Requires RPM specfile’s entries respectively. Sources of dependencies can be either well known standard formats like core metadata, PEP518, PEP517 or tool-specific formats like pip reqfile, tox deps, etc.
Typical RPM specfile for Python package using this feature looks like:
... Source1: %pyproject_deps_config_name %pyproject_runtimedeps_metadata BuildRequires(pre): rpm-build-pyproject %pyproject_builddeps_build %if_with check %pyproject_builddeps_metadata %endif ... %prep %setup %pyproject_deps_resync_build %pyproject_deps_resync_metadata
%pyproject_deps-based RPM macros are shipped in rpm-build-pyproject RPM package, that can be used as drop-in replacement for rpm-build-python3 in RPM specfile.
Bootstrap
How to start using this automation.
add build dependency(pre) on rpm-build-pyproject
BuildRequires(pre): rpm-build-pyproject
add empty configuration for dependencies sources
configuration
echo '{"sources":{}}' > .gear/pyproject_deps.json
gear rules
echo 'copy: .gear/pyproject_deps.json' >> .gear/rules
RPM spec
SourceN: %pyproject_deps_config_name
add %pyproject_builddeps_build and %pyproject_deps_resync_build (requires unpacked sources) RPM macros for configuring and syncing build dependencies
add %pyproject_runtimedeps_metadata and %pyproject_deps_resync_metadata (requires unpacked sources) RPM macros for configuring and syncing runtime dependencies
add %pyproject_builddeps_metadata and specific check RPM macros for configuring and syncing tests dependencies
BUILD
Build dependencies format is standardized by PEP518 and PEP517.
add %pyproject_builddeps_build macro into RPM specfile that:
- eval dependencies of PEP518 and PEP517 sources (they will be defined later) in Python environment
- map dependencies to distro names
- generate RPM BuildRequires tags
append %pyproject_deps_resync_build macro to RPM %prep section. This macro:
- configure PEP518 source of dependencies if it doesn’t exist yet
- verify PEP518 source and resync it if needed
- configure PEP517 source of dependencies if it doesn’t exist yet
- verify PEP517 source and resync it if needed
run build. The RPM build may fail on verification of PEP518 dependencies (if any). Update stored config with produced one if needed. For example,
cp ~/hasher/chroot/usr/src/RPM/SOURCES/pyproject_deps.json .gear/
%pyproject_deps-based RPM macros expect to find its configuration in RPM sources directory by default. Hence,
- add copy: .gear/pyproject_deps.json to .gear/rules
- add Source1: %pyproject_deps_config_name to RPM specfile (the source number doesn’t matter)
exclude filters can be applied on evaluation of dependencies. For example, filter out dependencies having normalized names that match types-:
%set_pyproject_deps_build_filter types-
or
%add_pyproject_deps_build_filter types-
rerun build. The RPM build may fail on verification of PEP517 dependencies (if any). Update stored config with produced one if needed. For example,
cp ~/hasher/chroot/usr/src/RPM/SOURCES/pyproject_deps.json .gear/
This is enough to manage Python build dependencies of vast of majority of Python projects.
RUNTIME
By default, runtime dependencies of Python projects in ALTLinux are generated automatically by parsing source code and fetching of all of external imports (this feature is known as autoreq). This may or may not work well. For example:
- conditional or optional dependencies can’t be automatically accounted
- installation tools like pip may expect all installed dependencies of projects
Downstream maintainers have to manually handle such cases.
Runtime dependencies format is standardized by core metadata and it’s usually generated by build backend from some tool-specific format. Produced list of dependencies is not necessary and sufficient. But it may complement auto-generated dependencies to cover unhandled cases.
add %pyproject_runtimedeps_metadata macro into RPM specfile that:
- eval dependencies of metadata source (it will be defined later) in Python environment
- map dependencies to distro names
- generate RPM Requires tags
append %pyproject_deps_resync_metadata macro to RPM %prep section. This macro:
- configure core metadata source of dependencies if it doesn’t exist yet
- verify core metadata source and resync it if needed
rerun build. The RPM build may fail on verification of metadata dependencies (if any). Update stored config with produced one if needed. For example,
cp ~/hasher/chroot/usr/src/RPM/SOURCES/pyproject_deps.json .gear/
exclude filters can be applied on evaluation of dependencies: For example, filter out dependencies having normalized names that match types-:
%set_pyproject_deps_runtime_filter types-
or
%add_pyproject_deps_runtime_filter types-
CHECK
Tests dependencies format is not standardized, but pyproject_installer supports most common of them like:
Usually tests require all runtime dependencies of project and additional packages for test environment (e.g. pytest). Thus, configuration of several sources may be required (e.g. metadata + pip_reqfile).
check filter
There can be many of redundant dependencies like:
- code style checkers that are really helpful in development but they often report different results from version to version.
- coverage measurement, it’s pointless overhead to do this for RPM packaging.
- publishing tools
- documentation tools
Hence, there is a predefined filter of check dependencies %pyproject_deps_check_filter. This filter is applied automatically on %pyproject_builddeps_check or %pyproject_builddeps_metadata macro usage.
Today’s default filter includes:
- %{?pypi_name:‘%{pep503_name %pypi_name}$’}
- flake8 and its plugins
- black and its plugins
- coverage and its plugins
- codecov and its plugins
- coveralls and its plugins
- mypy and its plugins
- isort and its plugins
- pre-commit and its plugins
- pytest-checkdocs
- twine and its plugins
- ruff and its plugins
- typing stubs
- pyright and its plugins
- Sphinx and its plugins
This filter can be changed like:
%set_pyproject_deps_check_filter foo bar
or
%add_pyproject_deps_check_filter foo bar
Note: this is exposed as deps –exclude option that accepts regex patterns and excludes requirement having PEP503-normalized name which matches one of these patterns. Thus, if only the name should be excluded then this name should be normalized first:
%add_pyproject_deps_check_filter '%{pep503_name Foo-Project.my}$'
core metadata
Specification:
- https://packaging.python.org/en/latest/specifications/core-metadata/
- https://packaging.python.org/en/latest/specifications/core-metadata/#requires-dist-multiple-use
For example, upstream use core metadata as source of tests dependencies and mark them with testing extra (this means that all runtime dependencies of current package and any dependencies that are listed in the testing extra of current package will be installed, see extras). Usually this is exposed as extras = testing in tox.ini or pip install .[testing] in CI config.
add macro to RPM specfile
%if_with check %pyproject_builddeps_metadata_extra testing %endif
This macro:
- eval dependencies of metadata source (it’s already defined as source of runtime dependencies, otherwise it should be defined manually) in Python environment with testing extra
- filter out dependencies according to %pyproject_deps_check_filter
- map dependencies to distro names
- generate RPM BuildRequires tags
filter out extra dependencies with check-filter
pip reqfile
Specification:
Limitations:
- supported only PEP508 requirements
- line continuations are not supported
- pip’s inline options are not supported
For example, upstream use pip’s reqfile as source of tests dependencies and it’s saved as test-requirements.txt.
add macro to RPM specfile
%if_with check %pyproject_builddeps_check %endif
This macro:
- eval dependencies of check source (it will be defined later) in Python environment
- filter out dependencies according to %pyproject_deps_check_filter
- map dependencies to distro names
- generate RPM BuildRequires tags
configure check source of dependencies by appending to RPM %prep section:
%if_with check %pyproject_deps_resync_check_pipreqfile test-requirements.txt %endif
or source having arbitrary name
%if_with check %pyproject_deps_resync check pip_reqfile test-requirements.txt %endif
rerun build. The RPM build may fail on verification of check dependencies (if any). Update stored config with produced one if needed. For example,
cp ~/hasher/chroot/usr/src/RPM/SOURCES/pyproject_deps.json .gear/
filter out extra dependencies with check-filter
poetry group
Specification:
- https://python-poetry.org/docs/dependency-specification/
- https://python-poetry.org/docs/managing-dependencies/
Limitations:
- versions are not supported (poetry version format is not compatible with PEP440)
- python field is not supported
For example, upstream uses poetry dependencies group dev as source of tests dependencies. By default, poetry installs dependencies of all non-optional groups including dependencies of project. Hence, runtime should be configured first.
add macro to RPM specfile
%if_with check %pyproject_builddeps_metadata %pyproject_builddeps_check %endif
These macros:
- eval dependencies of metadata and check sources (metadata is already defined as source of runtime dependencies, otherwise it should be defined manually) in Python environment
- filter out dependencies according to %pyproject_deps_check_filter
- map dependencies to distro names
- generate RPM BuildRequires tags
configure check source of dependencies by appending to RPM %prep section:
%if_with check %pyproject_deps_resync_check_poetry dev %endif
or source having arbitrary name
%if_with check %pyproject_deps_resync check poetry dev %endif
rerun build. The RPM build may fail on verification of check dependencies (if any). Update stored config with produced one if needed. For example,
cp ~/hasher/chroot/usr/src/RPM/SOURCES/pyproject_deps.json .gear/
filter out extra dependencies with check-filter
tox env
Specification:
- https://tox.wiki/en/latest/config.html#deps
- config formats: https://tox.wiki/en/latest/config.html#discovery-and-file-types
Limitations:
- supported only PEP508 requirements
- config substitutions are not supported
- testenv inheritance is not supported
- pip’s req files or options are not supported
Note: if deps includes pip’s requirements file then use pip’s reqfile instead. If testenv has extras then use core metadata instead.
For example, upstream uses tox’ deps of testenv section of tox.ini config as source of tests dependencies. By default, tox installs these dependencies prior to package installation and then installs package and its runtime dependencies. Hence, runtime should be configured first.
add macro to RPM specfile
%if_with check %pyproject_builddeps_metadata %pyproject_builddeps_check %endif
These macros:
- eval dependencies of metadata and check sources (metadata is already defined as source of runtime dependencies, otherwise it should be defined manually) in Python environment
- filter out dependencies according to %pyproject_deps_check_filter
- map dependencies to distro names
- generate RPM BuildRequires tags
configure check source of dependencies by appending to RPM %prep section:
%if_with check %pyproject_deps_resync_check_tox tox.ini testenv %endif
or source having arbitrary name
%if_with check %pyproject_deps_resync check tox tox.ini testenv %endif
rerun build. The RPM build may fail on verification of check dependencies (if any). Update stored config with produced one if needed. For example,
cp ~/hasher/chroot/usr/src/RPM/SOURCES/pyproject_deps.json .gear/
filter out extra dependencies with check-filter
hatch env
Specification:
- https://hatch.pypa.io/latest/config/environment/overview/#dependencies
- https://hatch.pypa.io/latest/intro/#configuration
Limitations:
- supported only PEP508 requirements
- env inheritance is not supported
- context formatting is not supported
- features(extra) is not supported
Note: if env has features then use core metadata with extra instead.
For example, upstream uses hatch’s dependencies of test env in hatch.toml config as source of tests dependencies. By default, hatch installs these dependencies in addition to the runtime ones. Hence, runtime should be configured first.
add macro to RPM specfile
%if_with check %pyproject_builddeps_metadata %pyproject_builddeps_check %endif
These macros:
- eval dependencies of metadata and check sources (metadata is already defined as source of runtime dependencies, otherwise it should be defined manually) in Python environment
- filter out dependencies according to %pyproject_deps_check_filter
- map dependencies to distro names
- generate RPM BuildRequires tags
configure check source of dependencies by appending to RPM %prep section:
%if_with check %pyproject_deps_resync_check_hatch hatch.toml test %endif
or source having arbitrary name
%if_with check %pyproject_deps_resync YOURSOURCENAME hatch hatch.toml test %endif
rerun build. The RPM build may fail on verification of check dependencies (if any). Update stored config with produced one if needed. For example,
cp ~/hasher/chroot/usr/src/RPM/SOURCES/pyproject_deps.json .gear/
filter out extra dependencies with check-filter
pdm group
Specification:
Limitations:
- supported only PEP508 requirements
For example, upstream uses pdm dev dependencies’ group test as source of tests dependencies. By default, pdm installs dependencies of all non-optional groups including runtime dependencies of project. Hence, runtime should be configured first.
add macro to RPM specfile
%if_with check %pyproject_builddeps_metadata %pyproject_builddeps_check %endif
These macros:
- eval dependencies of metadata and check sources (metadata is already defined as source of runtime dependencies, otherwise it should be defined manually) in Python environment
- filter out dependencies according to %pyproject_deps_check_filter
- map dependencies to distro names
- generate RPM BuildRequires tags
configure check source of dependencies by appending to RPM %prep section:
%if_with check %pyproject_deps_resync_check_pdm test %endif
or source having arbitrary name
%if_with check %pyproject_deps_resync YOURSOURCENAME pdm test %endif
rerun build. The RPM build may fail on verification of check dependencies (if any). Update stored config with produced one if needed. For example,
cp ~/hasher/chroot/usr/src/RPM/SOURCES/pyproject_deps.json .gear/
filter out extra dependencies with check-filter
pipenv category
Specification:
Limitations:
- standalone PEP508 specifiers as keys are not supported
For example, upstream uses pipenv dev dependencies’ category dev-packages as a source of tests dependencies. By default, pipenv installs dependencies of packages category and runtime dependencies should be configured first.
add macro to RPM specfile
%if_with check %pyproject_builddeps_metadata %pyproject_builddeps_check %endif
These macros:
- eval dependencies of metadata and check sources (metadata is already defined as source of runtime dependencies, otherwise it should be defined manually) in Python environment
- filter out dependencies according to %pyproject_deps_check_filter
- map dependencies to distro names
- generate RPM BuildRequires tags
configure check source of dependencies by appending to RPM %prep section:
%if_with check %pyproject_deps_resync_check_pipenv Pipfile dev-packages %endif
or source having arbitrary name
%if_with check %pyproject_deps_resync YOURSOURCENAME pipenv Pipfile dev-packages %endif
rerun build. The RPM build may fail on verification of check dependencies (if any). Update stored config with produced one if needed. For example,
cp ~/hasher/chroot/usr/src/RPM/SOURCES/pyproject_deps.json .gear/
filter out extra dependencies with check-filter
Mapping project names to distro names
Python ecosystem uses PEP503 normalization rules for names comparison:
- https://packaging.python.org/en/latest/specifications/name-normalization/#package-name-normalization
- https://peps.python.org/pep-0503/#normalized-names
Mapping rule in ALTLinux:
project_name => python3-module-%{pep503_name project_name}
This almost always works out of the box, but distro packages having unnormalized names must provide such the name:
Provides: python3-module-%{pep503_name %pypi_name} = %EVR
The following naming rule for new Python distro packages is strongly encouraged:
Name: python3-module-%{pypi_nname}
where %pypi_nname must be equal to the result of evaluation of %{pep503_name %pypi_name} macro.