3 ms·

The metadata problem is unrelated to eggs. Eggs haven’t played much of a role in a long time but the metadata system still exists. Range requests are used by b

The metadata problem is unrelated to eggs. Eggs haven’t played much of a role in a long time but the metadata system still exists. Range requests are used by both uv and pip if the index supports it, but they have to make educated guesses about how reliable that metadata is. The main problem are local packages during development and source distributions.

PaulHoule 2y ago
Back in the case of eggs you couldn't count on having the metadata until you ran setup.py which forced pip to be unreliable because so much stuff got installed and uninstalled in the process of a build. There is a need for a complete answer for dev and private builds, I'll grant that. Private repos like we are used to in maven would help.
- the_mitsuhiko 2y ago
  Eggs did not contain setup.py files. The metadata like for wheels was embedded in the egg (in the EGG-INFO folder) from my recollection. Eggs were zip importable after all.
  - PaulHoule 2y ago
    It looks like you can embed dependency data in an egg but it is also true that (1) a lot of eggs have an internal setup.py that does things like compile C code, and (2) it's a hassle for developers on some platforms who might not have the right C installed, and (3) eggs reserve the right to decide what dependencies they include when they are installed based on the environment and such.
    - westurner 2y ago
      I decided to look into how this works these days. These days you need toml to parse pyproject.toml, and there's not a parser in the Python standard library for TOML: https://packaging.python.org/en/latest/guides/writing-pyproject-toml/ https://packaging.python.org/en/latest/guides/writing-pyproj... pip's docs strongly prefer project.toml: https://pip.pypa.io/en/stable/reference/build-system/pyproject-toml/#pyproject-toml https://pip.pypa.io/en/stable/reference/build-system/pyproje... Over setup.py's setup(,setup_requires=[], install_requires=[]) https://pip.pypa.io/en/stable/reference/build-system/setup-py/ https://pip.pypa.io/en/stable/reference/build-system/setup-p... Blaze and Bazel have Skylark/Starlark to support procedural build configuration with maintainable conditionals Bazel docs > Starlark > Differences with Python: https://bazel.build/rules/language https://bazel.build/rules/language cibuildwheel: https://github.com/pypa/cibuildwheel https://github.com/pypa/cibuildwheel ; > Builds manylinux, musllinux, macOS 10.9+ (10.13+ for Python 3.12+), and Windows wheels for CPython and PyPy; manylinux used to specify a minimum libc version for each build tag like manylinux2 or manylinux2014; pypa/manylinux: https://github.com/pypa/manylinux#manylinux https://github.com/pypa/manylinux#manylinux A manylinux_x_y wheel requires glibc>=x.y. A musllinux_x_y wheel requires musl libc>=x.y; per PEP 600: https://github.com/mayeut/pep600_compliance#distro-compatibility https://github.com/mayeut/pep600_compliance#distro-compatibi... > Works on GitHub Actions, Azure Pipelines, Travis CI, AppVeyor, CircleCI, GitLab CI, and Cirrus CI; Further software supply chain security controls: SLSA.dev provenance, Sigstore, and the new PyPI attestations storage too > Bundles shared library dependencies on Linux and macOS through `auditwheel` and `delocate` delvewheel (Windows) is similar to auditwheel (Linux) and delocate (Mac) in that it copies DLL files into the wheel: https://github.com/adang1345/delvewheel https://github.com/adang1345/delvewheel > Runs your library's tests against the wheel-installed version of your library Conda runs tests of installed packages; Conda docs > Defining metadata (meta.yaml) https://docs.conda.io/projects/conda-build/en/latest/resources/define-metadata.html#test-section https://docs.conda.io/projects/conda-build/en/latest/resourc... : > If this section exists or if there is a `run_test.[py,pl,sh,bat,r]` file in the recipe, the package is installed into a test environment after the build is finished and the tests are run there. Things that support conda meta.yml declarative package metadata: conda and anaconda, mamba and mambaforge, picomamba and emscripten-forge, pixi / uv, repo2docker REES, and probably repo2jupyterlite (because jupyterlite's jupyterlite-xeus docs mention mamba but not yet picomamba) https://jupyterlite.readthedocs.io/en/latest/howto/configure/kernels.html#jupyterlite-xeus https://jupyterlite.readthedocs.io/en/latest/howto/configure... The `setup.py test` command has been removed: https://github.com/pypa/setuptools/issues/1684 https://github.com/pypa/setuptools/issues/1684 `pip install -e .[tests]` expects extras_require['tests'] to include the same packages as the tests_require argument to setup.py: https://github.com/pypa/setuptools/issues/267 https://github.com/pypa/setuptools/issues/267 TODO: is there a new one command to run tests like `setup.py test`? `make test` works with my editor. A devcontainers.json can reference a Dockerfile that runs something like this: python -m ensurepip && python -m pip install -U pip setuptools But then still I want to run the tests of the software with one command. Are you telling me there's a way to do an HTTPS Content Range request for the toml file in a wheel for the package dependency version constraints and/or package hashes (but not GPG pubkey fingerprints to match .asc manifest signature) and the build & test commands, but you still need an additional file in addition to the TOML syntax pyproject.toml like Pipfile.lock or poetry.lock to store the hashes for each ~bdist wheel on each platform, though there's now a -c / PIP_CONSTRAINT option to specify an additional requirements.txt but that doesn't solve for windows or mac only requirements in a declarative requirements.txt? https://pip.pypa.io/en/stable/user_guide/#constraints-files https://pip.pypa.io/en/stable/user_guide/#constraints-files conda supports putting `[win]` at the end of a YAML list item if it's for windows only. Re: optimizing builds for conda-forge (and PyPI (though PyPI doesn't build packages (when there's a new PR, and then sign each build for each platform))) https://news.ycombinator.com/item?id=41306658 https://news.ycombinator.com/item?id=41306658

PaulHoule 2y ago

Back in the case of eggs you couldn't count on having the metadata until you ran setup.py which forced pip to be unreliable because so much stuff got installed and uninstalled in the process of a build. There is a need for a complete answer for dev and private builds, I'll grant that. Private repos like we are used to in maven would help.

the_mitsuhiko 2y ago
Eggs did not contain setup.py files. The metadata like for wheels was embedded in the egg (in the EGG-INFO folder) from my recollection. Eggs were zip importable after all.
- PaulHoule 2y ago
  It looks like you can embed dependency data in an egg but it is also true that (1) a lot of eggs have an internal setup.py that does things like compile C code, and (2) it's a hassle for developers on some platforms who might not have the right C installed, and (3) eggs reserve the right to decide what dependencies they include when they are installed based on the environment and such.
  - westurner 2y ago
    I decided to look into how this works these days. These days you need toml to parse pyproject.toml, and there's not a parser in the Python standard library for TOML: https://packaging.python.org/en/latest/guides/writing-pyproject-toml/ https://packaging.python.org/en/latest/guides/writing-pyproj... pip's docs strongly prefer project.toml: https://pip.pypa.io/en/stable/reference/build-system/pyproject-toml/#pyproject-toml https://pip.pypa.io/en/stable/reference/build-system/pyproje... Over setup.py's setup(,setup_requires=[], install_requires=[]) https://pip.pypa.io/en/stable/reference/build-system/setup-py/ https://pip.pypa.io/en/stable/reference/build-system/setup-p... Blaze and Bazel have Skylark/Starlark to support procedural build configuration with maintainable conditionals Bazel docs > Starlark > Differences with Python: https://bazel.build/rules/language https://bazel.build/rules/language cibuildwheel: https://github.com/pypa/cibuildwheel https://github.com/pypa/cibuildwheel ; > Builds manylinux, musllinux, macOS 10.9+ (10.13+ for Python 3.12+), and Windows wheels for CPython and PyPy; manylinux used to specify a minimum libc version for each build tag like manylinux2 or manylinux2014; pypa/manylinux: https://github.com/pypa/manylinux#manylinux https://github.com/pypa/manylinux#manylinux A manylinux_x_y wheel requires glibc>=x.y. A musllinux_x_y wheel requires musl libc>=x.y; per PEP 600: https://github.com/mayeut/pep600_compliance#distro-compatibility https://github.com/mayeut/pep600_compliance#distro-compatibi... > Works on GitHub Actions, Azure Pipelines, Travis CI, AppVeyor, CircleCI, GitLab CI, and Cirrus CI; Further software supply chain security controls: SLSA.dev provenance, Sigstore, and the new PyPI attestations storage too > Bundles shared library dependencies on Linux and macOS through `auditwheel` and `delocate` delvewheel (Windows) is similar to auditwheel (Linux) and delocate (Mac) in that it copies DLL files into the wheel: https://github.com/adang1345/delvewheel https://github.com/adang1345/delvewheel > Runs your library's tests against the wheel-installed version of your library Conda runs tests of installed packages; Conda docs > Defining metadata (meta.yaml) https://docs.conda.io/projects/conda-build/en/latest/resources/define-metadata.html#test-section https://docs.conda.io/projects/conda-build/en/latest/resourc... : > If this section exists or if there is a `run_test.[py,pl,sh,bat,r]` file in the recipe, the package is installed into a test environment after the build is finished and the tests are run there. Things that support conda meta.yml declarative package metadata: conda and anaconda, mamba and mambaforge, picomamba and emscripten-forge, pixi / uv, repo2docker REES, and probably repo2jupyterlite (because jupyterlite's jupyterlite-xeus docs mention mamba but not yet picomamba) https://jupyterlite.readthedocs.io/en/latest/howto/configure/kernels.html#jupyterlite-xeus https://jupyterlite.readthedocs.io/en/latest/howto/configure... The `setup.py test` command has been removed: https://github.com/pypa/setuptools/issues/1684 https://github.com/pypa/setuptools/issues/1684 `pip install -e .[tests]` expects extras_require['tests'] to include the same packages as the tests_require argument to setup.py: https://github.com/pypa/setuptools/issues/267 https://github.com/pypa/setuptools/issues/267 TODO: is there a new one command to run tests like `setup.py test`? `make test` works with my editor. A devcontainers.json can reference a Dockerfile that runs something like this: python -m ensurepip && python -m pip install -U pip setuptools But then still I want to run the tests of the software with one command. Are you telling me there's a way to do an HTTPS Content Range request for the toml file in a wheel for the package dependency version constraints and/or package hashes (but not GPG pubkey fingerprints to match .asc manifest signature) and the build & test commands, but you still need an additional file in addition to the TOML syntax pyproject.toml like Pipfile.lock or poetry.lock to store the hashes for each ~bdist wheel on each platform, though there's now a -c / PIP_CONSTRAINT option to specify an additional requirements.txt but that doesn't solve for windows or mac only requirements in a declarative requirements.txt? https://pip.pypa.io/en/stable/user_guide/#constraints-files https://pip.pypa.io/en/stable/user_guide/#constraints-files conda supports putting `[win]` at the end of a YAML list item if it's for windows only. Re: optimizing builds for conda-forge (and PyPI (though PyPI doesn't build packages (when there's a new PR, and then sign each build for each platform))) https://news.ycombinator.com/item?id=41306658 https://news.ycombinator.com/item?id=41306658