4 ms·

Can "pip install" actually run code? My understanding was that it doesn't, and a package needs to be imported by a script before anything gets run.

woodruffw 3y ago
> Can "pip install" actually run code? Indeed it can, and in fact has to by necessity in many cases: `pip install` boils down to (a variant of) `python setup.py install` for many packages, where `setup.py` is package-controlled arbitrary Python code. This is also true for subcommands like `pip download`, as well as `pip install --dry-run`. (There's some subtlety here, since this only applies to source distributions and not wheels. But source distributions are still the "baseline" way to distribute Python packages.) > and a package needs to be imported by a script before anything gets run. Even with wheels, this part is unfortunately not true: a package can install a `.pth` file[1], which can be used to auto-load code on Python interpreter startup, even if the malicious package itself is never imported directly. [1]: https://docs.python.org/3/library/site.html https://docs.python.org/3/library/site.html
remexre 3y ago
I thought it runs setup.py
- dalke 3y ago
  There is a long-term goal to remove setup.py in favor of letting "a thousand flowers bloom". In modern Python setups, the pyproject.toml contains an entry for the 'build-backend'. There is a fall-back to 'setuptools.build_meta:__legacy__' because a large number of projects still use setup.py. PyPI recommends hatchling, as the example used in https://packaging.python.org/en/latest/tutorials/packaging-projects/ https://packaging.python.org/en/latest/tutorials/packaging-p... ("You can choose from a number of backends; this tutorial uses Hatchling by default, but it will work identically with setuptools, Flit, PDM, and others that support the [project] table for metadata.") These do not use setup.py. I use setup.py because the new build backends don't seem to support C extensions all that well. My package also has Cython code, and some Python-based code generation for the C code. Another package I know, RDKit, by default automatically downloads missing packages if not available. (These packages are provided as source from the respective authors.) I think distribution people really don't want the build step to be able to run arbitrary code, but they face a long, up-hill battle.
  - woodruffw 3y ago
    Just as a note: build backends still run arbitrary code. You’re exchanging a `setup.py` for an arbitrary Python package that specifies the build backend, and packages can specify their own backend (including one embedded in their own source). Python as an ecosystem will probably never be able to fully remove build-time code execution from sdists, since native extensions fundamentally rely on it. The best we can do is unify and streamline them so that as many people upload wheels as possible.
    - dalke 3y ago
      Thank you. I don't think I was clear enough that the desire really doesn't match reality, in part because I'm not clear about that myself. I don't have a deep understanding of the new way of doing things, having only read about them whilst trying to understand if/when I should migrate from setup.py. > including one embedded in their own source I didn't know about that! I thought it needed to be a pre-build dependency.

woodruffw 3y ago

> Can "pip install" actually run code? Indeed it can, and in fact has to by necessity in many cases: `pip install` boils down to (a variant of) `python setup.py install` for many packages, where `setup.py` is package-controlled arbitrary Python code. This is also true for subcommands like `pip download`, as well as `pip install --dry-run`. (There's some subtlety here, since this only applies to source distributions and not wheels. But source distributions are still the "baseline" way to distribute Python packages.) > and a package needs to be imported by a script before anything gets run. Even with wheels, this part is unfortunately not true: a package can install a `.pth` file[1], which can be used to auto-load code on Python interpreter startup, even if the malicious package itself is never imported directly. [1]: https://docs.python.org/3/library/site.html https://docs.python.org/3/library/site.html

remexre 3y ago

I thought it runs setup.py

dalke 3y ago
There is a long-term goal to remove setup.py in favor of letting "a thousand flowers bloom". In modern Python setups, the pyproject.toml contains an entry for the 'build-backend'. There is a fall-back to 'setuptools.build_meta:__legacy__' because a large number of projects still use setup.py. PyPI recommends hatchling, as the example used in https://packaging.python.org/en/latest/tutorials/packaging-projects/ https://packaging.python.org/en/latest/tutorials/packaging-p... ("You can choose from a number of backends; this tutorial uses Hatchling by default, but it will work identically with setuptools, Flit, PDM, and others that support the [project] table for metadata.") These do not use setup.py. I use setup.py because the new build backends don't seem to support C extensions all that well. My package also has Cython code, and some Python-based code generation for the C code. Another package I know, RDKit, by default automatically downloads missing packages if not available. (These packages are provided as source from the respective authors.) I think distribution people really don't want the build step to be able to run arbitrary code, but they face a long, up-hill battle.
- woodruffw 3y ago
  Just as a note: build backends still run arbitrary code. You’re exchanging a `setup.py` for an arbitrary Python package that specifies the build backend, and packages can specify their own backend (including one embedded in their own source). Python as an ecosystem will probably never be able to fully remove build-time code execution from sdists, since native extensions fundamentally rely on it. The best we can do is unify and streamline them so that as many people upload wheels as possible.
  - dalke 3y ago
    Thank you. I don't think I was clear enough that the desire really doesn't match reality, in part because I'm not clear about that myself. I don't have a deep understanding of the new way of doing things, having only read about them whilst trying to understand if/when I should migrate from setup.py. > including one embedded in their own source I didn't know about that! I thought it needed to be a pre-build dependency.