3 ms·

have always wondered why pypi doesn't generate whl files for pure-python sdists and why companies like travis / github aren't more active in language-level pac

have always wondered why pypi doesn't generate whl files for pure-python sdists and why companies like travis / github aren't more active in language-level packaging work github gives away so much free docker time -- faster installation would save them money directly

X-Istence 6y ago
Because there is no way to know if something is pure-python or not. Even more so with pep517 and being able to use different build systems.
- awinter-py 6y ago
  wait then how does bdist_wheel know to make platform-dependent vs platform-independent whl files?
  - icegreentea2 6y ago
    Your hunch is right, setuptools is able to determine if it's pure python or not (https://packaging.python.org/guides/distributing-packages-using-setuptools/#pure-python-wheels https://packaging.python.org/guides/distributing-packages-us...) Maybe it's not reliable enough? In any case, I can see why PyPI wouldn't want to increase the scope of their work. Sometimes is super valuable to just be good at one thing (distribute what others give you).
    - wirrbel 6y ago
      I think the pypi folks just don't have the resources / capacity to implement this. With appropriate funding this could have been done (development efforts wouldn't be that big, but probably operational costs for the build farms). In the past I ran (devpi builder)[https://github.com/blue-yonder/devpi-builder/ https://github.com/blue-yonder/devpi-builder/] with (devpi)[https://pypi.org/project/devpi/ https://pypi.org/project/devpi/] to build wheels for all my dependencies and keep them stored in a place I have control over.
  - X-Istence 6y ago
    That's a specific implementation that relies on setuptools. An sdist doesn't have to use setuptools to be an sdist. Wheels can be generated out of thin air, there is no requirement that setuptools or any tool is ever involved, if you wanted you could zip up a particular directory structure and it is a wheel.
    - awinter-py 6y ago
      are you saying pure-python detection is possible for standard packaging formats like setuptools and pyproject.toml, but not for sdist generally?
      - X-Istence 6y ago
        pyproject.toml specifies how to build a project using a builder. You'd have to execute the builder and let it build the project to know if the wheel it generates is pure python or not. As for setuptools: There's a class that setuptools relies on to do the actual building of the Python project, called distutils.dist.Distribution which has a function named is_pure(): https://github.com/python/cpython/blob/e42b705188271da108de42b55d9344642170aa2b/Lib/distutils/dist.py#L1011 https://github.com/python/cpython/blob/e42b705188271da108de4... Which makes sure that it doesn't have any known external modules or c libraries. So to do detection of whether an sdist is a pure python, you'd have to execute setuptools and setup.py because of course setup.py isn't just configuration, it's full on Python. So just looking at an sdist statically you can not know if the sdist is a pure python project or not.
wirrbel 6y ago
underrated comment. pypi could probably generate a lot of wheels. In fact, `pip` generates wheels from packages it downloades before installing them, so wheels can be generated for a lot of packages already directly from sdist. Why it isn't done already? Good question. I guess it has to do with a majority of python packages installing just fine from source (those that aren't a problem for packaging anyway), and then those packages that need wheels because they use native extensions kind of are high-maintenance to bundle and roll out, especially if upstream maintainers haven't invested in building their projects for that. I would love to see auto-bundled wheels. but I guess for this to fly, package maintainers would need to start standardizing a bit better (that also involves tooling). Easy things are missing, for example in setuptools you cannot even define smoke tests that distributors could use to verify successful bundling.

X-Istence 6y ago

Because there is no way to know if something is pure-python or not. Even more so with pep517 and being able to use different build systems.

awinter-py 6y ago
wait then how does bdist_wheel know to make platform-dependent vs platform-independent whl files?
- icegreentea2 6y ago
  Your hunch is right, setuptools is able to determine if it's pure python or not (https://packaging.python.org/guides/distributing-packages-using-setuptools/#pure-python-wheels https://packaging.python.org/guides/distributing-packages-us...) Maybe it's not reliable enough? In any case, I can see why PyPI wouldn't want to increase the scope of their work. Sometimes is super valuable to just be good at one thing (distribute what others give you).
  - wirrbel 6y ago
    I think the pypi folks just don't have the resources / capacity to implement this. With appropriate funding this could have been done (development efforts wouldn't be that big, but probably operational costs for the build farms). In the past I ran (devpi builder)[https://github.com/blue-yonder/devpi-builder/ https://github.com/blue-yonder/devpi-builder/] with (devpi)[https://pypi.org/project/devpi/ https://pypi.org/project/devpi/] to build wheels for all my dependencies and keep them stored in a place I have control over.
- X-Istence 6y ago
  That's a specific implementation that relies on setuptools. An sdist doesn't have to use setuptools to be an sdist. Wheels can be generated out of thin air, there is no requirement that setuptools or any tool is ever involved, if you wanted you could zip up a particular directory structure and it is a wheel.
  - awinter-py 6y ago
    are you saying pure-python detection is possible for standard packaging formats like setuptools and pyproject.toml, but not for sdist generally?
    - X-Istence 6y ago
      pyproject.toml specifies how to build a project using a builder. You'd have to execute the builder and let it build the project to know if the wheel it generates is pure python or not. As for setuptools: There's a class that setuptools relies on to do the actual building of the Python project, called distutils.dist.Distribution which has a function named is_pure(): https://github.com/python/cpython/blob/e42b705188271da108de42b55d9344642170aa2b/Lib/distutils/dist.py#L1011 https://github.com/python/cpython/blob/e42b705188271da108de4... Which makes sure that it doesn't have any known external modules or c libraries. So to do detection of whether an sdist is a pure python, you'd have to execute setuptools and setup.py because of course setup.py isn't just configuration, it's full on Python. So just looking at an sdist statically you can not know if the sdist is a pure python project or not.

wirrbel 6y ago

underrated comment. pypi could probably generate a lot of wheels. In fact, `pip` generates wheels from packages it downloades before installing them, so wheels can be generated for a lot of packages already directly from sdist. Why it isn't done already? Good question. I guess it has to do with a majority of python packages installing just fine from source (those that aren't a problem for packaging anyway), and then those packages that need wheels because they use native extensions kind of are high-maintenance to bundle and roll out, especially if upstream maintainers haven't invested in building their projects for that. I would love to see auto-bundled wheels. but I guess for this to fly, package maintainers would need to start standardizing a bit better (that also involves tooling). Easy things are missing, for example in setuptools you cannot even define smoke tests that distributors could use to verify successful bundling.