4 ms·

Latest Setuptools broke a bunch of packages

2 points by todsacerdoti 2y ago

zahlman 2y ago
> Removing a configuration format deprecated in 2021 surely won't cause any issues right? Of course not. This is, after all, the language where people were given an extra five years to migrate away from 2.7 and still complained for years after the extended date. By the way, being cowed into supporting all this outdated stuff has a lot to do with why the vanilla tools (Pip and Setuptools) are slow and bulky and why the design and implementation of better ideas takes so long. Everyone gets into arguments about what's a big enough change to actually matter, vs. what's small enough to let the laggards adapt; and about how to ensure there are migration paths. Check out PEP 777 (https://peps.python.org/pep-0777/ https://peps.python.org/pep-0777/) and the corresponding discussion (https://discuss.python.org/t/_/67484 https://discuss.python.org/t/_/67484) for just a small sample of how deep the meta-discussion gets.
zahlman 2y ago
Some more thoughts after digging in a bit further: * At first blush, the "breaks the Internet" title on Reddit seems quite histrionic. The initial bug report against Setuptools referenced a project (https://github.com/tomoh1r/ansible-vault https://github.com/tomoh1r/ansible-vault) that has only a few dozen starts on GitHub and only a few thousand daily downloads from PyPI (https://pypistats.org/packages/ansible-vault https://pypistats.org/packages/ansible-vault) (compare to tens of millions for the most popular packages). [Removed a point about semver that was not well considered. It turns out that Setuptools version 78.0.0 never actually got released, despite being listed separately in the version history; the 78.0.1 patch was applied first.] * Projects that try to follow modern packaging standards - and pay attention to the basics - seem to run into far fewer problems overall. In particular, ansible-vault is using its pyproject.toml only for tool config, and relies on both a setup.cfg and setup.py. Developers and maintainers in 2021 should have been well aware of standards like https://peps.python.org/pep-0621/ https://peps.python.org/pep-0621/, and have heard of the problems caused by setup.py-based metadata. The code is 100% Python (even Ansible itself is, as far as I can tell) so a setup.py is completely unnecessary. (Ansible includes a "dummy" setup.py "for compatibility with older tools", but the ecosystem really is supposed to have moved on from that by now.) * But none of this should have been a problem for the OP anyway, because - being 100% Python - the ansible-vault package should have been distributing a wheel (https://pradyunsg.me/blog/2022/12/31/wheels-are-faster-pure-python/ https://pradyunsg.me/blog/2022/12/31/wheels-are-faster-pure-...), but doesn't (https://pypi.org/project/ansible-vault/#files https://pypi.org/project/ansible-vault/#files). (Yes, I did notice that Mr. Gedam published that article at the end of 2022. Education efforts about Python packaging around 2020-2021 were apparently quite lacking.) * The purpose of 78.0.1 was to "temporarily remove Requests from integration tests" (https://github.com/pypa/setuptools/pull/4909 https://github.com/pypa/setuptools/pull/4909). In other words, they knew that this would break the build for Requests. Fortunately, Requests does follow good practice and publish a wheel, or there would have indeed been a problem worthy of a sensationalized title. (It's one of the most popular packages out there, and the stats don't even count the vendored copies included in Pip.) * Requests has been bitten by a Setuptools removal before, last July (https://github.com/psf/requests/issues/6775 https://github.com/psf/requests/issues/6775). That time, the removal was the only reason for bumping the version number, and then it got promptly restored in 72.1.0 anyway. This time, there was only a bugfix besides the one removal in 78.0.0, and now it's been undone in 78.0.2. (So undoing a removal is sometimes a minor update and sometimes a patch, interesting...) * When that problem occurred in July, I proposed that the packaging system should be modernized - and was told by a Requests maintainer that this was already planned since the previous release last May, to appear in the next minor version (2.33). But it's been almost a year and that release hasn't appeared yet. (I have to wonder what else they're planning to include...) * The history of the decision for the original deprecation (https://github.com/pypa/setuptools/issues/1608 https://github.com/pypa/setuptools/issues/1608) is also rather interesting.

2 points by todsacerdoti 2y ago

> Removing a configuration format deprecated in 2021 surely won't cause any issues right? Of course not. This is, after all, the language where people were given an extra five years to migrate away from 2.7 and still complained for years after the extended date. By the way, being cowed into supporting all this outdated stuff has a lot to do with why the vanilla tools (Pip and Setuptools) are slow and bulky and why the design and implementation of better ideas takes so long. Everyone gets into arguments about what's a big enough change to actually matter, vs. what's small enough to let the laggards adapt; and about how to ensure there are migration paths. Check out PEP 777 (https://peps.python.org/pep-0777/ https://peps.python.org/pep-0777/) and the corresponding discussion (https://discuss.python.org/t/_/67484 https://discuss.python.org/t/_/67484) for just a small sample of how deep the meta-discussion gets.

zahlman 2y ago

Some more thoughts after digging in a bit further: * At first blush, the "breaks the Internet" title on Reddit seems quite histrionic. The initial bug report against Setuptools referenced a project (https://github.com/tomoh1r/ansible-vault https://github.com/tomoh1r/ansible-vault) that has only a few dozen starts on GitHub and only a few thousand daily downloads from PyPI (https://pypistats.org/packages/ansible-vault https://pypistats.org/packages/ansible-vault) (compare to tens of millions for the most popular packages). [Removed a point about semver that was not well considered. It turns out that Setuptools version 78.0.0 never actually got released, despite being listed separately in the version history; the 78.0.1 patch was applied first.] * Projects that try to follow modern packaging standards - and pay attention to the basics - seem to run into far fewer problems overall. In particular, ansible-vault is using its pyproject.toml only for tool config, and relies on both a setup.cfg and setup.py. Developers and maintainers in 2021 should have been well aware of standards like https://peps.python.org/pep-0621/ https://peps.python.org/pep-0621/, and have heard of the problems caused by setup.py-based metadata. The code is 100% Python (even Ansible itself is, as far as I can tell) so a setup.py is completely unnecessary. (Ansible includes a "dummy" setup.py "for compatibility with older tools", but the ecosystem really is supposed to have moved on from that by now.) * But none of this should have been a problem for the OP anyway, because - being 100% Python - the ansible-vault package should have been distributing a wheel (https://pradyunsg.me/blog/2022/12/31/wheels-are-faster-pure-python/ https://pradyunsg.me/blog/2022/12/31/wheels-are-faster-pure-...), but doesn't (https://pypi.org/project/ansible-vault/#files https://pypi.org/project/ansible-vault/#files). (Yes, I did notice that Mr. Gedam published that article at the end of 2022. Education efforts about Python packaging around 2020-2021 were apparently quite lacking.) * The purpose of 78.0.1 was to "temporarily remove Requests from integration tests" (https://github.com/pypa/setuptools/pull/4909 https://github.com/pypa/setuptools/pull/4909). In other words, they knew that this would break the build for Requests. Fortunately, Requests does follow good practice and publish a wheel, or there would have indeed been a problem worthy of a sensationalized title. (It's one of the most popular packages out there, and the stats don't even count the vendored copies included in Pip.) * Requests has been bitten by a Setuptools removal before, last July (https://github.com/psf/requests/issues/6775 https://github.com/psf/requests/issues/6775). That time, the removal was the only reason for bumping the version number, and then it got promptly restored in 72.1.0 anyway. This time, there was only a bugfix besides the one removal in 78.0.0, and now it's been undone in 78.0.2. (So undoing a removal is sometimes a minor update and sometimes a patch, interesting...) * When that problem occurred in July, I proposed that the packaging system should be modernized - and was told by a Requests maintainer that this was already planned since the previous release last May, to appear in the next minor version (2.33). But it's been almost a year and that release hasn't appeared yet. (I have to wonder what else they're planning to include...) * The history of the decision for the original deprecation (https://github.com/pypa/setuptools/issues/1608 https://github.com/pypa/setuptools/issues/1608) is also rather interesting.