9 ms·

For improvements I commented: Remove setup.py files and mandate wheels. This is the root cause of a lot of the evil in the ecosystem. Next on the list would be

by orf 4y ago

For improvements I commented: Remove setup.py files and mandate wheels. This is the root cause of a lot of the evil in the ecosystem. Next on the list would be pypi namespaces, but there are good reasons why that is very hard. The mission statement they are proposing, “a packaging ecosystem for all”, completely misses the mark. How about a “packaging ecosystem that works” first? I spent a bunch of time recently fixing our internal packaging repo (nexus) because the switch from md5 hashes to sha256 hashes broke everything, and re-locking a bajillion lock files would take literally months of man hours time. I’ve been a Python user for the last 17 years, so I’m sympathetic of how we got to the current situation and aware that we’ve actually come quite far. But every time I use Cargo I am insanely jealous, impressed and sad that we don’t have something like it. Poetry is closest, but it’s a far cry.

kevin_thibedeau 4y ago
Setup.py can do things wheels can't. Most notably it's the only installation method that can invoke 2to3 at runtime without requiring a dev to create multiple packages.
- orf 4y ago
  It’s lucky Python 2 isn’t supported anymore then, and everyone has had like a decade to run 2to3 once and publish a package for Python 3, so that use case becomes meaningless.
  - coldtea 4y ago
    You'd be surprised at how many billions lines of production code are still at 2 (and could not care less whether it's end-of-lined)
    - orf 4y ago
      I'm not surprised at all, but regardless they also should not be similarly surprised if people could not care less about that use case.
  - mistrial9 4y ago
    very unfortunately the direct burden of python2 is placed on the packagers.. users of Python 2 like their libs (me) and have no horse in this demonization campaign
    - orf 4y ago
      Pay for support for Python 2 then? At which point it’s a burden on the person you are paying. Or don't, in which case you're complaining that people are demonizing you because they are not doing your work for free?
      - mistrial9 4y ago
        this is battle-fatigue in action! I did not complain, in fact I am faced with the serious burden that is placed on packagers of python, on a regular basis, and have put many cycles of thought into it.. It is obvious that packagers and language designers are firmly in one end of a sort of spectrum.. while many users, perhaps engineering managers in production with sunk costs, are firmly at the other.. with many in between .. more could be said.. no one is an enemy on this, it is complicated to solve it with many tradeoffs and moving parts.. evidence in the topic today.
korijn 4y ago
...and ensure _all_ package metadata required to perform dependency resolution can be retrieved through an API (in other words without downloading wheels).
- orf 4y ago
  Yeah, that’s sort of what I meant by my suggestion. Requirements that can only be resolved by downloading and executing code is a huge burden on tooling
  - korijn 4y ago
    Yeah. Well, mandating wheels and getting rid of setup.py at least avoids having to run scripts, and indeed enables the next step which would be indexing all the metadata and exposing it through an API. I just thought it wouldn't necessarily be obvious to all readers of your comment.
    - orf 4y ago
      Just to be clear, package metadata already is sort of available through the pypi json api. I've got the entire set of all package metadata here: https://github.com/orf/pypi-data https://github.com/orf/pypi-data $ gzcat release_data/c/d/cdklabs.cdk-hyperledger-fabric-network.json.gz | jq '. | to_entries | .[].value.info.requires_dist' | head [ "typeguard (~=2.13.3)", "publication (>=0.0.3)", "jsii (<2.0.0,>=1.63.2)", "constructs (<11.0.0,>=10.0.5)", "aws-cdk-lib (<3.0.0,>=2.33.0)" ] It's just not everything has it, and there isn't a way to differenciate between "missing" and "no dependencies". And it's also only for the `dist` releases. But anyway, poetry uses this information during dependency resolution.
      - korijn 4y ago
        I'm aware! The issue is the mixed content (wheels, and... not wheels) on pypi. If the data is incomplete, it's useless in the sense that you're never going to be able to guarantee good results.
  - 4y ago
ziml77 4y ago
I tend to just give up on a package if it requires a C toolchain to install. Even if I do end up getting things set up in a way that the library's build script is happy with, I'll be inflicting pain on anyone else who then tries to work with my code.
- deleted 4y ago
  [deleted]
- tux3 4y ago
  It feels so suboptimal to need the C toolchain to do things, but having no solid way to depend on it as a non-C library (especially annoying in Rust, which insists on building everything from source and never installing libraries globally). I make a tool/library that requires the C toolchain at runtime. That's even worse than build time, I need end users to have things like lld, objdump, ranlib, etc installed anywhere they use it. My options are essentially: - Requiring users to just figure it out with their system package manager - Building the C toolchain from source at build time and statically linking it (so you get to spend an hour or two recompiling all of LLVM each time you update or clear your package cache! Awesome!), - Building just LLD/objdump/.. at build-time (but user still need to install LLVM. So you get both slow installs AND have to deal with finding a compatible copy of libLLVM), - Pre-compiling all the C tools and putting them in a storage bucket somewhere, for all architectures and all OS versions. But then not have support when things like the M1 or new OS versions right away, or people on uncommon OSes. And now need to maintain a build machine for all of these myself. - Pre-compile the whole C toolchain to WASM, build Wasmtime from source instead, and just eat the cost of Cranelift running LLVM 5-10x slower than natively... I keep trying to work around the C toolchain, but I still can't see any very good solution that doesn't make my users have extra problems one way or another. Hey RiiR evangelism people, anyone want to tackle all of LLVM? .. no? No one? :)
  - gwenzek 4y ago
    I feel Zig could help here. The binaries ship with LLVM statically linked. You could rely on them to provide binaries for a variety of architecture / OS, and use it to compile code on the target machine. I'll probably explore this at some point for Pip.
dalke 4y ago
> Remove setup.py files and mandate wheels What alternative is there for me? My package has a combination of hand-built C extensions and Cython extensions, as well as a code generation step during compilation. These are handled through a subclass of setuptools.command.build_ext.build_ext. Furthermore, I have compile-time options to enable/disable certain configuration options, like enabling/disabling support for OpenMP, via environment variables so they can be passed through from pip. OpenMP is a compile-time option because the default C compiler on macOS doesn't include OpenMP. You need to install it, using one of various approaches. Which is why I only have a source distribution for macOS, along with a description of the approaches. I have not found a non-setup.py way to handle my configuration, nor to provide macOS wheels. Even for the Linux wheels, I have to patch the manylinux Docker container to whitelist libomp (the OpenMP library), using something like this: RUN perl -i -pe 's/"libresolv.so.2"/"libresolv.so.2", "libgomp.so.1"/' /opt/_internal/pipx/venvs/auditwheel/lib/python3.9/site-packages/ auditwheel/policy/manylinux-policy.json Oh, and if compiling where platform.machine() == "arm64" then I need to not add the AVX2 compiler flag. The non-setup.py packaging systems I've looked at are for Python-only code bases. Or, if I understand things correctly, I'm supposed to make a new specialized package which implements PEP 518, which I can then use to boot-strap my code. Except, that's still going to use effectively arbitrary code during the compilation step (to run the codegen) and still use setup.py to build the extension. So it's not like the evil disappears.
- deleted 4y ago
  [deleted]
- orf 4y ago
  To be clear, I’m not suggesting we remove the ability to compile native extensions. I’m suggesting we find a better way to build them, something a bit more structured, and decouple that specific use case from setup.py. It would be cool to be able to structure this in a way that means I can describe what system libraries I may need without having to execute setup.py and find out, and express compile time flags or options in a structured way. Think of it like cargo.toml va build.rs.
  - dalke 4y ago
progval 4y ago
> For improvements I commented: Remove setup.py files and mandate wheels. This would make most C extensions impossible to install on anything other than x86_64-pc-linux-gnu (or arm-linux-gnueabihf/aarch64-linux-gnu if you are lucky) because developers don't want to bother building wheels for them.
- mathstuf 4y ago
  I think it'd make other things impossible too. One project I help maintain is C++ and is mainly so. It optionally has Python bindings. It also has something like 150 options to the build that affect things. There is zero chance of me ever attempting to make `setup.py` any kind of sensible "entry point" to the build. Instead, the build detects "oh, you want a wheel" and generates `setup.py` to just grab what the C++ build then drops into a place where `build_ext` or whatever expects them to be using some fun globs. It also fills in "features" or whatever the post-name `[name]` stuff is called so you can do some kind of post-build "ok, it has a feature I need" inspection.
- urschrei 4y ago
  cibuildwheel (which is an official, supported tool) has made this enormously easier. I test and generate wheels with a compiled (Rust! Because of course) extension using a Cython bridge for all supported Python versions for 32-bit and 64-bit Windows, macOS x86_64 and arm64, and whatever manylinux is calling itself this week. No user compilation required. It took about half a day to set up, and is extremely well documented.
blibble 4y ago
> The mission statement they are proposing, “a packaging ecosystem for all”, completely misses the mark. How about a “packaging ecosystem that works” first? I think at the point a programming language is going on about "mission statements" for a packaging tool, you know they've lost the plot copy Maven from 2004 (possibly with less XML) that's it, problem solved
baggiponte 4y ago
I recommend PDM over poetry!
x3n0ph3n3 4y ago
I suggested "one packaging system to rule them all." The fragmentation in this space is frustrating.
cozzyd 4y ago
Many "Python" packages include native code in some form either as bindings or to workaround Python being agonizingly slow. Which means you often need to call make, or cmake or some other build system anyway... unless you want to build wheels for every possible configuration a user might have (which is virtually impossible, considering every combination of OS, architecture, debug options, etc. you may want to support). Plus you need a build system to build the wheels anyway...

by orf 4y ago

kevin_thibedeau 4y ago

Setup.py can do things wheels can't. Most notably it's the only installation method that can invoke 2to3 at runtime without requiring a dev to create multiple packages.

orf 4y ago
It’s lucky Python 2 isn’t supported anymore then, and everyone has had like a decade to run 2to3 once and publish a package for Python 3, so that use case becomes meaningless.
- coldtea 4y ago
  You'd be surprised at how many billions lines of production code are still at 2 (and could not care less whether it's end-of-lined)
  - orf 4y ago
    I'm not surprised at all, but regardless they also should not be similarly surprised if people could not care less about that use case.
- mistrial9 4y ago
  very unfortunately the direct burden of python2 is placed on the packagers.. users of Python 2 like their libs (me) and have no horse in this demonization campaign
  - orf 4y ago
    Pay for support for Python 2 then? At which point it’s a burden on the person you are paying. Or don't, in which case you're complaining that people are demonizing you because they are not doing your work for free?
    - mistrial9 4y ago
      this is battle-fatigue in action! I did not complain, in fact I am faced with the serious burden that is placed on packagers of python, on a regular basis, and have put many cycles of thought into it.. It is obvious that packagers and language designers are firmly in one end of a sort of spectrum.. while many users, perhaps engineering managers in production with sunk costs, are firmly at the other.. with many in between .. more could be said.. no one is an enemy on this, it is complicated to solve it with many tradeoffs and moving parts.. evidence in the topic today.

korijn 4y ago

...and ensure _all_ package metadata required to perform dependency resolution can be retrieved through an API (in other words without downloading wheels).

orf 4y ago
Yeah, that’s sort of what I meant by my suggestion. Requirements that can only be resolved by downloading and executing code is a huge burden on tooling
- korijn 4y ago
  Yeah. Well, mandating wheels and getting rid of setup.py at least avoids having to run scripts, and indeed enables the next step which would be indexing all the metadata and exposing it through an API. I just thought it wouldn't necessarily be obvious to all readers of your comment.
  - orf 4y ago
    Just to be clear, package metadata already is sort of available through the pypi json api. I've got the entire set of all package metadata here: https://github.com/orf/pypi-data https://github.com/orf/pypi-data $ gzcat release_data/c/d/cdklabs.cdk-hyperledger-fabric-network.json.gz | jq '. | to_entries | .[].value.info.requires_dist' | head [ "typeguard (~=2.13.3)", "publication (>=0.0.3)", "jsii (<2.0.0,>=1.63.2)", "constructs (<11.0.0,>=10.0.5)", "aws-cdk-lib (<3.0.0,>=2.33.0)" ] It's just not everything has it, and there isn't a way to differenciate between "missing" and "no dependencies". And it's also only for the `dist` releases. But anyway, poetry uses this information during dependency resolution.
    - korijn 4y ago
      I'm aware! The issue is the mixed content (wheels, and... not wheels) on pypi. If the data is incomplete, it's useless in the sense that you're never going to be able to guarantee good results.
- 4y ago

ziml77 4y ago

I tend to just give up on a package if it requires a C toolchain to install. Even if I do end up getting things set up in a way that the library's build script is happy with, I'll be inflicting pain on anyone else who then tries to work with my code.

deleted 4y ago
[deleted]
tux3 4y ago
It feels so suboptimal to need the C toolchain to do things, but having no solid way to depend on it as a non-C library (especially annoying in Rust, which insists on building everything from source and never installing libraries globally). I make a tool/library that requires the C toolchain at runtime. That's even worse than build time, I need end users to have things like lld, objdump, ranlib, etc installed anywhere they use it. My options are essentially: - Requiring users to just figure it out with their system package manager - Building the C toolchain from source at build time and statically linking it (so you get to spend an hour or two recompiling all of LLVM each time you update or clear your package cache! Awesome!), - Building just LLD/objdump/.. at build-time (but user still need to install LLVM. So you get both slow installs AND have to deal with finding a compatible copy of libLLVM), - Pre-compiling all the C tools and putting them in a storage bucket somewhere, for all architectures and all OS versions. But then not have support when things like the M1 or new OS versions right away, or people on uncommon OSes. And now need to maintain a build machine for all of these myself. - Pre-compile the whole C toolchain to WASM, build Wasmtime from source instead, and just eat the cost of Cranelift running LLVM 5-10x slower than natively... I keep trying to work around the C toolchain, but I still can't see any very good solution that doesn't make my users have extra problems one way or another. Hey RiiR evangelism people, anyone want to tackle all of LLVM? .. no? No one? :)
- gwenzek 4y ago
  I feel Zig could help here. The binaries ship with LLVM statically linked. You could rely on them to provide binaries for a variety of architecture / OS, and use it to compile code on the target machine. I'll probably explore this at some point for Pip.

dalke 4y ago

> Remove setup.py files and mandate wheels What alternative is there for me? My package has a combination of hand-built C extensions and Cython extensions, as well as a code generation step during compilation. These are handled through a subclass of setuptools.command.build_ext.build_ext. Furthermore, I have compile-time options to enable/disable certain configuration options, like enabling/disabling support for OpenMP, via environment variables so they can be passed through from pip. OpenMP is a compile-time option because the default C compiler on macOS doesn't include OpenMP. You need to install it, using one of various approaches. Which is why I only have a source distribution for macOS, along with a description of the approaches. I have not found a non-setup.py way to handle my configuration, nor to provide macOS wheels. Even for the Linux wheels, I have to patch the manylinux Docker container to whitelist libomp (the OpenMP library), using something like this: RUN perl -i -pe 's/"libresolv.so.2"/"libresolv.so.2", "libgomp.so.1"/' /opt/_internal/pipx/venvs/auditwheel/lib/python3.9/site-packages/ auditwheel/policy/manylinux-policy.json Oh, and if compiling where platform.machine() == "arm64" then I need to not add the AVX2 compiler flag. The non-setup.py packaging systems I've looked at are for Python-only code bases. Or, if I understand things correctly, I'm supposed to make a new specialized package which implements PEP 518, which I can then use to boot-strap my code. Except, that's still going to use effectively arbitrary code during the compilation step (to run the codegen) and still use setup.py to build the extension. So it's not like the evil disappears.

deleted 4y ago
[deleted]
orf 4y ago
To be clear, I’m not suggesting we remove the ability to compile native extensions. I’m suggesting we find a better way to build them, something a bit more structured, and decouple that specific use case from setup.py. It would be cool to be able to structure this in a way that means I can describe what system libraries I may need without having to execute setup.py and find out, and express compile time flags or options in a structured way. Think of it like cargo.toml va build.rs.
- dalke 4y ago

progval 4y ago

> For improvements I commented: Remove setup.py files and mandate wheels. This would make most C extensions impossible to install on anything other than x86_64-pc-linux-gnu (or arm-linux-gnueabihf/aarch64-linux-gnu if you are lucky) because developers don't want to bother building wheels for them.

mathstuf 4y ago
I think it'd make other things impossible too. One project I help maintain is C++ and is mainly so. It optionally has Python bindings. It also has something like 150 options to the build that affect things. There is zero chance of me ever attempting to make `setup.py` any kind of sensible "entry point" to the build. Instead, the build detects "oh, you want a wheel" and generates `setup.py` to just grab what the C++ build then drops into a place where `build_ext` or whatever expects them to be using some fun globs. It also fills in "features" or whatever the post-name `[name]` stuff is called so you can do some kind of post-build "ok, it has a feature I need" inspection.
urschrei 4y ago
cibuildwheel (which is an official, supported tool) has made this enormously easier. I test and generate wheels with a compiled (Rust! Because of course) extension using a Cython bridge for all supported Python versions for 32-bit and 64-bit Windows, macOS x86_64 and arm64, and whatever manylinux is calling itself this week. No user compilation required. It took about half a day to set up, and is extremely well documented.

blibble 4y ago

> The mission statement they are proposing, “a packaging ecosystem for all”, completely misses the mark. How about a “packaging ecosystem that works” first? I think at the point a programming language is going on about "mission statements" for a packaging tool, you know they've lost the plot copy Maven from 2004 (possibly with less XML) that's it, problem solved

baggiponte 4y ago

I recommend PDM over poetry!

x3n0ph3n3 4y ago

I suggested "one packaging system to rule them all." The fragmentation in this space is frustrating.

cozzyd 4y ago

Many "Python" packages include native code in some form either as bindings or to workaround Python being agonizingly slow. Which means you often need to call make, or cmake or some other build system anyway... unless you want to build wheels for every possible configuration a user might have (which is virtually impossible, considering every combination of OS, architecture, debug options, etc. you may want to support). Plus you need a build system to build the wheels anyway...