6 ms·

It seems most people agree that Python's packaging isn't great, but, conversely, is there a language where most people agree that the approach to packaging is a

It seems most people agree that Python's packaging isn't great, but, conversely, is there a language where most people agree that the approach to packaging is awesome? I mean, what's the gold standard to aspire to?

nixpulvis 4y ago
Ruby gems and Cargo are pretty awesome. Also, I find I like Arch linux's Pacman quite a bit, though that's a slightly different use case where versioning isn't resolved.
- oconnor663 4y ago
  Cargo is a gold standard imho, but there are definitely some simplifying decisions it makes that might not fit Python: - Dependencies and executables can run arbitrary code during build. For example, Cargo knows almost nothing about how to build C code, and the common workflow is to pull in the popular `cc` library for this and call it in your build.rs. - There's mostly no such thing as "installing a library", and each project builds its dependencies from source. This is baked in at the Cargo level (the install command just doesn't work for library crates) and also at the language level (no stable ABI besides "extern C"). - Related to that, the final product of a Cargo workflow is a mostly-statically-linked binary, so there isn't really an equivalent to virtualenv.
  - Kwpolska 4y ago
    > - Dependencies and executables can run arbitrary code during build. For example, Cargo knows almost nothing about how to build C code, and the common workflow is to pull in the popular `cc` library for this and call it in your build.rs. Python packaging involves arbitrary code execution, even today. While it’s less obvious than it was with `setup.py`, if you’re installing from source, the package can specify any build backend it wants, and that build backend can do anything. This could be mitigated by having an allowlist of build backends, but who would be responsible for vetting the tools, and how would it be guaranteed that an allowed tool is not taken over by someone hostile, and that an allowed tool does not have a system to run arbitrary package-provided code?
  - earthling8118 4y ago
    I don't think the final build output being statically linked has any connection to a virtualenv. I've only ever used Python's virtualenvs to avoid having to install dependencies globally. When using Cargo you don't need to worry about that because their data is saved per-project already and Cargo can sort itself out when you invole a command
TheChaplain 4y ago
Don't know if it counts but Java has done pretty well in my opinion. A single JAR-file can combine everything you need to run an application or be distributed as a library, and as long as the java command is installed it just works. Likewise WAR's make it dead simple to add/remove/update applications during runtime. Of course there are always special cases, but packaging and distribution in the Java world have always been painless to me.
- int_19h 4y ago
  Python has similar facilities for distribution, since it can load code directly from .zip files, treating them as if they were directories - very similar to JAR. The problem is on the stage where you have to acquire and manage those dependencies.
Kwpolska 4y ago
I don’t think there’s much complaining in the C#/.NET ecosystem. Sure, there are alternate tools (FAKE for F#), but I believe most people are happy enough with dotnet.exe/msbuild/nuget.
orhmeh09 4y ago
R with CRAN. renv for reproducibility
neillyons 4y ago
I'm a big fan of `mix` for Elixir. https://hexdocs.pm/mix/Mix.html https://hexdocs.pm/mix/Mix.html
- di4na 4y ago
  This. Mix is basically a mix (pun unintended) of Leiningen and Bundler. It works. Quite well. Part of this is due to erlang solving a lot of the problems upfront (releases) and then the Hex team handling a lot of the rest. But in general, the "Yehuda Katz" lineage (Bundler, Cargo, Yarn v1, mix as José used to be mentored by Yehuda) have pretty good tools worth copying. At least as a base.
DangitBobby 4y ago
I think this is what success for unification looks like: Let's say there are two tools, one called pyup and one called pygo. pyup: responsible for which versions of python are available on your machine, as well as which one is the default "system" python. It also keeps itself and pygo updated. pygo: responsible for (among other things) - allowing a user to specify a python version for a project - allowing a user to specify a python version for a script - helping the user install a compatible python version for the project with pyup - helping the user install a compatible python version for a script with pyup - selecting a compatible python version/env for project - selection a compatible python version/env for a script - allowing a user to specify project dependencies - allowing a user to specify script dependencies within a script - determining which dependencies are required for a project - determining which dependencies are required for a script - installing project dependencies - installing script dependencies - installing projects or scripts as executables I MUST NOT need a base python install to make any of this work. Tools are constantly mucking with system python and which one is on my path so I can't trust system python, period. pyup and pygo should be their own binaries which invoke python. example.py: #!/usr/bin/env pygo # version: ~3.11 # requirements: # requests~=2 import requests requests.get("https://example.com") When I run ./example.py for the first time: - pygo helps me install python 3.11 - pygo installs requests in a location that I don't have to worry about When I run ./example.py for the second time, the script runs without error. If I still need to use something like virtualenv, poetry, or conda on top of this, the unification project has failed.
- alwayslikethis 4y ago
  Gentoo's system sort of does both of these things. Among other things, you can control which packages are installed for which Python versions, dependencies across versions and default version for the system. Though it doesn't use the PyPI ecosystem directly, so everything has to be individually packaged for it to work.
gen220 4y ago
I think Go (if you started using it in the `go mod` era) and Rust have the best stories around package management. The interface around `go mod` is kind of confusing, but I have actual trust in the dependency graph it generates for me. Cargo has, afaict, nailed both the interface and trust in what's going on under-the-hood. In the Python world, Poetry isn't too bad. It's terribly slow in comparison to `go mod` or Cargo, but I generally trust and understand what's happening on the inside, and it's interface is fairly legible to newcomers.
EdwardDiego 4y ago
After joining a company with a Python codebase, goddamn I miss Java so damn much. For a single example, all packages in repositories are namespaced, and namespace ownership is verified. [0] So there's no chance of typosquatting an existing package, nor do you have to worry about someone jumping in and claiming all the "good" names. [0]: https://central.sonatype.org/publish/#individual-projects-open-source-software-repository-hosting-ossrh https://central.sonatype.org/publish/#individual-projects-op...

nixpulvis 4y ago

Ruby gems and Cargo are pretty awesome. Also, I find I like Arch linux's Pacman quite a bit, though that's a slightly different use case where versioning isn't resolved.

oconnor663 4y ago
Cargo is a gold standard imho, but there are definitely some simplifying decisions it makes that might not fit Python: - Dependencies and executables can run arbitrary code during build. For example, Cargo knows almost nothing about how to build C code, and the common workflow is to pull in the popular `cc` library for this and call it in your build.rs. - There's mostly no such thing as "installing a library", and each project builds its dependencies from source. This is baked in at the Cargo level (the install command just doesn't work for library crates) and also at the language level (no stable ABI besides "extern C"). - Related to that, the final product of a Cargo workflow is a mostly-statically-linked binary, so there isn't really an equivalent to virtualenv.
- Kwpolska 4y ago
  > - Dependencies and executables can run arbitrary code during build. For example, Cargo knows almost nothing about how to build C code, and the common workflow is to pull in the popular `cc` library for this and call it in your build.rs. Python packaging involves arbitrary code execution, even today. While it’s less obvious than it was with `setup.py`, if you’re installing from source, the package can specify any build backend it wants, and that build backend can do anything. This could be mitigated by having an allowlist of build backends, but who would be responsible for vetting the tools, and how would it be guaranteed that an allowed tool is not taken over by someone hostile, and that an allowed tool does not have a system to run arbitrary package-provided code?
- earthling8118 4y ago
  I don't think the final build output being statically linked has any connection to a virtualenv. I've only ever used Python's virtualenvs to avoid having to install dependencies globally. When using Cargo you don't need to worry about that because their data is saved per-project already and Cargo can sort itself out when you invole a command

TheChaplain 4y ago

Don't know if it counts but Java has done pretty well in my opinion. A single JAR-file can combine everything you need to run an application or be distributed as a library, and as long as the java command is installed it just works. Likewise WAR's make it dead simple to add/remove/update applications during runtime. Of course there are always special cases, but packaging and distribution in the Java world have always been painless to me.

int_19h 4y ago
Python has similar facilities for distribution, since it can load code directly from .zip files, treating them as if they were directories - very similar to JAR. The problem is on the stage where you have to acquire and manage those dependencies.

Kwpolska 4y ago

I don’t think there’s much complaining in the C#/.NET ecosystem. Sure, there are alternate tools (FAKE for F#), but I believe most people are happy enough with dotnet.exe/msbuild/nuget.

orhmeh09 4y ago

R with CRAN. renv for reproducibility

neillyons 4y ago

I'm a big fan of `mix` for Elixir. https://hexdocs.pm/mix/Mix.html https://hexdocs.pm/mix/Mix.html

di4na 4y ago
This. Mix is basically a mix (pun unintended) of Leiningen and Bundler. It works. Quite well. Part of this is due to erlang solving a lot of the problems upfront (releases) and then the Hex team handling a lot of the rest. But in general, the "Yehuda Katz" lineage (Bundler, Cargo, Yarn v1, mix as José used to be mentored by Yehuda) have pretty good tools worth copying. At least as a base.

DangitBobby 4y ago

I think this is what success for unification looks like: Let's say there are two tools, one called pyup and one called pygo. pyup: responsible for which versions of python are available on your machine, as well as which one is the default "system" python. It also keeps itself and pygo updated. pygo: responsible for (among other things) - allowing a user to specify a python version for a project - allowing a user to specify a python version for a script - helping the user install a compatible python version for the project with pyup - helping the user install a compatible python version for a script with pyup - selecting a compatible python version/env for project - selection a compatible python version/env for a script - allowing a user to specify project dependencies - allowing a user to specify script dependencies within a script - determining which dependencies are required for a project - determining which dependencies are required for a script - installing project dependencies - installing script dependencies - installing projects or scripts as executables I MUST NOT need a base python install to make any of this work. Tools are constantly mucking with system python and which one is on my path so I can't trust system python, period. pyup and pygo should be their own binaries which invoke python. example.py: #!/usr/bin/env pygo # version: ~3.11 # requirements: # requests~=2 import requests requests.get("https://example.com") When I run ./example.py for the first time: - pygo helps me install python 3.11 - pygo installs requests in a location that I don't have to worry about When I run ./example.py for the second time, the script runs without error. If I still need to use something like virtualenv, poetry, or conda on top of this, the unification project has failed.

alwayslikethis 4y ago
Gentoo's system sort of does both of these things. Among other things, you can control which packages are installed for which Python versions, dependencies across versions and default version for the system. Though it doesn't use the PyPI ecosystem directly, so everything has to be individually packaged for it to work.

gen220 4y ago

I think Go (if you started using it in the `go mod` era) and Rust have the best stories around package management. The interface around `go mod` is kind of confusing, but I have actual trust in the dependency graph it generates for me. Cargo has, afaict, nailed both the interface and trust in what's going on under-the-hood. In the Python world, Poetry isn't too bad. It's terribly slow in comparison to `go mod` or Cargo, but I generally trust and understand what's happening on the inside, and it's interface is fairly legible to newcomers.

EdwardDiego 4y ago

After joining a company with a Python codebase, goddamn I miss Java so damn much. For a single example, all packages in repositories are namespaced, and namespace ownership is verified. [0] So there's no chance of typosquatting an existing package, nor do you have to worry about someone jumping in and claiming all the "good" names. [0]: https://central.sonatype.org/publish/#individual-projects-open-source-software-repository-hosting-ossrh https://central.sonatype.org/publish/#individual-projects-op...