5 ms·

Cargo has the benefit of being much later to the game, and has thoughtfully incorporated lessons from the past. It's easier to work with 'downstream', and seems

by pxc 4y ago

Cargo has the benefit of being much later to the game, and has thoughtfully incorporated lessons from the past. It's easier to work with 'downstream', and seems like a good model for new languages/runtimes to start with for their own packaging ecosystems.

no_wizard 4y ago
Python had multiple different points in its history to fix its env and package problems. For reasons I don’t understand the PSF has not taken any hard stance on a solid way forward, it’s always pushed back as a community issue. The problem is lack of leadership on this more than anything
- maegul 4y ago
  It certainly seems that way. Guido wasn’t interested in solving it and it was probably one of the things, like a gradual typing system which he was interested in, that he should have made a core issue because of its centrality. Curiously, I wonder whether current leadership are unwilling to acknowledge that the current problems are essentially a leadership problem. See for example my short conversation with Pradyun on Mastodon (https://mas.to/@maegul/109726564552419983 https://mas.to/@maegul/109726564552419983) where I’m not sure they were being open and rational (though they clearly know more than me). This thread was in response to their blog post in Python packaging: https://pradyunsg.me/blog/2023/01/21/thoughts-on-python-packaging/ https://pradyunsg.me/blog/2023/01/21/thoughts-on-python-pack...
- reidrac 4y ago
  They included virtual envs in Python 3, perhaps they should have done the same with some of the packaging solutions.
  - ElectricalUnion 4y ago
    > perhaps they should have done the same with some of the packaging solutions. The packaging solution is pip? (although for some reason it doesn't come with the now standard wheel, so it's less useful that it should be)
- BiteCode_dev 4y ago
  It has. So many things were fixed in python packaging. - "-m" was introduced. - Wheel replaced eggs. - Manylinux target now exists. - pypi was scaled uo, a new api introduced and 2fa added. - setup.py got replaced by setup.cfg and now pyproject.toml is gaining speed. - the py launcher is a thing. - Import priority changed. - Import lib changed. - Zipapps were added. - Venv ships with python. - Pip replaced easy_install, then ensurepip was created. - The new dependancy solver was funded. - Distutils has been sunsetted. Setuptools is on its way. This doesn't include the dozens of third party projects that target improving package that came out in the last 2 decades. Pip-tools by itself has changed the game. A lot has been done, and the situation is 10 times better than it used to be. It is still bad, but not because nothing was done, simply because there is still a lot to do. People dont realize the sheer scale of the task, and the inertia that comes with a language as successful and old as python. You have a language that is 4 years older than java, highly dynamic, yet uses fortran/c/assembly code in very popular lib (scipy/numpy) accross all popular OS, and webassembly and arm, in 32 and 64 bits. People routinely install multiple versions of python on their machine, linux repos freeze the upgrades and split the packages, mac and windows stores ship broken versions, and half the python userbase is composed of people that are not professional coders that can't use a terminal. Of course companies will complain if you change anything that breaks their 5 years old server, and devs will complain if they don't get the last new great feature. It's a really, really hard problem, dealt by a FOSS community that 10 years ago was still running on mostly volunteers and the budget of a small start up. Of course, just to get heat by comments on social media and no thanks for all the effort, as if you own total strangers your free work, with no flaws on top of that. Not to mention you can imagine packaging is not the only things the core devs have to work on.
  - noisy_boy 4y ago
    > This doesn't include the dozens of third party projects that target improving package that came out in the last 2 decades. One would think that two decades of package improvement initiatives should have been a strong enough signal to PSF to prioritize building an official standard solution.
- sdiacom 4y ago
  Not disputing that there's a lack of leadership on the matter, but there's also conflicting use-cases: for example, the requirements for the "Python is just a tool that comes with my OS, and if I need any additional modules I'll install them with my OS package manager" mindset, and the "I'm developing a web application with Python and I want an isolated development environment with controlled dependencies" mindset, are quite different.
  - pxc 4y ago
    > "Python is just a tool that comes with my OS, and if I need any additional modules I'll install them with my OS package manager Imo this isn't really viable because you will eventually run into version conflicts in the transitive dependencies of your the Python applications you're using/developing on your system, on most operating systems. The version(s) that ships with an OS should only be used for shipping applications that are themselves part of the OS/distro.
pjmlp 4y ago
It failed to learn from binary repos used in C and C++ builds, which are the main reason why C++ with all its issues can still manage to compile faster in many cases.
- kibwen 4y ago
  No, Cargo does support this, via sccache. Binaries just aren't provided by crates.io itself, for reasons of economics.
  - pjmlp 4y ago
    No it doesn't, as that is a 3rd party dependency.
    - kibwen 4y ago
      Cargo deliberately supports configurable caches in order to let users provide caching layers.
  - mid-kid 4y ago
    ccache is not a suitable replacement for a stable ABI that allows binaries to be reused across systems.
    - kibwen 4y ago
      I'm referring to sccache, not ccache. The former is a distributed ccache that allows artifacts to be reused across systems.
- pxc 4y ago
  Ecosystems where binary redistribution at the language level is the norm tend to have really gnarly downstream packaging issues, so that dependency management by a general purpose/distro package manager is basically an unsupported use case (.NET) or that for large projects, there may not be an known, working fully from-source build (Java). I would much rather have binary redistribution left up to downstreams.
lloeki 4y ago
Looking at this from the Ruby world, cargo appears to be mostly a clone of the bundler model (with a sprinkle of rake folded in) which has existed for a looooong while ;) It's quite funny to see languages fumbling on dependency management and somehow "refusing" to look at how bundler operates, then stumbling on a very close solution and then community rejoices at their package management being the best thing since slice bread.
- Tade0 4y ago
  How is the approach in Ruby different/better? Asking out of ignorance.
- geewee 4y ago
  I'm pretty sure that Rust hired the people who worked on bundler to build cargo - so it would make sense that they're pretty similar.
- kibwen 4y ago
  Mozilla literally contracted the Bundler authors to create Cargo, so the similarity is intentional. :P
  - lloeki 4y ago
    I'm quite sure it is! I did not know the bundler authors were actively contacted though, thanks for that piece of info!
oblio 4y ago
It's not perfect but pip (and npm...) could have learned a thing or two from Maven. You know, prior art that appeared about 5 years later and had huge adoption. Insularity is a software ecosystem disease.
- POPOSYS 4y ago
  Please describe what is better in Maven. Or please send a source with a comparison or some explanation. Thanks!
  - oblio 4y ago
    Maven uses groupIds and artifactIds to group dependencies. GroupIds are reverse DNS notation based on a subdomain you control. They both group dependencies (duh!) and avoid the top-level squatting problem. It's much harder to mix up http-utils like in npm and pypi, since there's no http-utils. There's com.google:http-utils and there's org.apache:http-utils (all the names are made up for example purposes). Maven has a local centralized artifact repository/cache where artifacts are downloaded and then referred to by every project. In Java there isn't even a need for symlinks, since Java has a dynamic classpath aka the places where the libraries are searched. Though Python could do the same thing with the PYTHONPATH. Maven is plugin based and it's reasonably easy to build your own, so it's been extended like crazy to do all sorts of wonderful and loony things like build C++. It's actually expected to be extended, the tool itself implements Java builds through plugins, the core doesn't do that. Oh, from the start it had tools to cache/proxy your own packages. So you'd be able to have a mini-centralized repo for your company with just the stuff you need, in case the main repo is down. Artifactory, Nexus, there are others. And not just cache/mirror, but proxy: you'd call your local repo, and if it wouldn't find the thing, it would download it from internet repos you configure. Another thing, the package format and repo structure are simple and straightforward. So Gradle, sbt, other fancier Java build tools just use Maven repos. There are a million other things like that, it's a very robust tool with a very robust ecosystem. The main pain point is the early-2000s style XML configuration format :-| That makes a lot of people avoid it due to verbosity. But you know what? The format is stable. It's quite readable. There are a gazillion tools to manage it, IDEs have advanced autocompletion for it, etc. And I'm not saying that other folks should copy the config file format, just the rest of the ideas. Heck, even Maven has Polyglot Maven, to use different config file formats (not super adopted, but it's there).

by pxc 4y ago

no_wizard 4y ago

Python had multiple different points in its history to fix its env and package problems. For reasons I don’t understand the PSF has not taken any hard stance on a solid way forward, it’s always pushed back as a community issue. The problem is lack of leadership on this more than anything

maegul 4y ago
It certainly seems that way. Guido wasn’t interested in solving it and it was probably one of the things, like a gradual typing system which he was interested in, that he should have made a core issue because of its centrality. Curiously, I wonder whether current leadership are unwilling to acknowledge that the current problems are essentially a leadership problem. See for example my short conversation with Pradyun on Mastodon (https://mas.to/@maegul/109726564552419983 https://mas.to/@maegul/109726564552419983) where I’m not sure they were being open and rational (though they clearly know more than me). This thread was in response to their blog post in Python packaging: https://pradyunsg.me/blog/2023/01/21/thoughts-on-python-packaging/ https://pradyunsg.me/blog/2023/01/21/thoughts-on-python-pack...
reidrac 4y ago
They included virtual envs in Python 3, perhaps they should have done the same with some of the packaging solutions.
- ElectricalUnion 4y ago
  > perhaps they should have done the same with some of the packaging solutions. The packaging solution is pip? (although for some reason it doesn't come with the now standard wheel, so it's less useful that it should be)
BiteCode_dev 4y ago
It has. So many things were fixed in python packaging. - "-m" was introduced. - Wheel replaced eggs. - Manylinux target now exists. - pypi was scaled uo, a new api introduced and 2fa added. - setup.py got replaced by setup.cfg and now pyproject.toml is gaining speed. - the py launcher is a thing. - Import priority changed. - Import lib changed. - Zipapps were added. - Venv ships with python. - Pip replaced easy_install, then ensurepip was created. - The new dependancy solver was funded. - Distutils has been sunsetted. Setuptools is on its way. This doesn't include the dozens of third party projects that target improving package that came out in the last 2 decades. Pip-tools by itself has changed the game. A lot has been done, and the situation is 10 times better than it used to be. It is still bad, but not because nothing was done, simply because there is still a lot to do. People dont realize the sheer scale of the task, and the inertia that comes with a language as successful and old as python. You have a language that is 4 years older than java, highly dynamic, yet uses fortran/c/assembly code in very popular lib (scipy/numpy) accross all popular OS, and webassembly and arm, in 32 and 64 bits. People routinely install multiple versions of python on their machine, linux repos freeze the upgrades and split the packages, mac and windows stores ship broken versions, and half the python userbase is composed of people that are not professional coders that can't use a terminal. Of course companies will complain if you change anything that breaks their 5 years old server, and devs will complain if they don't get the last new great feature. It's a really, really hard problem, dealt by a FOSS community that 10 years ago was still running on mostly volunteers and the budget of a small start up. Of course, just to get heat by comments on social media and no thanks for all the effort, as if you own total strangers your free work, with no flaws on top of that. Not to mention you can imagine packaging is not the only things the core devs have to work on.
- noisy_boy 4y ago
  > This doesn't include the dozens of third party projects that target improving package that came out in the last 2 decades. One would think that two decades of package improvement initiatives should have been a strong enough signal to PSF to prioritize building an official standard solution.
sdiacom 4y ago
Not disputing that there's a lack of leadership on the matter, but there's also conflicting use-cases: for example, the requirements for the "Python is just a tool that comes with my OS, and if I need any additional modules I'll install them with my OS package manager" mindset, and the "I'm developing a web application with Python and I want an isolated development environment with controlled dependencies" mindset, are quite different.
- pxc 4y ago
  > "Python is just a tool that comes with my OS, and if I need any additional modules I'll install them with my OS package manager Imo this isn't really viable because you will eventually run into version conflicts in the transitive dependencies of your the Python applications you're using/developing on your system, on most operating systems. The version(s) that ships with an OS should only be used for shipping applications that are themselves part of the OS/distro.

pjmlp 4y ago

It failed to learn from binary repos used in C and C++ builds, which are the main reason why C++ with all its issues can still manage to compile faster in many cases.

kibwen 4y ago
No, Cargo does support this, via sccache. Binaries just aren't provided by crates.io itself, for reasons of economics.
- pjmlp 4y ago
  No it doesn't, as that is a 3rd party dependency.
  - kibwen 4y ago
    Cargo deliberately supports configurable caches in order to let users provide caching layers.
- mid-kid 4y ago
  ccache is not a suitable replacement for a stable ABI that allows binaries to be reused across systems.
  - kibwen 4y ago
    I'm referring to sccache, not ccache. The former is a distributed ccache that allows artifacts to be reused across systems.
pxc 4y ago
Ecosystems where binary redistribution at the language level is the norm tend to have really gnarly downstream packaging issues, so that dependency management by a general purpose/distro package manager is basically an unsupported use case (.NET) or that for large projects, there may not be an known, working fully from-source build (Java). I would much rather have binary redistribution left up to downstreams.

lloeki 4y ago

Looking at this from the Ruby world, cargo appears to be mostly a clone of the bundler model (with a sprinkle of rake folded in) which has existed for a looooong while ;) It's quite funny to see languages fumbling on dependency management and somehow "refusing" to look at how bundler operates, then stumbling on a very close solution and then community rejoices at their package management being the best thing since slice bread.

Tade0 4y ago
How is the approach in Ruby different/better? Asking out of ignorance.
geewee 4y ago
I'm pretty sure that Rust hired the people who worked on bundler to build cargo - so it would make sense that they're pretty similar.
kibwen 4y ago
Mozilla literally contracted the Bundler authors to create Cargo, so the similarity is intentional. :P
- lloeki 4y ago
  I'm quite sure it is! I did not know the bundler authors were actively contacted though, thanks for that piece of info!

oblio 4y ago

It's not perfect but pip (and npm...) could have learned a thing or two from Maven. You know, prior art that appeared about 5 years later and had huge adoption. Insularity is a software ecosystem disease.

POPOSYS 4y ago
Please describe what is better in Maven. Or please send a source with a comparison or some explanation. Thanks!
- oblio 4y ago
  Maven uses groupIds and artifactIds to group dependencies. GroupIds are reverse DNS notation based on a subdomain you control. They both group dependencies (duh!) and avoid the top-level squatting problem. It's much harder to mix up http-utils like in npm and pypi, since there's no http-utils. There's com.google:http-utils and there's org.apache:http-utils (all the names are made up for example purposes). Maven has a local centralized artifact repository/cache where artifacts are downloaded and then referred to by every project. In Java there isn't even a need for symlinks, since Java has a dynamic classpath aka the places where the libraries are searched. Though Python could do the same thing with the PYTHONPATH. Maven is plugin based and it's reasonably easy to build your own, so it's been extended like crazy to do all sorts of wonderful and loony things like build C++. It's actually expected to be extended, the tool itself implements Java builds through plugins, the core doesn't do that. Oh, from the start it had tools to cache/proxy your own packages. So you'd be able to have a mini-centralized repo for your company with just the stuff you need, in case the main repo is down. Artifactory, Nexus, there are others. And not just cache/mirror, but proxy: you'd call your local repo, and if it wouldn't find the thing, it would download it from internet repos you configure. Another thing, the package format and repo structure are simple and straightforward. So Gradle, sbt, other fancier Java build tools just use Maven repos. There are a million other things like that, it's a very robust tool with a very robust ecosystem. The main pain point is the early-2000s style XML configuration format :-| That makes a lot of people avoid it due to verbosity. But you know what? The format is stable. It's quite readable. There are a gazillion tools to manage it, IDEs have advanced autocompletion for it, etc. And I'm not saying that other folks should copy the config file format, just the rest of the ideas. Heck, even Maven has Polyglot Maven, to use different config file formats (not super adopted, but it's there).