3 ms·

I'm less pessimistic about packaging than most. No language that I know of has ever attempted a standardization effort like this. I think it will pay off in the

I'm less pessimistic about packaging than most. No language that I know of has ever attempted a standardization effort like this. I think it will pay off in the long term. But it's taken years to get here and it will take more years to wrap up the project. > An attempt at a specification was rejected due to “lukewarm reception”, even though there exist at least four implementations which are achieving roughly the same goals, and other ecosystems also went through this before. Python has been weird lately. We got structural pattern matching, which is entirely new syntax that is essentially just syntactic sugar and only kind-of solves a problem that relatively few users ever had. But then we reject __pypackages__ and null-coalescing/safe-navigation operators, which solve problems that everyone has and are unambiguous improvements and modernizations of the language. Even if the PEPs had problems and needed to be rewritten from scratch, there is now approximately zero chance that will ever happen. I love Python but sometimes I feel like the decisions are arbitrary and do not reflect what is best for the language. It drives me nuts because I'm a captive user. There's too much momentum for data science and machine learning to switch and be taken seriously in a non-solo workplace. Also I just like Python.

ngrilly 3y ago
> No language that I know of has ever attempted a standardization effort like this. What about Ruby, Rust, or Go, for example?
- oblio 3y ago
  Or, you know, Java, the language that solved packaging around 2004.
  - nerdponx 3y ago
    I don't know how Java packaging works. Got a short summary?
    - oblio 3y ago
      Well, I'll start from the original sin for JS/Python/Ruby/PHP/Perl: 0. Java was designed to be fast enough to be comparable with C/C++. Think 2x slower or less, not 10-100x slower. This leads to: 1. Most Java libraries are in Java, they're not native. And they're portable as-is across platforms. If they do have platform specific code, it's their job to make sure they bundle and load everything needed for the specific problem. 2. In 2004 Maven 1 was launched, then in 2005 Maven 2 came with a repository format update. Packages are zip files called jars (Java ARchive), there are also wars (Web ARchive) which are zip-of-zips meant to be unpacked by the application server before the first launch, or ear (Enterprise ARchive), which are also zip-of-zips but I forget the exact details for using these. 3. The Maven repo format is the same, locally and remotely. When you work with Maven (or Gradle, or any Java build tool, since they're ALL compatible with Maven, otherwise nobody would use them), you get a local cache/mirror/proxy of the remote repos. 4. Because Java has CLASSPATH (sort of like PYTHONPATH but probably better, and it probably inspired PYTHONPATH because I'm sure the Java CLASSPATH predates it), packages are not copied over to the local folder when developing. Maven & co just assemble the correct CLASSPATH and everything is referred to directly from the local repo. You don't have venvs or node_modules because those are just silly hacks that aren't needed here. 5. If you need to package Java stuff, the standard approaches are: - cross platform jars for libraries (these are usually published to the Maven repo and can be used natively by any Java package manager) - zips for desktop apps; shell scripts or executable launchers inside the zips to launch the apps - wars for web applications - ears for huge, enterprise applications It's obviously very deep and complex when you want to look at everything, but that's it. Because Java is really close to "Write Once, Run Anywhere" in practice, for most platforms yeah, you just copy over the jar/zip. The hardest part that people complain is installing the JRE, which is SUPER silly since for a technical person it should be trivial to do. For non technical person, since at least 10 years ago, you can just bundle the JRE with your app. These days (at least 5+ years), I'm fairly sure you can AOT compile the app, even. Python, by comparison, is a horror show. And it all comes from 0. -> Python is slow, it was meant to be used with C libraries, so it carries all the baggage from that ancient and creaky ecosystem. Packaging a Python (or Ruby, or...) app to deploy on Windows, Linux (multiple distros), MacOS is such a horror show that they invented an entire layer with Docker, to just put the whole thing into an almost literal shipping container and not bother with the craziness inside.
- nerdponx 3y ago
  All of those use a single package management tool, which is considered the only correct and acceptable tool to use. That is what everybody wished Python had done 20 years ago, but they didn't, and now we are engaged in this unique experiment, in order to avoid forcing everyone to migrate to one particular tool. Instead, Python is developing a set of interoperable standards and APIs that any number of build and packaging tools can use. So projects can choose whatever build tool makes sense for them, and and users can build any project via a uniform interface. As an example, let's say that I am writing a web app which serves a machine learning model. I have three dependencies: a web framework, a database driver, and a machine learning framework. The web framework might be packaged using Flit, the database driver might be packaged using Setuptools, and the ML framework might be packaged using CMake or Meson. And for my own project I might choose to use Hatch or Poetry. When I use Pip to install my dependencies, the details of what tool they were packaged with are completely abstracted away. Even if I need to compile something because there is no binary package published for my system, Pip will automatically install the required build dependencies in an isolated environment, and it will magically know how to build the package, using nothing but the package's own declaration of its "build backend". And then when I want to publish my own work, and users will have the same experience when they need to install my dependencies. The system doesn't work 100% perfectly in all cases (mostly due to things that are out of scope, like managing shared libraries at the system level), but it actually works most of the time for most people in most situations. And they were no doubt issues transitioning from older systems. But as far as I know, no system like this has ever even been attempted, let alone rolled out to millions of users and working so seamlessly that most people didn't even realize it was happening. So it's not perfect. And maybe the whole idea is backwards and would've been avoided if there had been a single coherent package management story in the first place. But given the scale of the challenge, I think we should all be a little less eager to wave the "Python steering council is stupid & bad" banner.
PurpleRamen 3y ago
Which problem or usecase would null-coalescing/safe-navigation operators solve, which are not already solved with conditional expression or walrus-operator?
- Kwpolska 3y ago
  You can certainly replace `a?.b?.c` with some `if`s. But it gets tedious and unreadable fast.

ngrilly 3y ago

> No language that I know of has ever attempted a standardization effort like this. What about Ruby, Rust, or Go, for example?

oblio 3y ago
Or, you know, Java, the language that solved packaging around 2004.
- nerdponx 3y ago
  I don't know how Java packaging works. Got a short summary?
  - oblio 3y ago
    Well, I'll start from the original sin for JS/Python/Ruby/PHP/Perl: 0. Java was designed to be fast enough to be comparable with C/C++. Think 2x slower or less, not 10-100x slower. This leads to: 1. Most Java libraries are in Java, they're not native. And they're portable as-is across platforms. If they do have platform specific code, it's their job to make sure they bundle and load everything needed for the specific problem. 2. In 2004 Maven 1 was launched, then in 2005 Maven 2 came with a repository format update. Packages are zip files called jars (Java ARchive), there are also wars (Web ARchive) which are zip-of-zips meant to be unpacked by the application server before the first launch, or ear (Enterprise ARchive), which are also zip-of-zips but I forget the exact details for using these. 3. The Maven repo format is the same, locally and remotely. When you work with Maven (or Gradle, or any Java build tool, since they're ALL compatible with Maven, otherwise nobody would use them), you get a local cache/mirror/proxy of the remote repos. 4. Because Java has CLASSPATH (sort of like PYTHONPATH but probably better, and it probably inspired PYTHONPATH because I'm sure the Java CLASSPATH predates it), packages are not copied over to the local folder when developing. Maven & co just assemble the correct CLASSPATH and everything is referred to directly from the local repo. You don't have venvs or node_modules because those are just silly hacks that aren't needed here. 5. If you need to package Java stuff, the standard approaches are: - cross platform jars for libraries (these are usually published to the Maven repo and can be used natively by any Java package manager) - zips for desktop apps; shell scripts or executable launchers inside the zips to launch the apps - wars for web applications - ears for huge, enterprise applications It's obviously very deep and complex when you want to look at everything, but that's it. Because Java is really close to "Write Once, Run Anywhere" in practice, for most platforms yeah, you just copy over the jar/zip. The hardest part that people complain is installing the JRE, which is SUPER silly since for a technical person it should be trivial to do. For non technical person, since at least 10 years ago, you can just bundle the JRE with your app. These days (at least 5+ years), I'm fairly sure you can AOT compile the app, even. Python, by comparison, is a horror show. And it all comes from 0. -> Python is slow, it was meant to be used with C libraries, so it carries all the baggage from that ancient and creaky ecosystem. Packaging a Python (or Ruby, or...) app to deploy on Windows, Linux (multiple distros), MacOS is such a horror show that they invented an entire layer with Docker, to just put the whole thing into an almost literal shipping container and not bother with the craziness inside.
nerdponx 3y ago
All of those use a single package management tool, which is considered the only correct and acceptable tool to use. That is what everybody wished Python had done 20 years ago, but they didn't, and now we are engaged in this unique experiment, in order to avoid forcing everyone to migrate to one particular tool. Instead, Python is developing a set of interoperable standards and APIs that any number of build and packaging tools can use. So projects can choose whatever build tool makes sense for them, and and users can build any project via a uniform interface. As an example, let's say that I am writing a web app which serves a machine learning model. I have three dependencies: a web framework, a database driver, and a machine learning framework. The web framework might be packaged using Flit, the database driver might be packaged using Setuptools, and the ML framework might be packaged using CMake or Meson. And for my own project I might choose to use Hatch or Poetry. When I use Pip to install my dependencies, the details of what tool they were packaged with are completely abstracted away. Even if I need to compile something because there is no binary package published for my system, Pip will automatically install the required build dependencies in an isolated environment, and it will magically know how to build the package, using nothing but the package's own declaration of its "build backend". And then when I want to publish my own work, and users will have the same experience when they need to install my dependencies. The system doesn't work 100% perfectly in all cases (mostly due to things that are out of scope, like managing shared libraries at the system level), but it actually works most of the time for most people in most situations. And they were no doubt issues transitioning from older systems. But as far as I know, no system like this has ever even been attempted, let alone rolled out to millions of users and working so seamlessly that most people didn't even realize it was happening. So it's not perfect. And maybe the whole idea is backwards and would've been avoided if there had been a single coherent package management story in the first place. But given the scale of the challenge, I think we should all be a little less eager to wave the "Python steering council is stupid & bad" banner.

PurpleRamen 3y ago

Which problem or usecase would null-coalescing/safe-navigation operators solve, which are not already solved with conditional expression or walrus-operator?

Kwpolska 3y ago
You can certainly replace `a?.b?.c` with some `if`s. But it gets tedious and unreadable fast.