5 ms·

This is fantastic - first hand I have noticed this trend is making working with scripting languages less friendly though. For a Node example, with esbuild, I c

This is fantastic - first hand I have noticed this trend is making working with scripting languages less friendly though. For a Node example, with esbuild, I can no longer ever install packages and assume they will work fine when running under the host (macOS) and Docker VM (Linux). Unless Pydantic is downloading all OS binaries with the package and loading the right one at runtime, this would become a "problem" as well. For those coming strictly from scripting language land, this could screw up their workflow quite a bit. A lot more of these people have never worked with native binaries than you think. I, like many here, understand this basic nuance and would rather my tools be fast. However, I think this is quickly eroding the value of using a dynamic scripting language. The platform portability advantage is completely removed as stuff like this spreads and it makes working with those scripting languages a pain in the ass to the point I wouldn't even want to use a scripting language anymore. It becomes totally opaque whether or not your code can actually run on another target. In my case this is OK, but I think I'm in the minority.

ivoflipse 4y ago
Welcome to the problem the Scientific Python community has been struggling with for years. Providing binaries for a plurality of platforms brings challenges with it, but I reckon on the major platforms Pydantic will be fine as they won't have hard to compile parts. But Rust isn't available everywhere, so surely someone will be losing out.
- jeroenhd 4y ago
  Rust is available for almost all common platforms. There are crates that don't run on some platforms (i.e. relying on CPU intrinsics that aren't available cross platform) but there's a Linux, macOS, Windows, FreeBSD, NetBSD, and more. It's supported on x64, ARM, MIPS, PowerPC, RISC-V, and s390x. There's even a rustup version for running the compiler on i686 Android. If you're running software on a Pentium 2 or a DEC machine then maybe you're out of luck, but there aren't that many platforms out there that are still in use that will run into problems here. Perhaps if you're stuck on a long unsupported version of Windows or CentOS and need the latest version of Python packages you're in trouble, but you already were when you got stuck on a legacy platform anyway; I doubt you'd be updating your dependencies to a version where this is a problem if your OS is that outdated. You'll have the same problem with your dependencies needing a C compiler more recent than GCC 4.
  - iudqnolq 4y ago
    But with python the custom is to ship prebuilt binaries. And there won't be prebuilt binaries for plenty of normal-ish platforms, like Alpine Linux. So those users will now have to set up local complication for the first time. This is no big deal for us, but it can be quite confusing and frustrating for users who haven't written compiler code before.
    - jammycrisp 4y ago
      It looks like pydantic-core is distributing musllinux wheels, which should work fine on alpine. Fwiw tooling like cibuildwheel makes building and publishing wheels for all the common platforms fairly straightforward now.
    - jeroenhd 4y ago
      My experience is that though the error messages Python produces are quite obscure and verbose, the installation of the necessary packages went quite smoothly. I've pkg added Rust in Termux on my phone, and it that works I don't think getting stuff built will be all that challenging. What Python dependency managers need to improve, in my opinion, is for their toolsets to inform users about native dependencies when compilation is required.
- peoplefromibiza 4y ago
  once again Elixir community proves that tooling is as important as features of the language or number of libraries available https://dashbit.co/blog/rustler-precompiled https://dashbit.co/blog/rustler-precompiled
rtpg 4y ago
I think that you've been lucky, since even for mostly-pure Python packages, you often can't install in one env and just use it in another, because there might be some deeper dependencies. My wish would be for there to be a way to get "most of the installation" working in a portable way and for OS-specific stuff to somehow be isolated into a final step, but in the world of Docker images that feels like a rare user story.
- deivid 4y ago
  Try out shiv[1], it will package up your source and depndencies into a single file, though you still require the Python interpreter to run it on the target [1] https://github.com/linkedin/shiv https://github.com/linkedin/shiv
kodablah 4y ago
> Unless Pydantic is downloading all OS binaries with the package and loading the right one at runtime, this would become a "problem" as well. Nah, it's not that bad. I built a Rust-backed Python library used by many [0], and with setuptools-rust (maturin wasn't flexible enough at the time) and cibuildwheel and GH actions, the wheels are built/shipped with the shared libraries embedded and the end user never has to worry or even be aware of its presence. Pydantic has already been shipping a binary mode with an option for pure Python, so maybe they'll keep the pure Python mode around. 0 - https://github.com/temporalio/sdk-python https://github.com/temporalio/sdk-python

ivoflipse 4y ago

Welcome to the problem the Scientific Python community has been struggling with for years. Providing binaries for a plurality of platforms brings challenges with it, but I reckon on the major platforms Pydantic will be fine as they won't have hard to compile parts. But Rust isn't available everywhere, so surely someone will be losing out.

jeroenhd 4y ago
Rust is available for almost all common platforms. There are crates that don't run on some platforms (i.e. relying on CPU intrinsics that aren't available cross platform) but there's a Linux, macOS, Windows, FreeBSD, NetBSD, and more. It's supported on x64, ARM, MIPS, PowerPC, RISC-V, and s390x. There's even a rustup version for running the compiler on i686 Android. If you're running software on a Pentium 2 or a DEC machine then maybe you're out of luck, but there aren't that many platforms out there that are still in use that will run into problems here. Perhaps if you're stuck on a long unsupported version of Windows or CentOS and need the latest version of Python packages you're in trouble, but you already were when you got stuck on a legacy platform anyway; I doubt you'd be updating your dependencies to a version where this is a problem if your OS is that outdated. You'll have the same problem with your dependencies needing a C compiler more recent than GCC 4.
- iudqnolq 4y ago
  But with python the custom is to ship prebuilt binaries. And there won't be prebuilt binaries for plenty of normal-ish platforms, like Alpine Linux. So those users will now have to set up local complication for the first time. This is no big deal for us, but it can be quite confusing and frustrating for users who haven't written compiler code before.
  - jammycrisp 4y ago
    It looks like pydantic-core is distributing musllinux wheels, which should work fine on alpine. Fwiw tooling like cibuildwheel makes building and publishing wheels for all the common platforms fairly straightforward now.
  - jeroenhd 4y ago
    My experience is that though the error messages Python produces are quite obscure and verbose, the installation of the necessary packages went quite smoothly. I've pkg added Rust in Termux on my phone, and it that works I don't think getting stuff built will be all that challenging. What Python dependency managers need to improve, in my opinion, is for their toolsets to inform users about native dependencies when compilation is required.
peoplefromibiza 4y ago
once again Elixir community proves that tooling is as important as features of the language or number of libraries available https://dashbit.co/blog/rustler-precompiled https://dashbit.co/blog/rustler-precompiled

rtpg 4y ago

I think that you've been lucky, since even for mostly-pure Python packages, you often can't install in one env and just use it in another, because there might be some deeper dependencies. My wish would be for there to be a way to get "most of the installation" working in a portable way and for OS-specific stuff to somehow be isolated into a final step, but in the world of Docker images that feels like a rare user story.

deivid 4y ago
Try out shiv[1], it will package up your source and depndencies into a single file, though you still require the Python interpreter to run it on the target [1] https://github.com/linkedin/shiv https://github.com/linkedin/shiv

kodablah 4y ago

> Unless Pydantic is downloading all OS binaries with the package and loading the right one at runtime, this would become a "problem" as well. Nah, it's not that bad. I built a Rust-backed Python library used by many [0], and with setuptools-rust (maturin wasn't flexible enough at the time) and cibuildwheel and GH actions, the wheels are built/shipped with the shared libraries embedded and the end user never has to worry or even be aware of its presence. Pydantic has already been shipping a binary mode with an option for pure Python, so maybe they'll keep the pure Python mode around. 0 - https://github.com/temporalio/sdk-python https://github.com/temporalio/sdk-python