5 ms·

Yes, that's a fundamental reason python packaging is a mess. Python success is largely due to the availability of key mixed language packages. No other mainstre

by cdavid 3y ago

Yes, that's a fundamental reason python packaging is a mess. Python success is largely due to the availability of key mixed language packages. No other mainstream language package manager has to deal with this. For example, cargo for rust, which is great, can assume to package mostly rust-only code. And while it is compiled, the language "owns" the compiler, which means building from sources as distribution strategy works. I don't know how/if cargo can deal with e.g. fortran out of the box, but I doubt cargo on windows would work well if top cargo packages required fortran code. The single biggest improvement for python ecosystem was the standardisation of a binary package format, wheel. It is only then that the whole scientific python ecosystem started to thrive on windows. But binary compatibility is a huge PITA, especially across languages and CPUs.

NewJazz 3y ago
Doesn't R handle fortran and C++ packages?
- azalemeth 3y ago
  It does -- and to a much lesser extent shims exist for other languages as well. R also has imperfect packaging but I think it's handled well albeit with a level of complexity that also goes up a lot unexpectedly at times. For a truly great experience, call python packages within R in a custom conda environment in order to get data out of pandas in a particularly unholy way...
  - disgruntledphd2 3y ago
    R has always shipped binary packages on Windows and Mac to avoid lots of the pain we see in Python. Also, all packages must build with the latest version of R, or they are removed from CRAN. This makes the dep problems a lot less severe than we see with Python.
winstonewert 3y ago
Many rust crates actually do package code written in other languages. There are plenty of useful C/C++/Fortran libraries that nobody has rewritten into Rust but for which wrappers have been created that call into C. It works in Rust because the build.rs lets libraries do whatever they want during the build process, including invoking compilers for other languages. Various factors still make the Rust and Python story different (Python uses more mixed-language packages, the Rust demographic is more technically advanced, etc). But a big one is that in Rust, the FFI is defined in Rust. Recompiling just the Rust code gives you an updated FFI compatible with your version of Rust. In Python, the FFI is typically defined in C, so recompiling the python won't get you a compatible FFI. If Python did all FFI through something like ctypes, it would be much more smooth.
- glandium 3y ago
  > It works in Rust because the build.rs lets libraries do whatever they want during the build process, including invoking compilers for other languages. It's still its own mess. You'll find plenty of people having problems with openssl, for instance.
  - oefrha 3y ago
    I have the pleasure of maintaining a reasonably popular -sys crate, and getting it working on Windows is an absolute nightmare, especially considering the competing toolchains & ABIs (msvc and gnu) and millions of ways the C dependency can be installed (it’s a big one, and users may choose a specific version, so we don’t bundle it), with no pkg-config telling you where to look. No idea how build.rs being able to run any Rust code is an advantage, it’s not like setup.py can’t run any Python code and shell out. In fact, bespoke code capable of spitting out inscrutable errors during `setup.py install` is the old way that newer tooling tries to avoid. Rust evangelism is puzzling.
    - oblio 3y ago
      Wait a second, I need to understand this better. If you cargo build, can that run a dependencies' build including trying to compile C and stuff?
the__alchemist 3y ago
As you might expect, compiling rust crates that use C libraries can lead to the inscrutable-block-of-text linker errors we know and love. I have been having a rough time with CUDA, CMSIS-DSP, Tensorflow, and OpenCV during the past few weeks. One of them requires LLVM=v15 to be installed; another requires v16+. A diff one requires an old version of rustc to be installed. On a diff one, when I posted on Github, the maintainers assured me the crate is, in fact, fine, and my system is misconfigured; phew!
- oefrha 3y ago
  Lol, as a maintainer of a reasonably popular and complex -sys crate, our crate is “fine” on Windows, at least in theory, and I’ve heard of successes using it. However, I can’t even port my own app depending on said -sys crate to Windows; there’s always a wall of linker errors. If you report a Windows problem to me, I won’t tell you it’s fine, I just throw my hands in the air.
  - cshokie 3y ago
    Out of curiosity is it a “Windows is objectively difficult” problem, or a “Windows is not Linux and I know Linux best” problem? I’ve only begun using it so my expertise is limited, but I think vcpkg aims to help with some of these difficulties by shipping code as source and then running make on dependencies so they are guaranteed ABI compatible because the same compiler builds everything.
    - oefrha 3y ago
      That I don’t know Windows well is certainly a factor, but I think it’s at most 40% of the problem. Several notable issues, not all: - Competing msvc and gnu toolchains and ABIs, with native and Windows-first dependencies working better or exclusively with msvc, and *ix-first dependencies working better or exclusively with gnu, is a uniquely Windows situation. Which is which for a given build is also not clearly labeled most of the time. (You might mention glibc vs musl, but there’s basically nothing uniquely musl, and when you’re compiling for musl you can almost always get/compile everything for musl from the ground up.) - Confusing coexistence of x86 and x64 is another thing largely unique to Windows. (amd64 and arm64 are much more clearly separated in Apple land.) - Package management is a complete mess. Choco, scoop, win-get, nuget, vcpkg, ad hoc msi, ad hoc exe installer, ad hoc zip, etc. etc. There’s no pkg-config telling you where to look and which compiler flags to use. If you want to pick up a user-installed dep, you special case everything (e.g. looking in vcpkg path) and/or ask user to supply the search path(s) in env var(s). Anyway, shit mostly(tm) just work(tm) on *ix if you follow the happy path. There’s no happy path on Windows more often than not.
jchw 3y ago
I don't think this is really true. NodeJS has quite a similar problem.
shoo 3y ago
> The single biggest improvement for python ecosystem was the standardisation of a binary package format, wheel. I agree. Some people love to complain about python packaging. But from one perspective, it's arguably been a solved problem since wheels were introduced 10 years ago. The introduction of wheels was a massive step forward. Only depend on wheel archives, don't depend on packages that need to be built from source, and drag in all manner of exciting compile-time dependencies. If there's a package you want to depend on for your target platform, and the maintainers don't produce a prebuilt wheel archive for your platform -- well, set up a build server and build some wheels yourself, and host them somewhere, or pick a different platform.
sertbdfgbnfgsd 3y ago
> Yes, that's a fundamental reason python packaging is a mess. Python success is largely due to the availability of key mixed language packages. No other mainstream language package manager has to deal with this. Admittedly I'm not a python expert, but julia handles this just fine? It doesn't seem like it's a difficulty inherent to "mixed language packages". Somehow it appears to me that python's approach is just bad somehow.
- rightbyte 3y ago
  Then again jl seems to take 10s to start doing something non-trivial each time while modules are being compiled.
  - sertbdfgbnfgsd 3y ago
    THEN AGAIN, If you want to restart repeatedly, for whatever reason (??), maybe you should just compile the modules once...? See e.g. [1] TLDR: There's a package called PrecompileTools.jl [2] which allows you to list: I want to compile and cache native code for the following methods on the following types. This isn't complicated stuff. [1] https://julialang.org/blog/2023/04/julia-1.9-highlights/#caching_of_native_code https://julialang.org/blog/2023/04/julia-1.9-highlights/#cac... [2] https://julialang.github.io/PrecompileTools.jl/stable/ https://julialang.github.io/PrecompileTools.jl/stable/
    - markkitti 3y ago
      Julia 1.9's native precompilation is definitely helpful in that regard but loading those native shared libraries (.so files on Linux) into Julia does take some to verify. If the main objective is to reduce time to load and time to first task then PackageCompiler.jl [3] is still the ultimate way to do so. Because Julia is a dynamic language, there are some complicated compilation issues such as invalidation and recompilation that arise. Adding new methods or packages may result in already compiled code no longer statically dispatching correctly requiring invalidation and recompilation of that code. It slightly more complicated than what you stated. It's "I want to compile and cache native code for the following methods on the following types in this particular environment". PackageCompiler.jl can wrap the entire environment into a single native library, the system image. [3] https://github.com/JuliaLang/PackageCompiler.jl https://github.com/JuliaLang/PackageCompiler.jl

by cdavid 3y ago

NewJazz 3y ago

Doesn't R handle fortran and C++ packages?

azalemeth 3y ago
It does -- and to a much lesser extent shims exist for other languages as well. R also has imperfect packaging but I think it's handled well albeit with a level of complexity that also goes up a lot unexpectedly at times. For a truly great experience, call python packages within R in a custom conda environment in order to get data out of pandas in a particularly unholy way...
- disgruntledphd2 3y ago
  R has always shipped binary packages on Windows and Mac to avoid lots of the pain we see in Python. Also, all packages must build with the latest version of R, or they are removed from CRAN. This makes the dep problems a lot less severe than we see with Python.

winstonewert 3y ago

Many rust crates actually do package code written in other languages. There are plenty of useful C/C++/Fortran libraries that nobody has rewritten into Rust but for which wrappers have been created that call into C. It works in Rust because the build.rs lets libraries do whatever they want during the build process, including invoking compilers for other languages. Various factors still make the Rust and Python story different (Python uses more mixed-language packages, the Rust demographic is more technically advanced, etc). But a big one is that in Rust, the FFI is defined in Rust. Recompiling just the Rust code gives you an updated FFI compatible with your version of Rust. In Python, the FFI is typically defined in C, so recompiling the python won't get you a compatible FFI. If Python did all FFI through something like ctypes, it would be much more smooth.

glandium 3y ago
> It works in Rust because the build.rs lets libraries do whatever they want during the build process, including invoking compilers for other languages. It's still its own mess. You'll find plenty of people having problems with openssl, for instance.
- oefrha 3y ago
  I have the pleasure of maintaining a reasonably popular -sys crate, and getting it working on Windows is an absolute nightmare, especially considering the competing toolchains & ABIs (msvc and gnu) and millions of ways the C dependency can be installed (it’s a big one, and users may choose a specific version, so we don’t bundle it), with no pkg-config telling you where to look. No idea how build.rs being able to run any Rust code is an advantage, it’s not like setup.py can’t run any Python code and shell out. In fact, bespoke code capable of spitting out inscrutable errors during `setup.py install` is the old way that newer tooling tries to avoid. Rust evangelism is puzzling.
  - oblio 3y ago
    Wait a second, I need to understand this better. If you cargo build, can that run a dependencies' build including trying to compile C and stuff?

the__alchemist 3y ago

As you might expect, compiling rust crates that use C libraries can lead to the inscrutable-block-of-text linker errors we know and love. I have been having a rough time with CUDA, CMSIS-DSP, Tensorflow, and OpenCV during the past few weeks. One of them requires LLVM=v15 to be installed; another requires v16+. A diff one requires an old version of rustc to be installed. On a diff one, when I posted on Github, the maintainers assured me the crate is, in fact, fine, and my system is misconfigured; phew!

oefrha 3y ago
Lol, as a maintainer of a reasonably popular and complex -sys crate, our crate is “fine” on Windows, at least in theory, and I’ve heard of successes using it. However, I can’t even port my own app depending on said -sys crate to Windows; there’s always a wall of linker errors. If you report a Windows problem to me, I won’t tell you it’s fine, I just throw my hands in the air.
- cshokie 3y ago
  Out of curiosity is it a “Windows is objectively difficult” problem, or a “Windows is not Linux and I know Linux best” problem? I’ve only begun using it so my expertise is limited, but I think vcpkg aims to help with some of these difficulties by shipping code as source and then running make on dependencies so they are guaranteed ABI compatible because the same compiler builds everything.
  - oefrha 3y ago
    That I don’t know Windows well is certainly a factor, but I think it’s at most 40% of the problem. Several notable issues, not all: - Competing msvc and gnu toolchains and ABIs, with native and Windows-first dependencies working better or exclusively with msvc, and *ix-first dependencies working better or exclusively with gnu, is a uniquely Windows situation. Which is which for a given build is also not clearly labeled most of the time. (You might mention glibc vs musl, but there’s basically nothing uniquely musl, and when you’re compiling for musl you can almost always get/compile everything for musl from the ground up.) - Confusing coexistence of x86 and x64 is another thing largely unique to Windows. (amd64 and arm64 are much more clearly separated in Apple land.) - Package management is a complete mess. Choco, scoop, win-get, nuget, vcpkg, ad hoc msi, ad hoc exe installer, ad hoc zip, etc. etc. There’s no pkg-config telling you where to look and which compiler flags to use. If you want to pick up a user-installed dep, you special case everything (e.g. looking in vcpkg path) and/or ask user to supply the search path(s) in env var(s). Anyway, shit mostly(tm) just work(tm) on *ix if you follow the happy path. There’s no happy path on Windows more often than not.

jchw 3y ago

I don't think this is really true. NodeJS has quite a similar problem.

shoo 3y ago

> The single biggest improvement for python ecosystem was the standardisation of a binary package format, wheel. I agree. Some people love to complain about python packaging. But from one perspective, it's arguably been a solved problem since wheels were introduced 10 years ago. The introduction of wheels was a massive step forward. Only depend on wheel archives, don't depend on packages that need to be built from source, and drag in all manner of exciting compile-time dependencies. If there's a package you want to depend on for your target platform, and the maintainers don't produce a prebuilt wheel archive for your platform -- well, set up a build server and build some wheels yourself, and host them somewhere, or pick a different platform.

sertbdfgbnfgsd 3y ago

> Yes, that's a fundamental reason python packaging is a mess. Python success is largely due to the availability of key mixed language packages. No other mainstream language package manager has to deal with this. Admittedly I'm not a python expert, but julia handles this just fine? It doesn't seem like it's a difficulty inherent to "mixed language packages". Somehow it appears to me that python's approach is just bad somehow.

rightbyte 3y ago
Then again jl seems to take 10s to start doing something non-trivial each time while modules are being compiled.
- sertbdfgbnfgsd 3y ago
  THEN AGAIN, If you want to restart repeatedly, for whatever reason (??), maybe you should just compile the modules once...? See e.g. [1] TLDR: There's a package called PrecompileTools.jl [2] which allows you to list: I want to compile and cache native code for the following methods on the following types. This isn't complicated stuff. [1] https://julialang.org/blog/2023/04/julia-1.9-highlights/#caching_of_native_code https://julialang.org/blog/2023/04/julia-1.9-highlights/#cac... [2] https://julialang.github.io/PrecompileTools.jl/stable/ https://julialang.github.io/PrecompileTools.jl/stable/
  - markkitti 3y ago
    Julia 1.9's native precompilation is definitely helpful in that regard but loading those native shared libraries (.so files on Linux) into Julia does take some to verify. If the main objective is to reduce time to load and time to first task then PackageCompiler.jl [3] is still the ultimate way to do so. Because Julia is a dynamic language, there are some complicated compilation issues such as invalidation and recompilation that arise. Adding new methods or packages may result in already compiled code no longer statically dispatching correctly requiring invalidation and recompilation of that code. It slightly more complicated than what you stated. It's "I want to compile and cache native code for the following methods on the following types in this particular environment". PackageCompiler.jl can wrap the entire environment into a single native library, the system image. [3] https://github.com/JuliaLang/PackageCompiler.jl https://github.com/JuliaLang/PackageCompiler.jl