4 ms·

One annoying thing about trying to install Makie (although it would be silly to blame Makie for it) is that installing it pulls a ton of compiled C/C++ binary d

by nsajko 3y ago

One annoying thing about trying to install Makie (although it would be silly to blame Makie for it) is that installing it pulls a ton of compiled C/C++ binary dependencies. I'm talking Fontconfig, Xorg(!?), Cairo, a bunch of audio and video codecs and compression libraries, among other things. Most/all of these are transitive dependencies, though. And all of these are already installed system-wide, independently of Julia. Although Julia is my favorite PL by far, it's sometimes shocking how easily the Julia community sidesteps traditional programming values. Often this turns out to be a good thing, but the above issue is surely not a good example. I'd very much prefer to leave the work of packaging all those to the Archlinux maintainers. EDIT: to be clear, this is more of a Julia-wide issue than a Makie issue. Or partly an issue with some of the Julia packages that Makie depends on. But Makie is perhaps the worst offender among the popular packages (transitively).

adgjlsfhk1 3y ago
The counterpoint here is that shipping the compiled binaries reduces a ton of uncertainty in terms of package version or compilation with non-standard options (cue potential reproducibility problems). Also different OSes will put libraries in different places (including different Linuxes). The last major benefit is that most OSes require root permissions to install packages, so Julia's approach works a lot better on systems where the user isn't root (e.g. most servers), and also means that the package manager doesn't have to know how to talk to the OS package manager for every OS that we support. Lastly, if you want use system versions of C/C++ packages, you can do so using the preference system (see https://docs.binarybuilder.org/stable/jll/#Non-dev'ed-JLL-packages) https://docs.binarybuilder.org/stable/jll/#Non-dev'ed-JLL-pa.... This isn't the default because it would make package installation a lot more annoying, but for deployment it very well may be what you want to do.
- deleted 3y ago
  [deleted]
krumbie 3y ago
Personally, I like the state now better than a couple years ago, where it was a struggle to get Cairo.jl working because it was interacting with users' systems in weird ways. It admittedly takes much more harddrive space, sometimes to a silly extent. I'm not sure if it's technically possible to have weak binary dependencies, I think if Cairo the C library declares binary deps, we have to add them all (hence Pango, Fontconfig, etc. etc.). By the way, we get very few installation issues nowadays. They usually have to do with either graphics drivers on Linux, or because people have somehow messed with their library load paths and pull in the wrong dylibs. From a maintainer's perspective it's nice not having to worry about all that too much.
eigenspace 3y ago
I pretty strongly disagree here. The fact of the matter is just that tonnes of bugs, incompatibilities, permission problems, and bad installation experiences every Linux user is intimately familiar with, often occur because package code has no control over which versions or whatever of system-level dependencies like Xorg, Fontconfi, Cairo, etc. are already installed. This also makes it much much easier for package authors to write code which works on multiple OSs without being experts on the details of all those OSs. Now, if there was no way around this, then you might have a point, but users always can opt-out of binary artifacts like that and point julia to their system libraries instead of the default vendoring of binary artifacts[1]. I think this is the right way to do it. Experts who want to get involved in the nitty gritty details of this stuff and who are also equipped to debug and deal with failures, can opt into using their system libraries, and the rest of us can pay some hard drive space and get a better out-of-the-box experience. ______________________________ [1]: https://pkgdocs.julialang.org/v1/artifacts/#Overriding-artifact-locations https://pkgdocs.julialang.org/v1/artifacts/#Overriding-artif...
- WanderPanda 3y ago
  I agree. Since nowadays the disk space saving argument is almost irrelevant the only thing in favor of shared libraries seems to be the automatic improvement of all dependents with one update. But I think history has shown that this is a major source of incompatibilities and since Julia software is not really supposed to be running public facing services the security argument is moot anyways.
  - nsajko 3y ago
    > since Julia software is not really supposed to be running public facing services the security argument is moot anyways 1. both Julialang.org and Juliahub.com run Julia as far as I understand 2. even if your premise were true, it wouldn't be acceptable for the Julia project to condone pwning of Julia users
  - eigenspace 3y ago
    > the security argument is moot anyways Sorry, I’m missing something, what security concerns are you referring to?

by nsajko 3y ago

adgjlsfhk1 3y ago

The counterpoint here is that shipping the compiled binaries reduces a ton of uncertainty in terms of package version or compilation with non-standard options (cue potential reproducibility problems). Also different OSes will put libraries in different places (including different Linuxes). The last major benefit is that most OSes require root permissions to install packages, so Julia's approach works a lot better on systems where the user isn't root (e.g. most servers), and also means that the package manager doesn't have to know how to talk to the OS package manager for every OS that we support. Lastly, if you want use system versions of C/C++ packages, you can do so using the preference system (see https://docs.binarybuilder.org/stable/jll/#Non-dev'ed-JLL-packages) https://docs.binarybuilder.org/stable/jll/#Non-dev'ed-JLL-pa.... This isn't the default because it would make package installation a lot more annoying, but for deployment it very well may be what you want to do.

deleted 3y ago
[deleted]

krumbie 3y ago

Personally, I like the state now better than a couple years ago, where it was a struggle to get Cairo.jl working because it was interacting with users' systems in weird ways. It admittedly takes much more harddrive space, sometimes to a silly extent. I'm not sure if it's technically possible to have weak binary dependencies, I think if Cairo the C library declares binary deps, we have to add them all (hence Pango, Fontconfig, etc. etc.). By the way, we get very few installation issues nowadays. They usually have to do with either graphics drivers on Linux, or because people have somehow messed with their library load paths and pull in the wrong dylibs. From a maintainer's perspective it's nice not having to worry about all that too much.

eigenspace 3y ago

I pretty strongly disagree here. The fact of the matter is just that tonnes of bugs, incompatibilities, permission problems, and bad installation experiences every Linux user is intimately familiar with, often occur because package code has no control over which versions or whatever of system-level dependencies like Xorg, Fontconfi, Cairo, etc. are already installed. This also makes it much much easier for package authors to write code which works on multiple OSs without being experts on the details of all those OSs. Now, if there was no way around this, then you might have a point, but users always can opt-out of binary artifacts like that and point julia to their system libraries instead of the default vendoring of binary artifacts[1]. I think this is the right way to do it. Experts who want to get involved in the nitty gritty details of this stuff and who are also equipped to debug and deal with failures, can opt into using their system libraries, and the rest of us can pay some hard drive space and get a better out-of-the-box experience. ______________________________ [1]: https://pkgdocs.julialang.org/v1/artifacts/#Overriding-artifact-locations https://pkgdocs.julialang.org/v1/artifacts/#Overriding-artif...

WanderPanda 3y ago
I agree. Since nowadays the disk space saving argument is almost irrelevant the only thing in favor of shared libraries seems to be the automatic improvement of all dependents with one update. But I think history has shown that this is a major source of incompatibilities and since Julia software is not really supposed to be running public facing services the security argument is moot anyways.
- nsajko 3y ago
  > since Julia software is not really supposed to be running public facing services the security argument is moot anyways 1. both Julialang.org and Juliahub.com run Julia as far as I understand 2. even if your premise were true, it wouldn't be acceptable for the Julia project to condone pwning of Julia users
- eigenspace 3y ago
  > the security argument is moot anyways Sorry, I’m missing something, what security concerns are you referring to?