5 ms·

At the Mountains of Madness

153 points by wwilson 2y ago

wwilson 2y ago
Post author here. Feel free to ask me any questions about the piece of software that I most regret having had to write.
- pxc 2y ago
  Why did you want to use Nix to make impure binaries for other distros? Much of the appeal of it for me is using it to distribute software in a more reliably portable way, but that of course always means shipping a whole chunk of /nix/store, once way or another. What made your team/company want to use Nix to build binaries and then strip them down for old-fashioned, dependency hell-ish distribution? Why not install Nix on your target systems or use Nix bundle, generate containers, etc.?
- Klaster_1 2y ago
  No questions about Madness, but I really enjoyed the article tone and playfulness. Thank you.
- swayvil 2y ago
  Dig the purple anteater pix.
- jcgrillo 2y ago
  Also not a question, just want to say that "crt glow" (or maybe "Cerenkov glow"?) effect upon hovering over a link is awesome.
- limaoscarjuliet 2y ago
  Been there, done that. In my case, I symlinked myself out of this mess rather than modify ELF.
- dasyatidprime 2y ago
  I would like to half-seriously recommend that you overwrite a different character than the first when mangling the environment variable name. Specifically one beyond the third, so as to stay within the LD_ namespace (not yours exactly, but at least easier to keep track of and more justifiably excludable from random applications) and deny someone ten years from now the exciting journey of figuring out why their MD_PRELOAD environment variable is overwitten with garbage on some systems. How do you feel about LD_PRELOAF? Also it's probably better to leave LD_PRELOAD properly unset rather than just null if it was unset before; in particular I wonder if empty-but-set might still trip some software's “someone is playing tricks” alarms. There are probably other ways this is less than robust… (hi, I kind of have a Thing for GNU and Linux innards sometimes)
  - wwilson 2y ago
    Good suggestion on leaving LD_PRELOAD unset if it was previously unset. We will fix that. I’m torn on whether MD_PRELOAD or LD_PRELOAF is more obnoxious to other programs. Fun fact: A previous version of this program used an even more inscrutable `env[0][0]+=1`, which is great as a sort of multilingual C/English pun, but terrible in the way that all “clever” code is terrible.
- foobiekr 2y ago
  It's a shame that DLL hell was never resolved in the obvious way: deduplication of identical libraries through cryptographic hashes. Containers basically threw away any hope of sharing the bytes on disk - and more importantly _in ram_. Disk bytes are cheap, ram bytes are not, let alone TLB space, branch predictor context, and so on. There was a middle ground possible at one point where containers actually were packaged with all of their dependencies, but a container installer would fragment this assembly into cryptographically verifiable share dependencies, but we lost that because it was hard.
  - xenophonf 2y ago
    > deduplication of identical libraries through cryptographic hashes Isn't that how the .NET CLR's global assembly cache works?
    - foobiekr 2y ago
      A lot of things work that way because it’s obvious to everyone except the container runtimes on Linux.
      - pxc 2y ago
        The container runtimes have to cope with Dockerfiles and similar, which know nothing about packages. To get the kind of granularity you want here, you have to do actual packaging work, which is the thing Docker sold everyone on avoiding. If you are willing to do that kind of packaging work you can get the best of both worlds today with Nix or Guix. But containers are attractive because you can chuck whatever pathological build process your developers have evolved over the decades into a Containerfile and it'll mostly work.
  - rbanffy 2y ago
    > deduplication of identical libraries through cryptographic hashes Or, maybe, adding a version string to the file name, so, if you were compiled with data structures for libFoo1 (which you found on libFoo.h provided by libFoo1-devel) you’ll link to libFoo1 and not libFoo or libFoo2.
- foobiekr 2y ago
  As an aside, a lot of people don't know about ldd, and introducing it to them is very cool, but it should almost always come with a warning - maybe add a note that people should be careful with ldd - ldd _may_ execute arbitrary code. This is in the ldd man page, but most people never read documentation. It is unsafe to use on any binary you're not otherwise believed safe.
  - wwilson 2y ago
    Great point! I'll update the post to mention that.
- adamgordonbell 2y ago
  Love it. I came to this same insight about nix and containers being two approaches to a dynamic linking work around, but via a different path, of building my own little container runtime. Feels like we are building things who's original purpose is now holding us back, but path dependence leaves us stuck wrapping abstractions in other abstractions.
- trod123 2y ago
  Hi Will, I'm curious what your thoughts are about the Nix uniqueness problem, and the characterization of failures, or lack thereof under undefined behavior's failure domains. Exception handling generally requires a defined and deterministic state which can't be guaranteed given design choices to resolve DLL hell under Nix (i.e. its a stochastic process). I mention this since it is a similar form of the problem you mention in writing this piece of software, that can lead to madness. Also, operationally, the troubleshooting problem-space of keeping things running, segments nicely into deterministic and non-deterministic regions; which the latter ends up costing orders of magnitude more as a function of time to resolve since you can't perturb individual subsystems to test for correct function, without determinism and time in-variance (as system's properties), testing piecemeal has contradictions in stochastic processes. Hashing by rigorous definition is non-unique (i.e. its like navigating a circle), and there is no proof of uniformity. So problems in this space would be in the latter region. While, there are heuristics from cryptography that suggest using factional cubic roots to initialize the fields brings more uniformity to the examined space than not, there is no proof of such. When building resilient systems, engineers often try to remove any brittle features that promote failures. Interestingly, as a side note, ldd output injects non-determinism into the pipe by flattening empty columns non-deterministically (i.e. if you ldd ssh client, you'll see the null state for each input to output has more than a single meaning/edge on the traversal depending on object type, this ends up violating the 1:1 unique input-output state graph/map required for determinism as a property, though it won't be evident until you run use it as an input that problematically maps later in automation (i.e. grepping the output with RegEx will silently fail, providing what looks like legitimate output if one doesn't look too closely). PaX ended up forking the project with the fix, because the maintainers refused to admit the problem (reported 2016, forked in 2018), the bug remains in all current versions of ldd (to my knowledge). While based in theory, these types of problems crop up everywhere in computation and few seem to recognize them. Working with system's properties, and whether they are preserved; informs on whether the system can be safely and consistently used in later automated processes, as well as maintained at cheap cost. Businesses generally need a supportable and defensible infrastructure.
- jadbox 2y ago
  I'd like to express that while this article is WAY outside my wheelhouse, but I liked the writing style and the AI illustrations felt like they were emotionally additive to the section rather just a distraction (to me). Also, my head hurts trying to still understand this cursed thing: https://github.com/antithesishq/madness/blob/main/pkgs/madness.nix https://github.com/antithesishq/madness/blob/main/pkgs/madne...
- colinsane 2y ago
  > This minimal meta-loader will totally work if you invoke it directly like `$ meta_loader.sh foo`, and it will totally not work if you hardcode its path (or a symlink to it) in the ELF headers of a binary. why not have `foo` be a shell script which invokes the meta loader on the "real" foo? like: ``` #!/bin/sh # file: /bin/foo # invoke the real "foo" (renamed e.g. ".foo-wrapped" or "/libexec/foo" or anything else easy for the loader to locate but unlikely to be invoked accidentally) exec meta_loader.sh .foo-wrapped "$@" ``` it's a common enough idiom that nixpkgs provides the `wrapProgram` function to generate these kinds of wrapper scripts during your build: even with an option to build a statically-linked binary wrapper instead of a shell-script wrapper (`makeBinaryWrapper`).
- o11c 2y ago
  Somehow your website has semi-broken scrolling, which is impressive since that normally only happens when Javascript is enabled. Also, please QEFS (quote every string) in your shell script fragments.
NoraCodes 2y ago
What is that abominable diffusion output doing at the top of an otherwise interesting article?
- wwilson 2y ago
  Our artist is on vacation, and some fool gave the CEO access to Midjourney.
  - imagineerschool 2y ago
    These synthetic artifacts will come to be regarded as psychological asbestos. Please consider labelling it, and giving provenance data. And protecting public sanity by putting it behind a clickwall.
  - bloopernova 2y ago
    In my opinion, generative AI pictures make a blog post feel cheaper and less truthful. Just my view, I fully accept that I'm probably in a minority of opinion.
    - pizzalife 2y ago
      I agree with you since it adds absolutely no value to the article. Technical articles don't need unrelated pictures that add huge page breaks.
  - didsomeonesay 2y ago
    FWIW, I enjoyed how the pictures were adding a little theme, were consistent and broke up the reading nicely without being too "noisy" (compared to e.g. technical articles full of meme pictures).
- knowaveragejoe 2y ago
  What is this trifling and snobby driveby commentary doing in the comments of an otherwise interesting article?
DoreenMichele 2y ago
tl;dr: we are open-sourcing an internal tool that solves a problem that we think many NixOS shops are likely to run into. The rest of this post is just the story of how we came to write this tool, which is totally a skippable story. The tool happens to be called Madness, thus the Lovecraftian reference in this piece. Madness enables you to easily run the same binary on NixOS and non-NixOS systems https://github.com/antithesishq/madness https://github.com/antithesishq/madness
finnh 2y ago
Every time I look at NixOS, I think that it perfectly solves a problem that I only have once every 5 years, when buying a new computer. I think I even looked into it once to automate that exact process, but that idea fell apart at the first line of Nix syntax. I'll stick with OSX and `brew bundle` I guess... But then I read a piece like this and remember that some people do have to plumb the depths of C/C++ linkers, and I'm glad I'm not one of them. Great post! FWIW I always want to know the prompt text when seeing an AI-generated image, I wish there were a convention around that.
- shrx 2y ago
  Some generators like Automatic1111 embed the prompt in the image metadata.
  - Filligree 2y ago
    ComfyUI embeds the entire graph, but only if you use its internal “save image” function.
- api 2y ago
  I think it solves a problem that shouldn't exist: managing complex state in an operating system. OSes should be mostly immutable. Apps should own their state. Everything else should be in a neat tidy box that is portable. Mobile almost gets this right. The idea of installing things "on" the OS needs to die, badly. It's a security and privacy nightmare because it means everything more or less has root, and it makes every OS install a special snowflake that is under the hood a giant mixed pile of shit.
  - akvadrako 2y ago
    Image based OSes like Fedora silverblue are largely immutable. It's a much better model and think it's catching on.
  - TacticalCoder 2y ago
    > ... because it means everything more or less has root, and it makes every OS install a special snowflake that is under the hood a giant mixed pile of shit. In many Linux distros systemd as PID 1 comes to mind...
AdamH12113 2y ago
I'm still confused why static linking isn't a more common solution to versioning issues. Software developers normally have no problem using an order of magnitude more resources to solve organizational problems. Is there any technical advantage to dynamic linking other than smaller binaries and maybe slightly faster load times from disk?
- wwilson 2y ago
  Biggest advantages I know of for dynamic linking: * You can use the LD_PRELOAD trick to override behavior at runtime. * You can run with entirely different implementations of the dynamically linked library in different places. * Software can pick up interface-compatible upgrades to its dependencies without being re-compiled and distributed again. We use all three of these tricks in our SDKs, FWIW. But it is still a giant pain in the ass.
- klodolph 2y ago
  IIRC the underlying implementation may be different on other systems. I think in particular, DNS resolution. Linux is the only system where static linking all the way really makes any sense. For most systems, you don’t get a stable syscall ABI. Instead, you get a stable ABI to the library which does syscalls for you… Windows has kernel32.dll, macOS has libSystem. Note that on Linux, the vDSO is dynamically linked. Compilation speed is a big plus. For large projects, linking time can easily dominate the time needed for incremental rebuilds. Due to tooling issues, PIE is a lot easier with dynamic linking, and this gives you better ASLR. These issues are solvable, but it’s a lot easier to get PIE if you use dynamic linking. If you want static PIE, you need to compile all your static libs as PIE—doable, but you don’t get it out of the box.
- anyfoo 2y ago
  Static linking essentially freezes not only the ABI to the kernel, but many implementation details of the linked libraries as well. Including, for example, how library client code talks to daemons, or formats of files directly read in by the libraries. The timezone configuration would be one instance, or things related to NSS. It's really not viable in a lot of cases, unless you like rebuilding (or at least relinking) with every system software update. And then there's of course the memory savings. macOS and iOS for example have giant "shared caches" which are mapped into all processes and comprise of all the system libraries. (Other OSs often do this on the individual shared library level.) With static linking, you'd instead have many copies of lots of potentially-but-not-necessarily identical library code pages in DRAM.
klodolph 2y ago
> No such file or directory Anyone who’s run into this problem remembers it! (This isn’t a Nix problem—this is just the baffling errors you get because a.out exists, but one of the libraries it needs does not, and the error message doesn’t distinguish that case.) Anyway, Nix. Nix has the Nix way of building things. Nix doesn’t give you standard tools. It gives you wrappers around the standard tools that force you to do things a certain way. Part of that is futzing around with RPATH—because Nix stores everything in an unusual location. The user experience around this is awful, if you ever run into a case where Nix’s tooling doesn’t automatically do the right thing for you. It’s not just RPATH, but also other paths. What’s the solution? Honestly—I think it would make sense for Nix to have a “cross compilation” mode where you tell it to cross-compile for other Linuxes. You know, something like pkgsCross.x86_64-generic-linux. This comes with all the cross-compilation headaches, but you know what? You are cross-compiling.
- throwway120385 2y ago
  Yeah I wish the ld-linux.so interpreter would actually indicate when a file it's trying to link wasn't found. Something like "unable to locate shared object BLAH" would go a long way. It's like a rite of passage the first time you debug something like that.
- deleted 2y ago
  [deleted]
- nyarlathotep_ 2y ago
  I've only encountered this once and I've not forgotten it. Downloaded a release of some binary for Linux, but I'd downloaded the FreeBSD built binary. Was lost until I explored in the same fashion as the author.
- lilyball 2y ago
  Nix does have tools for running stuff in an FHS container. Something I have considered but not yet attempted is to use this to wrap the build such that building the binary happens in the FHS container (using the unwrapped versions of the compiler and associated tooling).
sjburt 2y ago
It seems like every article about nix goes on and on about DLL hell. I've been using Debian/Ubuntu for 15+ years and never really experienced dependency hell. I guess maybe this is thanks to hard work by Debian maintainers and rarely needing to run a bleeding edge library, but also, why do we need to run bleeding edge versions of everything and then invent an incredibly complicated scheme to keep multiple copies of each library, most of which are completely compatible with each other? And then when there's a security problem, who goes and checks that every version of every dependency of every application has actually been patched and updated? Why would I want to roll a system back to an (definitely insecure) state of a few months ago? What problem does Nix solve that SO numbers (properly used) doesn't? I have many of the same questions about Snap and even Docker.
- HideousKojima 2y ago
  I run into DLL hell any time I try to install some software that isn't in some sort of package repository or other happy path. Most recent example I can think of was about a year ago when I helped my father-in-law install Klipper on an RPi4 so that he could do input shaping on his 3d prints. All of the guides and documentation seemed to assume that you were using a specific version of Linux on the RPi, and if you weren't (like my FIL) then welcome to dependency hell. Took several hours of pulling my hair out to resolve them all, for a non-developer it would have been impossible.
  - michaelmrose 2y ago
    So the installation story for klipper is hilariously bad. Its a shell script that apt installs and pip installs things as root. It is the very definition of non-portable. The distribution story for Python apps in general is fairly bad and the developer really takes it to the next level. This isn't DLL hell as a function of leaving the happy path its DLL hell as a function of burning the happy path whilst dancing naked on the ashes of the greenery that once was the happy path. See Calibre for willful disobedience vs instead of insanity. It eshews system libraries and uses a shell script to install but it dumps everything including Python libs in a directory.
bbor 2y ago
I love what they’re going for, but I couldn’t help but react negatively at finding out that I had been hyped up for a post on some small technical topic for an OS I don’t know of. Maybe title it “At the Mountains of NIXos Madness”? But then again I’m just a grouch! Well written article regardless, from what I was able to get out of it
pizzalife 2y ago
Calling binaries using ld-linux used to be a popular way to get around noexec on filesystems, since the libraries are usually in a place that is executable..
dhash 2y ago
I loved this post, and patchelf is a real gem of a utility.
georgewsinger 2y ago
=======Technical Summary======== Here's a problem with NixOS: 1. Suppose we have a `./nixos_binary_program_with_glibc-newer` compiled on a NixOS machine against bleeding edge `glibc-newer`. 2. `./nixos_binary_program_with_glibc-newer` will have `/nix/store/glibc-newer/linux-ld.so` path hardcoded into its ELF header which will be used when the program launches to find all of the program's shared libraries, and so forth. (And this is a fact that `ldd` will obfuscate!). 3. When `./nixos_binary_program_with_glibc-newer` is distributed to machines which use `glibc-older` instead of `glibc-newer`, the hardcoded `linux-ld.so` from (2) will fail to be found, leading to a launch error. 4. (3) will also happen on machines which don't use nix in the first place. =======Will's Solution======== 1. Use `patchelf` to hardcode a standard FHS `ld-linux.so` location into `nixos_binary_program_with_glibc-newer`'s ELF header (using e.g. `/lib64/ld-linux-x86-64.so.2` as the path) 2. Use a metaloader to launch `nixos_binary_program_with_glibc-newer` with an augmented `RPATH` which has a bunch of different `/nix/store/glibc-newer` paths, so that nix machines can find a suitable `ld-linux.so` to launch the program with. This will make `nixos_binary_program_with_glibc-newer` work on any machine, including both non-nix machines and nix machines (which might be running older versions of glibc by default)!
banish-m4 2y ago
Completely missed that Nix solves RPM dependency hell, which is a superset of the shared library hell. Another problem not solved by NixOS and most other distros is conflating and mixing dependencies in a messy, fragile way rather than having a clear separation between the OS and add-ons that FreeBSD and others have. Congruent with this is proper configuration and lifecycle management. I'm also wondering about the security of this RPATH approach, if it does or doesn't introduce vulnerabilities.
dreamcompiler 2y ago
I once had a robotic cat litter box that cleaned itself. Except once a week it would get clogged and I would have to spend a quality hour disassembling it, scrubbing off the feces embedded on the delicate parts, and reassembling it. Every two weeks when it heated up its artificial litter to dry, it would have missed a small piece of cat shit that when baked, filled my house with an aroma that I would not recommend you even try to imagine. And of course it needed special, expensive supplies that you had to buy from the manufacturer because the bottles had numbered chips. I eventually threw the damn thing out and now I just use a manual litter box. Takes 15 seconds a day to clean. It's a chore but it's a small predictable chore. When I read about NixOS I remember that robot litter box. It seems like it solves a real problem of difficulty X but it brings five brand new problems from a parallel universe you didn't know existed and they're all written in an indecipherable language and have difficulty 10X.
- Zababa 2y ago
  In my experience it kind of just works. Takes a bit more maintenance than arch, around an hour a month at most. I had to reinstall it once and it was way easier than my arch, since way more stuff is in text files I can commit. Also, since it's very very easy to rollback to a previous version, managing unpredictable issues is easy. I have a colleague that lost lots of time to an arch kernel panic during an update that required a reinstall. On NixOS I can reboot, choose the last derivation, work and fix that if/when I want.
- NikkiA 2y ago
  And then some Go fanatic will come along and use this experience to explain why it makes perfect sense to static link bits of the standard library into every single binary ever produced, and dynamic linking was the worst idea since mankind's ancestors crawled out of the ocean and past the whales crawling back in.
- brookst 2y ago
  You are an excellent writer. I have not use Nix, but I have had a similar bad experience with a similar but different automatic litter box. Your “small predictable chore” point is spot on: how much human grief is created by elaborate, expensive, unreliable solutions to minor annoyances?

153 points by wwilson 2y ago

wwilson 2y ago

Post author here. Feel free to ask me any questions about the piece of software that I most regret having had to write.

pxc 2y ago
Why did you want to use Nix to make impure binaries for other distros? Much of the appeal of it for me is using it to distribute software in a more reliably portable way, but that of course always means shipping a whole chunk of /nix/store, once way or another. What made your team/company want to use Nix to build binaries and then strip them down for old-fashioned, dependency hell-ish distribution? Why not install Nix on your target systems or use Nix bundle, generate containers, etc.?
Klaster_1 2y ago
No questions about Madness, but I really enjoyed the article tone and playfulness. Thank you.
swayvil 2y ago
Dig the purple anteater pix.
jcgrillo 2y ago
Also not a question, just want to say that "crt glow" (or maybe "Cerenkov glow"?) effect upon hovering over a link is awesome.
limaoscarjuliet 2y ago
Been there, done that. In my case, I symlinked myself out of this mess rather than modify ELF.
dasyatidprime 2y ago
I would like to half-seriously recommend that you overwrite a different character than the first when mangling the environment variable name. Specifically one beyond the third, so as to stay within the LD_ namespace (not yours exactly, but at least easier to keep track of and more justifiably excludable from random applications) and deny someone ten years from now the exciting journey of figuring out why their MD_PRELOAD environment variable is overwitten with garbage on some systems. How do you feel about LD_PRELOAF? Also it's probably better to leave LD_PRELOAD properly unset rather than just null if it was unset before; in particular I wonder if empty-but-set might still trip some software's “someone is playing tricks” alarms. There are probably other ways this is less than robust… (hi, I kind of have a Thing for GNU and Linux innards sometimes)
- wwilson 2y ago
  Good suggestion on leaving LD_PRELOAD unset if it was previously unset. We will fix that. I’m torn on whether MD_PRELOAD or LD_PRELOAF is more obnoxious to other programs. Fun fact: A previous version of this program used an even more inscrutable `env[0][0]+=1`, which is great as a sort of multilingual C/English pun, but terrible in the way that all “clever” code is terrible.
foobiekr 2y ago
It's a shame that DLL hell was never resolved in the obvious way: deduplication of identical libraries through cryptographic hashes. Containers basically threw away any hope of sharing the bytes on disk - and more importantly _in ram_. Disk bytes are cheap, ram bytes are not, let alone TLB space, branch predictor context, and so on. There was a middle ground possible at one point where containers actually were packaged with all of their dependencies, but a container installer would fragment this assembly into cryptographically verifiable share dependencies, but we lost that because it was hard.
- xenophonf 2y ago
  > deduplication of identical libraries through cryptographic hashes Isn't that how the .NET CLR's global assembly cache works?
  - foobiekr 2y ago
    A lot of things work that way because it’s obvious to everyone except the container runtimes on Linux.
    - pxc 2y ago
      The container runtimes have to cope with Dockerfiles and similar, which know nothing about packages. To get the kind of granularity you want here, you have to do actual packaging work, which is the thing Docker sold everyone on avoiding. If you are willing to do that kind of packaging work you can get the best of both worlds today with Nix or Guix. But containers are attractive because you can chuck whatever pathological build process your developers have evolved over the decades into a Containerfile and it'll mostly work.
- rbanffy 2y ago
  > deduplication of identical libraries through cryptographic hashes Or, maybe, adding a version string to the file name, so, if you were compiled with data structures for libFoo1 (which you found on libFoo.h provided by libFoo1-devel) you’ll link to libFoo1 and not libFoo or libFoo2.
foobiekr 2y ago
As an aside, a lot of people don't know about ldd, and introducing it to them is very cool, but it should almost always come with a warning - maybe add a note that people should be careful with ldd - ldd _may_ execute arbitrary code. This is in the ldd man page, but most people never read documentation. It is unsafe to use on any binary you're not otherwise believed safe.
- wwilson 2y ago
  Great point! I'll update the post to mention that.
adamgordonbell 2y ago
Love it. I came to this same insight about nix and containers being two approaches to a dynamic linking work around, but via a different path, of building my own little container runtime. Feels like we are building things who's original purpose is now holding us back, but path dependence leaves us stuck wrapping abstractions in other abstractions.
trod123 2y ago
Hi Will, I'm curious what your thoughts are about the Nix uniqueness problem, and the characterization of failures, or lack thereof under undefined behavior's failure domains. Exception handling generally requires a defined and deterministic state which can't be guaranteed given design choices to resolve DLL hell under Nix (i.e. its a stochastic process). I mention this since it is a similar form of the problem you mention in writing this piece of software, that can lead to madness. Also, operationally, the troubleshooting problem-space of keeping things running, segments nicely into deterministic and non-deterministic regions; which the latter ends up costing orders of magnitude more as a function of time to resolve since you can't perturb individual subsystems to test for correct function, without determinism and time in-variance (as system's properties), testing piecemeal has contradictions in stochastic processes. Hashing by rigorous definition is non-unique (i.e. its like navigating a circle), and there is no proof of uniformity. So problems in this space would be in the latter region. While, there are heuristics from cryptography that suggest using factional cubic roots to initialize the fields brings more uniformity to the examined space than not, there is no proof of such. When building resilient systems, engineers often try to remove any brittle features that promote failures. Interestingly, as a side note, ldd output injects non-determinism into the pipe by flattening empty columns non-deterministically (i.e. if you ldd ssh client, you'll see the null state for each input to output has more than a single meaning/edge on the traversal depending on object type, this ends up violating the 1:1 unique input-output state graph/map required for determinism as a property, though it won't be evident until you run use it as an input that problematically maps later in automation (i.e. grepping the output with RegEx will silently fail, providing what looks like legitimate output if one doesn't look too closely). PaX ended up forking the project with the fix, because the maintainers refused to admit the problem (reported 2016, forked in 2018), the bug remains in all current versions of ldd (to my knowledge). While based in theory, these types of problems crop up everywhere in computation and few seem to recognize them. Working with system's properties, and whether they are preserved; informs on whether the system can be safely and consistently used in later automated processes, as well as maintained at cheap cost. Businesses generally need a supportable and defensible infrastructure.
jadbox 2y ago
I'd like to express that while this article is WAY outside my wheelhouse, but I liked the writing style and the AI illustrations felt like they were emotionally additive to the section rather just a distraction (to me). Also, my head hurts trying to still understand this cursed thing: https://github.com/antithesishq/madness/blob/main/pkgs/madness.nix https://github.com/antithesishq/madness/blob/main/pkgs/madne...
colinsane 2y ago
> This minimal meta-loader will totally work if you invoke it directly like `$ meta_loader.sh foo`, and it will totally not work if you hardcode its path (or a symlink to it) in the ELF headers of a binary. why not have `foo` be a shell script which invokes the meta loader on the "real" foo? like: ``` #!/bin/sh # file: /bin/foo # invoke the real "foo" (renamed e.g. ".foo-wrapped" or "/libexec/foo" or anything else easy for the loader to locate but unlikely to be invoked accidentally) exec meta_loader.sh .foo-wrapped "$@" ``` it's a common enough idiom that nixpkgs provides the `wrapProgram` function to generate these kinds of wrapper scripts during your build: even with an option to build a statically-linked binary wrapper instead of a shell-script wrapper (`makeBinaryWrapper`).
o11c 2y ago
Somehow your website has semi-broken scrolling, which is impressive since that normally only happens when Javascript is enabled. Also, please QEFS (quote every string) in your shell script fragments.

NoraCodes 2y ago

What is that abominable diffusion output doing at the top of an otherwise interesting article?

wwilson 2y ago
Our artist is on vacation, and some fool gave the CEO access to Midjourney.
- imagineerschool 2y ago
  These synthetic artifacts will come to be regarded as psychological asbestos. Please consider labelling it, and giving provenance data. And protecting public sanity by putting it behind a clickwall.
- bloopernova 2y ago
  In my opinion, generative AI pictures make a blog post feel cheaper and less truthful. Just my view, I fully accept that I'm probably in a minority of opinion.
  - pizzalife 2y ago
    I agree with you since it adds absolutely no value to the article. Technical articles don't need unrelated pictures that add huge page breaks.
- didsomeonesay 2y ago
  FWIW, I enjoyed how the pictures were adding a little theme, were consistent and broke up the reading nicely without being too "noisy" (compared to e.g. technical articles full of meme pictures).
knowaveragejoe 2y ago
What is this trifling and snobby driveby commentary doing in the comments of an otherwise interesting article?

DoreenMichele 2y ago

tl;dr: we are open-sourcing an internal tool that solves a problem that we think many NixOS shops are likely to run into. The rest of this post is just the story of how we came to write this tool, which is totally a skippable story. The tool happens to be called Madness, thus the Lovecraftian reference in this piece. Madness enables you to easily run the same binary on NixOS and non-NixOS systems https://github.com/antithesishq/madness https://github.com/antithesishq/madness

finnh 2y ago

Every time I look at NixOS, I think that it perfectly solves a problem that I only have once every 5 years, when buying a new computer. I think I even looked into it once to automate that exact process, but that idea fell apart at the first line of Nix syntax. I'll stick with OSX and `brew bundle` I guess... But then I read a piece like this and remember that some people do have to plumb the depths of C/C++ linkers, and I'm glad I'm not one of them. Great post! FWIW I always want to know the prompt text when seeing an AI-generated image, I wish there were a convention around that.

shrx 2y ago
Some generators like Automatic1111 embed the prompt in the image metadata.
- Filligree 2y ago
  ComfyUI embeds the entire graph, but only if you use its internal “save image” function.
api 2y ago
I think it solves a problem that shouldn't exist: managing complex state in an operating system. OSes should be mostly immutable. Apps should own their state. Everything else should be in a neat tidy box that is portable. Mobile almost gets this right. The idea of installing things "on" the OS needs to die, badly. It's a security and privacy nightmare because it means everything more or less has root, and it makes every OS install a special snowflake that is under the hood a giant mixed pile of shit.
- akvadrako 2y ago
  Image based OSes like Fedora silverblue are largely immutable. It's a much better model and think it's catching on.
- TacticalCoder 2y ago
  > ... because it means everything more or less has root, and it makes every OS install a special snowflake that is under the hood a giant mixed pile of shit. In many Linux distros systemd as PID 1 comes to mind...

AdamH12113 2y ago

I'm still confused why static linking isn't a more common solution to versioning issues. Software developers normally have no problem using an order of magnitude more resources to solve organizational problems. Is there any technical advantage to dynamic linking other than smaller binaries and maybe slightly faster load times from disk?

wwilson 2y ago
Biggest advantages I know of for dynamic linking: * You can use the LD_PRELOAD trick to override behavior at runtime. * You can run with entirely different implementations of the dynamically linked library in different places. * Software can pick up interface-compatible upgrades to its dependencies without being re-compiled and distributed again. We use all three of these tricks in our SDKs, FWIW. But it is still a giant pain in the ass.
klodolph 2y ago
IIRC the underlying implementation may be different on other systems. I think in particular, DNS resolution. Linux is the only system where static linking all the way really makes any sense. For most systems, you don’t get a stable syscall ABI. Instead, you get a stable ABI to the library which does syscalls for you… Windows has kernel32.dll, macOS has libSystem. Note that on Linux, the vDSO is dynamically linked. Compilation speed is a big plus. For large projects, linking time can easily dominate the time needed for incremental rebuilds. Due to tooling issues, PIE is a lot easier with dynamic linking, and this gives you better ASLR. These issues are solvable, but it’s a lot easier to get PIE if you use dynamic linking. If you want static PIE, you need to compile all your static libs as PIE—doable, but you don’t get it out of the box.
anyfoo 2y ago
Static linking essentially freezes not only the ABI to the kernel, but many implementation details of the linked libraries as well. Including, for example, how library client code talks to daemons, or formats of files directly read in by the libraries. The timezone configuration would be one instance, or things related to NSS. It's really not viable in a lot of cases, unless you like rebuilding (or at least relinking) with every system software update. And then there's of course the memory savings. macOS and iOS for example have giant "shared caches" which are mapped into all processes and comprise of all the system libraries. (Other OSs often do this on the individual shared library level.) With static linking, you'd instead have many copies of lots of potentially-but-not-necessarily identical library code pages in DRAM.

klodolph 2y ago

> No such file or directory Anyone who’s run into this problem remembers it! (This isn’t a Nix problem—this is just the baffling errors you get because a.out exists, but one of the libraries it needs does not, and the error message doesn’t distinguish that case.) Anyway, Nix. Nix has the Nix way of building things. Nix doesn’t give you standard tools. It gives you wrappers around the standard tools that force you to do things a certain way. Part of that is futzing around with RPATH—because Nix stores everything in an unusual location. The user experience around this is awful, if you ever run into a case where Nix’s tooling doesn’t automatically do the right thing for you. It’s not just RPATH, but also other paths. What’s the solution? Honestly—I think it would make sense for Nix to have a “cross compilation” mode where you tell it to cross-compile for other Linuxes. You know, something like pkgsCross.x86_64-generic-linux. This comes with all the cross-compilation headaches, but you know what? You are cross-compiling.

throwway120385 2y ago
Yeah I wish the ld-linux.so interpreter would actually indicate when a file it's trying to link wasn't found. Something like "unable to locate shared object BLAH" would go a long way. It's like a rite of passage the first time you debug something like that.
deleted 2y ago
[deleted]
nyarlathotep_ 2y ago
I've only encountered this once and I've not forgotten it. Downloaded a release of some binary for Linux, but I'd downloaded the FreeBSD built binary. Was lost until I explored in the same fashion as the author.
lilyball 2y ago
Nix does have tools for running stuff in an FHS container. Something I have considered but not yet attempted is to use this to wrap the build such that building the binary happens in the FHS container (using the unwrapped versions of the compiler and associated tooling).

sjburt 2y ago

It seems like every article about nix goes on and on about DLL hell. I've been using Debian/Ubuntu for 15+ years and never really experienced dependency hell. I guess maybe this is thanks to hard work by Debian maintainers and rarely needing to run a bleeding edge library, but also, why do we need to run bleeding edge versions of everything and then invent an incredibly complicated scheme to keep multiple copies of each library, most of which are completely compatible with each other? And then when there's a security problem, who goes and checks that every version of every dependency of every application has actually been patched and updated? Why would I want to roll a system back to an (definitely insecure) state of a few months ago? What problem does Nix solve that SO numbers (properly used) doesn't? I have many of the same questions about Snap and even Docker.

HideousKojima 2y ago
I run into DLL hell any time I try to install some software that isn't in some sort of package repository or other happy path. Most recent example I can think of was about a year ago when I helped my father-in-law install Klipper on an RPi4 so that he could do input shaping on his 3d prints. All of the guides and documentation seemed to assume that you were using a specific version of Linux on the RPi, and if you weren't (like my FIL) then welcome to dependency hell. Took several hours of pulling my hair out to resolve them all, for a non-developer it would have been impossible.
- michaelmrose 2y ago
  So the installation story for klipper is hilariously bad. Its a shell script that apt installs and pip installs things as root. It is the very definition of non-portable. The distribution story for Python apps in general is fairly bad and the developer really takes it to the next level. This isn't DLL hell as a function of leaving the happy path its DLL hell as a function of burning the happy path whilst dancing naked on the ashes of the greenery that once was the happy path. See Calibre for willful disobedience vs instead of insanity. It eshews system libraries and uses a shell script to install but it dumps everything including Python libs in a directory.

bbor 2y ago

I love what they’re going for, but I couldn’t help but react negatively at finding out that I had been hyped up for a post on some small technical topic for an OS I don’t know of. Maybe title it “At the Mountains of NIXos Madness”? But then again I’m just a grouch! Well written article regardless, from what I was able to get out of it

pizzalife 2y ago

Calling binaries using ld-linux used to be a popular way to get around noexec on filesystems, since the libraries are usually in a place that is executable..

dhash 2y ago

I loved this post, and patchelf is a real gem of a utility.

georgewsinger 2y ago

=======Technical Summary======== Here's a problem with NixOS: 1. Suppose we have a `./nixos_binary_program_with_glibc-newer` compiled on a NixOS machine against bleeding edge `glibc-newer`. 2. `./nixos_binary_program_with_glibc-newer` will have `/nix/store/glibc-newer/linux-ld.so` path hardcoded into its ELF header which will be used when the program launches to find all of the program's shared libraries, and so forth. (And this is a fact that `ldd` will obfuscate!). 3. When `./nixos_binary_program_with_glibc-newer` is distributed to machines which use `glibc-older` instead of `glibc-newer`, the hardcoded `linux-ld.so` from (2) will fail to be found, leading to a launch error. 4. (3) will also happen on machines which don't use nix in the first place. =======Will's Solution======== 1. Use `patchelf` to hardcode a standard FHS `ld-linux.so` location into `nixos_binary_program_with_glibc-newer`'s ELF header (using e.g. `/lib64/ld-linux-x86-64.so.2` as the path) 2. Use a metaloader to launch `nixos_binary_program_with_glibc-newer` with an augmented `RPATH` which has a bunch of different `/nix/store/glibc-newer` paths, so that nix machines can find a suitable `ld-linux.so` to launch the program with. This will make `nixos_binary_program_with_glibc-newer` work on any machine, including both non-nix machines and nix machines (which might be running older versions of glibc by default)!

banish-m4 2y ago

Completely missed that Nix solves RPM dependency hell, which is a superset of the shared library hell. Another problem not solved by NixOS and most other distros is conflating and mixing dependencies in a messy, fragile way rather than having a clear separation between the OS and add-ons that FreeBSD and others have. Congruent with this is proper configuration and lifecycle management. I'm also wondering about the security of this RPATH approach, if it does or doesn't introduce vulnerabilities.

dreamcompiler 2y ago

I once had a robotic cat litter box that cleaned itself. Except once a week it would get clogged and I would have to spend a quality hour disassembling it, scrubbing off the feces embedded on the delicate parts, and reassembling it. Every two weeks when it heated up its artificial litter to dry, it would have missed a small piece of cat shit that when baked, filled my house with an aroma that I would not recommend you even try to imagine. And of course it needed special, expensive supplies that you had to buy from the manufacturer because the bottles had numbered chips. I eventually threw the damn thing out and now I just use a manual litter box. Takes 15 seconds a day to clean. It's a chore but it's a small predictable chore. When I read about NixOS I remember that robot litter box. It seems like it solves a real problem of difficulty X but it brings five brand new problems from a parallel universe you didn't know existed and they're all written in an indecipherable language and have difficulty 10X.

Zababa 2y ago
In my experience it kind of just works. Takes a bit more maintenance than arch, around an hour a month at most. I had to reinstall it once and it was way easier than my arch, since way more stuff is in text files I can commit. Also, since it's very very easy to rollback to a previous version, managing unpredictable issues is easy. I have a colleague that lost lots of time to an arch kernel panic during an update that required a reinstall. On NixOS I can reboot, choose the last derivation, work and fix that if/when I want.
NikkiA 2y ago
And then some Go fanatic will come along and use this experience to explain why it makes perfect sense to static link bits of the standard library into every single binary ever produced, and dynamic linking was the worst idea since mankind's ancestors crawled out of the ocean and past the whales crawling back in.
brookst 2y ago
You are an excellent writer. I have not use Nix, but I have had a similar bad experience with a similar but different automatic litter box. Your “small predictable chore” point is spot on: how much human grief is created by elaborate, expensive, unreliable solutions to minor annoyances?