11 ms·
Oasis: Small statically-linked Linux system
- waynecochran 4y agoDeath to shared libraries. The headaches they cause are just not worth the benefit. I have been dreaming of someone cresting a system devoid of them.
- speed_spread 4y agoYou're just embracing another set of problems. Shared libraries were introduced for a reason. The are complex to handle, but that's a long standing tooling problem rather than an inherent issue of the model. I think we're actually just getting the hang of it with stuff like Nix and containers.
- shrimp_emoji 4y ago>Shared libraries were introduced for a reason. Lack of storage and bandwidth in the 1970s
- goodpoint 4y agoFalse. The main reason was to avoid having to maintain and update multiple copies.
- arccy 4y agonix and containers actually eliminate all the benefits of shared libraries while still having all their problems
- naasking 4y agoNon-deterministic and unpredictable behaviour is a bigger problem than whatever benefits you think shared libraries provide. That's why so many people here prefer static linking, and that's what Nix gives you even for dynamic linking.
- rattlesnakedave 4y ago“Shared libraries were introduced for a reason.” I’m pretty sure the reason is “X11 was large and badly factored, and people who should have known better thought it would save disk space.” The problems they claim to solve are updates across applications at once, but in the enterprise how often does that actually happen? Application authors still have to test against the new version, and sysadmins still have to roll out the changes required to support the update. In practice they may as well statically link. Let’s not forget the problems they introduce are significant. Implementations are complex. Applications become less portable. ABI versioning is a pain (versioned symbols are gross). Semantic changes in the shared lib cause chaos. Exec is slower because of the linking done at load time. Dynamic libs are bigger than static ones, AND they’re loaded into memory. More difficult to verify the correctness of a given executable. The alleged “benefit” here really does not outweigh the cost. If we were doing static linking from the beginning, it’s difficult to say if we would have wound up with things like docker or nix in the first place.
- edflsafoiewq 4y agoSurely the main reason is that the linked library may actually be different on different machines. Stuff like Open GL.
- cesarb 4y ago> If we were doing static linking from the beginning, it’s difficult to say if we would have wound up [...] We were doing static linking from the beginning, at least on Unix. Dynamic linking came later.
- IshKebab 4y agoAnd on Mac.
- a9h74j 4y agoPlenty of support for static linking, I am seeing. Would static linking conceivably make sandboxing, jailing, etc easier? Compared to all the effort which goes into dynamic linking, I suspect it would be better spent on sandboxing (both mechanism and UX).
- alkonaut 4y agoYes. But the other set of problems exists anyway. Even if you drop in a new system version of a shared library, you still need to consider what other versions are used apart from the system library. Any app (other than those provided by a system package manager of course) can have a local version different from the system version and any app can have linked it statically?
- flohofwoe 4y ago> Shared libraries were introduced for a reason. Those reasons were floppy disks and RAM that was measured in kilobytes though.
- onlyrealcuzzo 4y agoWhat are the headaches with shared libraries? Assuming that you pin the version of the library you need, two applications using the same library /shouldn't/ cause issues. In most *nix systems - each instance of a shared library has its own data.
- rattlesnakedave 4y ago“ Assuming that you pin the version of the library you need, two applications using the same library /shouldn't/ cause issues.” Then why bother?
- waynecochran 4y agoI am assuming you know the answer to the question if you have been developing code for awhile. What makes the problem worse is that it is transitive. I link against Colmap libs, which links against Ceres libs, which links against OpenMP, … these are ever changing code bases and I have to create executables that run on local machines, Azure cloud machines, AWS cloud machines, … chafed down shared lib mismatches becomes a nightmare.
- ksec 4y agoI dont disagree. But wondering if I am missing anything other than the usual cons such as Memory usage, code ( Storage ) size and longer compile time.
- waynecochran 4y agoI understand the motivation for shared libs is the reduced memory footprint for large common libs. I have an alternative idea that avoids the headaches of shared lib mismatches and still can reduce the memory footprint. Have relocatable code be statically linked but stored with a hash. When loading a binary see if common relocatable code is already loaded .. if it has the same hash then it is exactly the same and can be safely shared. No version mismatch headaches.
- 8organicbits 4y agoI think often the binaries will have been compiled against slightly different versions. With dynamic linking, I can swap 1.0.1 for 1.0.2 or 1.0.3 without issue. The hash would require an exact match, which is likely rare.
- waynecochran 4y agoI want an exact match. That is what I developed and tested against. A change, however small, can have unintended consequences. Changing versions is a huge issue from a reliability point of view.
- eru 4y agoWouldn't you want to randomise some stuff to make attacks harder? (Sorry, I forgot the details.)
- cesarb 4y ago> Have relocatable code be statically linked but stored with a hash. When loading a binary see if common relocatable code is already loaded .. if it has the same hash then it is exactly the same and can be safely shared. It's not actually "code" which is being shared, but instead the memory pages containing that code (the sharing granularity is a single page, since sharing works by pointing the page table entries on each process to the same physical page). So either you'd have to force page alignment for each and every function (highly wasteful, since the page size is at least 4096 bytes), or you're reinventing shared libraries but using a hash instead of a version.
- hlsnqp 4y agoSo next time a library such as OpenSSL has a critical security vulnerability that needs to be patched, you'd rather every single binary that has it statically linked is updated, rather than a single shared library file?
- rattlesnakedave 4y agoGoogle “OpenSSL dependency conflict” and view the results. There is significantly more work going on when OpenSSL releases and update than “just update the single shared library file.” Applications often need to be updated as well, so you may as well just statically link the new binary.
- nicce 4y agoIt really depends on whether the bug is in API or not. If it is not, it is likely that upgrading only the shared library is enough.
- convolvatron 4y agoHow lovely to live in a world where api contracts are absolute and complicated internal behaviors never conflict in unforeseen ways. We’ve really gotten on top of the software complexity problem
- hedora 4y agoThat's exactly the problem deb files and apt solve (via integration testing and documenting antidependences.)
- arinlen 4y ago> How lovely to live in a world where api contracts are absolute and complicated internal behaviors never conflict in unforeseen ways. Some libraries managed to preserve API contracts throughout the years just fine. Even in C++, a language which does not establish a standard ABI, has frameworks which have not been breaking the ABI throughout multiple releases, such as Qt.
- encryptluks2 4y agoI used to feel the same way. In an ideal world all packages would be automated and developers would have a way to alert packaging tools that a new version is release, so that it can be run through CI/CD and users would get new packages almost instantaneously without any breaking changes. However, until that happens there is nothing wrong with shared libraries. They exist for good reason. They save bandwidth and ensure that you're not getting old libraries.
- waynecochran 4y agoThe save bandwidth? Do you mean memory? You want to link against the exact same code you developed and tested your program with. Ensuing you get and exact match is critical for robust code.
- encryptluks2 4y agoYes, dynamic libraries save a descent amount of bandwidth when installing packages. Also, for the most part no I don't want the exact libraries always. If library creators have a security fix then I want the one with the fix, and there are plenty of times I've seen developers create something and barely maintain it afterwards. Package maintainers are really good at keeping packages for necessary libraries.
- icedchai 4y agoYou could try DEC Ultrix. (Oh, did you mean a modern system?)
- benreesman 4y agoWhen it comes to mandatory dynamic linking, it's always a Beautiful Day for an auto-da-fé: death is too good for mandatory `glibc` and `libstdc++`. As for a system devoid of it, I have grand plans, but for now: people do it with Docker every day! That's what Docker is. It's not broadly technically feasible right now to get a low-friction static link on a mainstream distro with mainstream libraries, but life, erm, technology Finds a Way. For now that's Docker, but the modest </s> enthusiasm around it shows that people want a way out pretty fucking bad and there will be better options soon.
- arinlen 4y ago> As for a system devoid of it, I have grand plans, but for now: people do it with Docker every day! That's what Docker is. No, Docker really isn't what you think it is. Docker isolates processes, and provides a convenient platform independent way to deliver stand alone applications, complete with software defined networking and file system abstraction, at the expense of shipping the whole tech stack, including everything plus the kitchen sink like full blown web servers. Thus a .DEB package which otherwise would be 10MB ends up being a Docker image often weighing over 100MB. And pray tell what's Docker's solution to software updates? Well, it's periodic builds that force downstream dependencies and users to re-download payloads of well over 100MB a pop. And you advocate this as an alternative to shared libraries? Who in their ri
- benreesman 4y agoI advocate `musl`/etc. and LLVM as an alternative to `fuck.this.so.0.0.4 -> ...`. I switched from Docker to Nix some time ago (because of many of the pain points you mention). But in a world where native applications worked on any kernel >= the one it was built on, you'd rapidly find out that bin-packing the webserver and webcrawler into a finite list of SKUs (which is where all this container mania originated) via namespaces and cgroups are really not problems that most Docker users have, and there are much simpler ways to get two versions of `node.js` looking at two different sets of libraries. Because you can't reliably know what devilry just stomped all over your `/usr/lib/x86_64/...` or whatever, you kind of have to eat that 100MB image if you want software that worked on Tuesday to still work on Friday. It's a sad state of affairs, but This Too Will Pass. Getting the word out that the reason Docker saves your ass on reproducibility is emulation of static linking behind the FSF's back will accelerate that process.
- arinlen 4y ago> Death to shared libraries. The headaches they cause are just not worth the benefit. Completely disagree. Even though one size does not fit all, anyone who makes sweeping statements about static libraries is just stating to the world how they are completely oblivious regarding basic software maintenance problems such as tracking which software package is updated, specially those who are not kept up to date on a daily basis.
- waynecochran 4y agoI like the old school days of just building everything from source. Similarly the Linux Kernel was statically built -- before modules. I knew the whole system from top to bottom.
- nicoburns 4y agoIf there was a standard for specifying which static library a binary has linked, this would be a non-issue.
- arinlen 4y ago> If there was a standard for specifying which static library a binary has linked, this would be a non-issue. You just gave a non-sequitur answer that rings as rational and true as stating that if the Pope would fly he would be an Airbus A330. Limiting which static library you can link to means absolutely nothing to the problems created by static linking, and how shared libraries solve all of them without any meaningful tradeoff at all.
- CuriousCosmic 4y agoIn a sense there kinda is. Pre-linking is a way to statically specify the lookup for a given shared library in an ELF executable. It's still kinda dynamic linking because the file is loaded at runtime but there's no library lookup, it's just "load this file at this path into address space while we load the rest of the main executable". Before PIC there would have been a bit more of a distinction but nowadays static and dynamic libraries are really only distinguished by where the symbols are (unless you want to be able to dynamically load and unload after startup).
- SkyMarshal 4y ago> I have been dreaming of someone cresting a system devoid of them. Doesn’t that pretty much already exist now with immutable system builds, eg Fedora Silverblue, GUIX, NixOS? Each package pulls in its own dependencies with the specified version and config, and then no changes can be made once the system is built. Only way to update is to rebuild the system with the updated versions. Costs more disk space but that is cheap these days, and solves shared dependency hell.
- goodpoint 4y agoAbsolutely not. Dynamic libraries are crucial for security. Static linking again and again has proved to be a security nightmare. It makes security updates dependent from upstream release or full rebuilds making things difficult.
- spc476 4y agoBut aren't you dependent from upstream anyway? Or do you recompile from source the broken dynamic library? In my experience, most sysadmins these days don't compile from source.
- goodpoint 4y agoNo to both. The Linux distribution you use is responsible for backporting patches, fixing the vulnerability, building updated packages, testing them and sending the updates. You get targeted security updates without changes in any other functionality.
- naasking 4y agoProved to whom, when, and in what context? Systems that were designed for easy reproducible builds from the ground up are fairly new, so how sure are you the context hasn't changed?
- deleted 4y ago[deleted]
- SuperSandro2000 4y agoWhich is not feasible for misters like qt WebKit or chromium.
- benreesman 4y ago“I tend to think the drawbacks of dynamic linking outweigh the advantages for many (most?) applications.” – John Carmack <rant> I don't like to be a parrot for the Plan9 people [1] (they've been winning their own fights for a while now), but they've assembled a better set of citations than I could in the time I'm willing to devote to an HN comment. Dynamic linking is fine, in very specific, very limited cases. Python extensions. Maybe you want to do a carve-out for like, some small hardened core of an SSL implementation or a browser because you've got actual hot-deploy in your (massive) installation/infrastructure. As a basically mandatory default it's a chemical fire next to an elementary school. It started gaining serious traction as a horrific hack to get a not-remotely-finished X Windows to run on Sun boxes in like, +/- 1990. It wasn't long before the FSF had figured out that they could bio-weaponize it against interoperability. The `glibc` docs are laughing in their beer while they explain why static linking on "GNU" systems is impossible, and also makes you a criminal. They don't even pretend to make a cogent argument (they said the quiet part loud on the mailing lists). It's to kill other `libc` and `libc++` (fuck you small-but-vocal anti-interoperability FSF people!). Don't want LLVM to finish you off? Start interoperating better. There are a lot of us who think "Free Software" has to mean that I can use it in legal ways that you philosophically disagree with, and if it happens to be better than your Win32-style lock-in? Well that's just groovy. Dynamic linking by (mandatory) default: Just Say No. Myths: - People want it: Docker. QED. - Secure: Dynamic linking is the source of tons of security vulnerabilities. More generally, the best way to not trust your system is to not be able to audit the literal code you're running without jumping through insane, sometimes practically impossible hoops. If you want "father knows best" security, you need self-patch like Chromium and shit do now. And if you're willing to trade performance for security by the gross, flat pack or something. - Performant: Dynamic linking forces address patch-ups, indirect calls, and defeats many optimizations. It thrashes TLB. It slows startup of even simple programs. - Necessary: Text pages are shared. I don't care if you've got 9, 90, or 90k processes using `printf`: it's going to be in the same text page unless something else went badly, badly wrong. </rant> [1] http://harmful.cat-v.org/software/dynamic-linking/ http://harmful.cat-v.org/software/dynamic-linking/
- medoc 4y agoYour last point is a bit off. Text pages are shared iff they come from the same file. So your point works if the 90k processes are the same program. Else: shared libs...
- snvzz 4y ago>and is probably better compared to a BSD. That's a very unfortunate comment in the second paragraph.
- Sakos 4y agoShould probably clarify in what ways (or for what) it's supposedly better. That's quite the inflammatory statement.
- camgunz 4y agoI think the meaning here is "better <to be> compared to a BSD", as in it's closer to a BSD than a Linux in terms of design and ethos.
- Sakos 4y agoAh, you're right.
- projektfu 4y agoI don't read that sentence to mean it's better than BSD, rather that Oasis is more like BSD than a typical Linux distribution. "It is better compared to" is an idiom meaning "the comparison is more apt to".
- deleted 4y ago[deleted]
- rafram 4y agoYou’re misreading it. The author means that it’s more similar to a BSD than to a typical Linux system, as the previous sentence makes clear.
- unsafecast 4y ago((better compared) (to BSD)), not (better (compared to BSD)) :)
- 4y ago
- mmargerum 4y agoNow if I could just boot this from a cartridge I'd approach the last time programming was truly joyful for me.
- projektfu 4y agoI played around with the Qemu image and I really liked it. I wasn't even aware of some of the software they chose, like Velox.
- KSPAtlas 4y agoI used this before, pretty impressive how much it can pull off
- dijit 4y agoI ran a really non-scientific test before on startup speed between Oasis vs a slimmed down direct kernel booted Debian with systemd. Obviously I'm writing this because Oasis absolutely trounced it. So; Oasis is currently my go-to for small deployed VPS's. (here's the video if you actually care: https://www.youtube.com/watch?v=X2Aw5SSqYFo https://www.youtube.com/watch?v=X2Aw5SSqYFo you'll note that the systemd one was delayed by the network on start- some people are going to argue that this is an artificial delay, but this is the truth of a startup and oasis had already booted before the delay anyway)
- zamadatix 4y agoFWIW I get a faster boot time than that slimmed Debian VM with a physical Celeron class physical box booting a stock kernel, even including the real BIOS and hardware initialization time. I don't think systemd really has much to do with why that VM boots slow.
- dijit 4y agoI don’t think that’s true. That machine is a threadripper 3970x with 256G 3.2GHz DDR4, with a 4xNVMe RAID0; it’s a disturbingly fast glass-cannon. I mean, it’s not an accurate test for sure; but I seriously doubt your VMs can boot faster stock and with the bios, on a slower machine.
- zamadatix 4y agoBoot speeds on the timescales in the video end up having little to do with horsepower once you get to having "some sort of SSD to load the boot image from" and "some sort of CPU that can at least decompress that in a reasonable time". It's more the "Wait here for 1 second for this hardware to probe", "Wait here for 5 seconds to see if the user wants to go into setup", "Wait here for network to timeout" type configuration items. Having a ton of high end hardware just makes this worse, more cores to initialize, more memory to initialize, more devices to probe. I picked the low end fanless box as example because of this - it boots significantly faster than my high end box after all. It still loses out to how little a VM has to do while booting but not by much. It defaults to assuming a very short "wait on user" time for BIOS/UEFI, it has a mostly static and very simple hardware topology so doesn't need to dynamically load much during BIOS/UEFI, and systemd isn't told to do all those boot time loads/delays the Debian default config seems to have. For whatever you have "slimmed down" on it you're still trying to load nvidia GPUs, run cloud-init scripts each boot, and whatever networked service you mentioned thinks it needs to wait for network before letting the box finish booting is. Those kinds of things are what make the Debian VM boot slow, not systemd or necessarily the size of the kernel until the more noticeable things are fixed. Systemd is just what Debian uses to do those boot time things and if you did the same things with OpenRC or SysVinit or your own init system without changing what was being done it'd be a pretty slow boot too. As for VPS if "not systemd" and small/fast/static are high or your list I'd highly recommend Alpine Linux over Oasis. It's known as the gold standard for containers but it makes a nice lightweight VM box too. I don't have anything against systemd but I did run Alpine as my standard VM image for a number of years and was very happy with it on the server side plus you get to keep convenient things like a package manager if you so desire. Their "Virtual" pre-built image will have the kernel image and firmware files slimmed down to the point you might not even care about trying to roll your own changes into it.
- hulitu 4y ago> Minimal bootstrap dependencies. Any POSIX system with git, lua, curl, a sha256 utility, standard compression utilities, and an x86_64-linux-musl cross compiler can be used to bootstrap oasis. So you need a custom POSIX system to bootstrap oasis. > All software in the base system is linked statically, this could be a good thing, however > including the display server (velox) and web browser (netsurf). No X server and an obscure web browser. Useless.
- castlec 4y agoum, it says if you have the libs and configured compiler, you can build it. there's also nothing stopping you from doing static builds of whatever application you want to include in your OS.
- sophacles 4y agoName me an OS that doesn't need an OS to build it.
- hulitu 4y agoIt needs a very special OS: "git, lua, curl, a sha256 utility, standard compression utilities, and an x86_64-linux-musl cross compiler"
- solar-ice 4y agoThat's... a list of things I already have installed, the Lua executable aside. This is a pretty normal-looking dependency list for native C software. Are you perhaps not used to building C software?
- yjftsjthsd-h 4y agoOf those, only an x86_64-linux-musl cross compiler strikes me as even remotely unusual, and that's still widely available.
- sophacles 4y agoYou mean besides just install the relevant packages by: "pacman -S git lua52 gzip coreutils gcc musl" on a standard arch install? (presumably similar for other distros)
- its_bbq 4y agoIs this meant more to be used on a small embedded Linux system? As a Serverless OS for fast startup?
- mro_name 4y agoThe core values are very appealing to me. what is the typical update routine a la 'sudo apt-get update && sudo apt-get dist-upgrade'?
- bragr 4y agoFrom the readme: No package manager. Instead, you configure a set of specifications of what files from which packages to include on your system, and the build system writes the resulting filesystem tree into a git repository. This can then be merged into /, or pulled from another machine. If you're looking for a package manager, you're in the wrong place.
- mforney 4y agogit -C /src/oasis pull && samu commit && doas git -C / merge The first command pulls updates from the source repository, the second command builds a new filesystem tree and commits it, the third command merges that commit into your / repository.
- mro_name 4y agothanks for oasis!
- tyingq 4y agoI'm curious how it handles the client side dns resolver. That's one of the sticking points for "fully static linking". Edit: Looks like it uses musl's resolver. Which is better than I remember it when using it a long time ago. It does still lack some things like IDN/punycode support. And things like this: https://twitter.com/richfelker/status/994629795551031296?lang=en https://twitter.com/richfelker/status/994629795551031296?lan... Pluggable authentication modules (PAM) may also be an issue with this setup, probably at least requires a lot of manual research and twiddling.
- suprjami 4y agoThe musl resolver also lacks "no-aaaa" which glibc grew a week or two ago.
- gravypod 4y agoI'd love to see someone port this to Bazel and build a ruleset to generate os images for KVM/VirtualBox/flashing to bare metal. This would be great for having a generic platform for building embedded systems from source.
- benreesman 4y agoI'm working on a Nix/NixOS based polyglot Bazel build infrastructure (that we're using in production) and I've almost got `glibc` and `libstdc++` squeezed all the way out. They're like fucking Kudzu (which is not an accident), but I'll get 'em eventually. Once `glibc` and `libstdc++` are all the way gone it will be a hop, skip, and a jump to doing what you're describing. I anticipate it being ready for open source as a standalone flake by the end of 2022.
- dijit 4y agoI literally had that conversation in their IRC channel and the resounding consensus was that “why would you want that” I was asking if anybody had done it yet, but since there’s people who want that I will submit some BUILD field to oasis. :)
- jrexilius 4y agoThis line is a bit concerning to me: "and libtls-bearssl, an alternative implementation of libtls based on BearSSL" It sounds like a custom rolled variant of a low-adoption SSL lib that isn't itself very well tested.. It may be better.. it may be horribly flawed or backdoored in a non-obvious way. But I also get how horrible OpenSSL is..
- synergy20 4y agosame here, in fact the last bearssl release was 4 years ago, very inactive if still alive. mbedtls might be better here, but it does not have tls1.3 fully ready(I don't think bearssl has 1.3 either). wolfssl could be the best, small and robust with new features, it is dual licensed if you want to use it in commercial releases.
- _8j50 4y agoDebian has reproducable builds going for it. I would be very interested in static gentoo like source distro though.
- numlock86 4y ago> BearSSL is incredibly small and well written, but is not widely adopted. Well, probably because the author himself discourages you from using it in anything serious. Crypto libraries need a lot of testing and reviews by experts, which BearSSL doesn't have yet. Thomas Pornin is an incredibly smart guy when it comes to crypto and BearSSL is my crypto library of choice for small (mostly embedded) projects, too. Projects like this might help to get it the attention it needs.
- intelVISA 4y agoI use BearSSL in a few toy projects, would love to see it get more traction and eventually some proper cryptographic audit if it moves out of alpha.
- jancsika 4y agoWould dlopen be disallowed on this system? E.g., technically speaking most (all?) audio plug-ins are shared libs...
- denton-scratch 4y agoSounded good to me, until I found out there's no repository; you just pull everything from github. Then I stopped reading. What could go wrong?
- rishav_sharan 4y agoAs someone who clearly doesn't understands this well enough - can I run a desktop environment on top of it?
- pas 4y agoit has Velox which is a Wayland compositor, so you can start running static linked Wayland programs. and of course you can add things like Firefox, LibreOffice or whatever, but you either need to spend many many cursed nights and dark days to get them to compile statically ... or just add a dynamic linker (ld) and add the dependencies... but at that point you built a second independent system on the same filesystem.
- gkhartman 4y agoI was ready to give this a try, but the use of BearSSL is concerning. It goes on to say that it's not a widely adopted SSL implementation, so how can it be trusted? I suppose adoption has to start somewhere, but I'll watch from a safe distance.
- baby 4y agoI used to audit tls libraries for a living and bearssl is the library I would choose today if I had to make a choice
- ComplexSystems 4y agoCan you elaborate?
- logdahl 4y agoThe discussions on static- vs dyamic linking piqued my interest. Is there any resources I should read about the history and concrete pros & cons of linking techniques?
- ttgurney 4y agoI'm really grateful for Michael Forney's work on this project since I found it a good resource for getting a variety of popular packages to build statically. I ended up starting my own statically-linked Linux distro, but with some creative differences like shell scripts (vs Lua scripts) and package management. I recently "finished" it (close enough anyway) to the point that I run it as my daily driver. Not trying to plug my project (it's not public, not yet anyway) but if anyone wants to discuss this sort of thing (distro creation and static linking in particular) and compare notes via email, I'd be happy to.
- lr1970 4y agoInteresting project but according to the github history it has had only one official release in 2017 and there has been only 34 commits so far this year. Is the project in low maintenance mode?