9 ms·
Swapping GNU coreutils for uutils coreutils on Gentoo Linux
- ComputerGuru 2y agoNo, please don't! At least, not if you aren't equipped to handle the fallout yourself and not until these tools are further refined! I'm a fish-shell developer and I just dealt with a user that was getting bizarre test failures. Turns out the developers didn't account for basic things like the normalcy of `cat` having its output fd closed mid-stream (e.g. you are piping cat to something and that something exits before cat does) and their vesrion bails with an exception dumped to stderr instead of silently closing with a non-zero exit code.
- pixelbeat 2y agoSounds like a case of: https://www.pixelbeat.org/programming/sigpipe_handling.html https://www.pixelbeat.org/programming/sigpipe_handling.html uutils uses the coreutils test suite, so it makes sense to add a test case for that, which uutils will eventually get to. I'll do that now.
- photonbeam 2y agoIt’s somewhat inevitable that something will try to replace the C implementation of coreutils. Hopefully uutils will fix whichever bugs are reported upstream to lower the compatibility load though
- initplus 2y agoAt least it's somewhat possible to reliably test behavior of these against GNU coreutils. Pull down some archaic C++ project with a pile of janky shell scripts involved in the build (Chromium might be a good target) and compare final results.
- yjftsjthsd-h 2y agoOne could argue that building Gentoo is an excellent way to exercise things:) Which, granted, is kinda cheating since you could just `emerge chromium` or whatever... but IMHO even just making portage itself happy is a good first step.
- lionkor 2y agoWhy? Are you unhappy with the IMO very good test coverage, or has it not been working well for you the past checks notes decades?
- tadfisher 2y agoAuthor didn't state a preference either way, but why do you care? Nothing is wrong with a reimplementation, as using it will nail down any untested/undocumented behavior of the original code.
- nubinetwork 2y ago> This Gentoo setup has /bin, /sbin, /usr/bin, and /usr/sbin all merged. I hope this isn't going to be the norm in the future, but Gentoo has been making my life difficult for years as it is... it's as if the developers don't test or use what they are forcing on everyone.
- rascul 2y agoWhat issues have you experienced due to the merging of those directories?
- nubinetwork 2y agoThat would imply that I've drank the Fedora/FDO/systemd Kool-Aid and trashed a perfectly working FHS. I have no interest in wasting the time trying to conform to their nonsense.
- yjftsjthsd-h 2y agoOkay; what problems would you expect to hit if you used a system where they were merged? (My best thought so far is having multiple hosts sharing a single /usr over NFS while having per-host root filesystems, but I've never actually seen that done. I've also thought about building a distro that kept its initramfs as root and just mounted everything else into it, but that's even further off the beaten path.)
- nubinetwork 2y agoI can think of a few off the top of my head... - broken shell scripts that are hardcoded to "/bin/bash" rather than "/usr/bin/env bash" (this might work for a while, but what happens when they remove the symlink?) - broken compiles because "/lib" and "/lib64" no longer exist, because of the lack of testing prior to making the change on behalf of users - broken boots because initramfs (dracut/etc) isn't structured correctly after the change (see lack of testing). I'm sure there's more but I'm not their personal testing infrastructure. I just want an OS that works without having to fight someone every 3 weeks because "everyone else is doing it, so we should too".
- teo_zero 2y agoI think the author swapped problems #1 and #2. The first one (name clash between different packages) is triggered only if the binaries are called the same, that is after you fix the second one (binaries having uu- prefix). Problem #5 is not well explained: if /usr/bin and /usr/sbin are conflated, how could cowsay not find its templates? Paths relative to the two directories are the same. For example, if cowsay is looking for its templates in ../share/cows, such relative path points to the same destination no matter if the binary is in bin or sbin.
- JoshMcguigan 2y agoThanks for reading so closely and providing great feedback! You are totally right on problems 1 and 2 being swapped. For problem #5, you caught me taking too large a logical leap and making some assumptions there. Turns out the issue is just that cowsay special cases directories called `bin` (and thus treats directories called `sbin` differently)[0]. I just pushed an update to the post correcting both of these. [0]: https://github.com/cowsay-org/cowsay/blob/d8c459357cc204723504e29be84607ceef2c5d42/cowsay#L85 https://github.com/cowsay-org/cowsay/blob/d8c459357cc2047235...
- LightFog 2y agoIt’s a shame to see all this effort going toward replacing core GPL licensed utilities with permissive ones. It seems like a particularly common thing in the Rust community. It feels disrespectful of the intentions of the work that went into the tools that are being cloned.
- pjmlp 2y agoGPL only took off, because Berkeley was rather busy with AT&T, and GNU was there for Linus to reach for. It isn't only Rust, it is any language that favours static linking by default.
- DeathArrow 2y agoI still wonder how an alternative world, where BSD won instead of Linux, would look like.
- pjmlp 2y agoMost likely it would be business as usual for all UNIXes, taking what they would feel like from BSD, as they were doing before during the whole AT&T vs BSD base model for UNIX architecture, like how Solaris evolved, or Windows used for its initial TCP/IP infrastructure (until Vista).
- happymellon 2y agoWe already have a fairly good idea based upon corporate actions today. Look at Mac OS. That's what happens when freedoms don't have to be honoured. Corporations have spent a lot of effort trying to work around the GPL, whether it was via network services or something else. If everyone had gone down the BSD route we would have been there, just a lot quicker. This is why I would never licence any of my work as anything other than GPL or AGPL (dual so that people can pay to avoid GPL, but they still contribute financially). This is all a team effort to make the world a better place and BSD is too idealistic.
- colordrops 2y ago
- vkaku 2y agoI welcome this. Would love to see more people attempt building with uutils.
- yjftsjthsd-h 2y agoTwo thoughts on the whole "alternate" thing: 1. When I first learned about the alternatives system, I initially assumed that it was in use for every single binary - that there was an alternatives selection to decide what provided /bin/ls, and one to choose /bin/sh, and one to determinue /bin/chmod, etc. (I mean, /bin/sh sometimes is depending on your distro and how they feel about bash/dash/ash but you get the idea.) And honestly I still kinda feel like that's a good idea, though it leans toward redoing how packages work in a way that reminds me of nix and Gobo; /bin becomes just a symlink farm pointing into per-package bin directories. 2. Although this kind of bulk-replacement is a good initial test, I feel like letting packages directly depend on GNU coreutils or not is maybe a good way to go - you can test packages one by one and switch them to point to a virtual package as they're validated, thereby letting the package manager properly manage dependencies by giving it enough information to asses the situation.
- parasti 2y agoWhat motivates the uutils project? I get rewriting things in language X. I don't really get naming your thing the same name, making a pretty website for it, and apparently trying to replace the original thing.
- daghamm 2y agoAren't the Rust binaries significantly larger? Wouldn't that be a problem for embedded system and containers? I'm also sceptical to the whole idea. There are tons of more interesting problems to solve than replacing an stable solution with a new one just because you don't like GPL.
- yjftsjthsd-h 2y ago> There are tons of more interesting problems to solve than replacing an stable solution with a new one just because you don't like GPL. I think the argument is usually memory safety rather than licensing.
- daghamm 2y agoIs that an strong argument in this particular case? This is after all cp and ls we are talking about. For me personally compatibility would be a much bigger issue
- IshKebab 2y agoIt uses a single binary like Busybox to solve that: https://uutils.github.io/coreutils/docs/multicall.html https://uutils.github.io/coreutils/docs/multicall.html I agree it's not what I would spend my time on, but I guess some people found this problem more interesting than us. I don't see a problem with that. Security vulnerabilities in GNU coreutils are rare but they happen. Also this would make building and editing the tools more accessible.
- DeathArrow 2y agoTo me "let's rewrite X in Y" seems like a wasted effort, but what do I know.
- beej71 2y agoThey rewrote Unix in C from assembly. Sometimes there's a gain to be had with new language tech, e.g. memory safety.
- jdoss 2y agoGentoo Linux will always have a special place in my heart. I learned so much about Linux doing Stage 1 installs back in 2002. I also learned patience with the long compile times and I heated my apartment during the winter. Pentium 4s kicked off a lot of heat. :) Leaving the should uutils be used over coreutils debate aside, this was a fun read for me and the urge to install Gentoo one one of my many old Thinkpads is flaring up hard.
- M95D 2y ago> Instead of modifying all the packages that depend on GNU coreutils (known as the package sys-apps/coreutils in Gentoo), I modified the GNU coreutils package to instead install uutils coreutils. That's not the way to do it (in Gentoo). He should have added coreutils to /etc/portage/profile/package.provided. Portage would then assume the package is installed even if it's not. This is used to install self-built binaries or packages from other distributions instead of packages provided by portage. https://wiki.gentoo.org/wiki//etc/portage/profile/package.provided https://wiki.gentoo.org/wiki//etc/portage/profile/package.pr...
- nubinetwork 2y agoI also cringed a little when I read that they used the same package name for their replacement, especially considering they were aware of virtuals and the alternatives frameworks... although I've never tried to do what they did, so maybe they had issues converting the entire portage tree to be compliant. I'll admit that it's a lot of work, but if you wanted to do it right, you'd have to put in that additional work.
- yjftsjthsd-h 2y agoI don't think this was meant to be "doing it right"; this was explicitly the quick and dirty way to test things.
- account42 2y agoUsing package.provided for this is as much of a hack as modifying the coreutils package.
- JoshMcguigan 2y agoThanks for pointing out `package.provided`. It does look like it could be a reasonable way to do this, but I'm not sure how exactly I'd be able to atomically swap GNU coreutils for uutils coreutils using that method? I think I'd need to add `sys-apps/coreutils` to `package.provided`, then install uutils coreutils while telling Portage to ignore file collisions (because I'd be overwriting GNU coreutils binaries). However, that would have hidden the fact that I would have also been overwriting binaries from other packages (for example `hostname`, which is provided by `sys-apps/net-tools` in Gentoo).
- hulitu 2y ago> Similar to busybox, coreutils uutils was being installed as a single binary, and each entrypoint (i.e. ls) was just a symlink to that binary. What can go wrong ? /s One tool to rule them all.
- yjftsjthsd-h 2y ago> What can go wrong ? /s Okay but seriously - what would you expect to go wrong? Tools explicitly looking for and breaking on symlinks is a bug on their side; that's a perfectly reasonable approach to take.
- vzaliva 2y agoUnlike some other commenters, I believe having alternatives to coreutils is beneficial. Since I started using Unix, I've worked with HPUX, IRIX, SunOS, Solaris, *BSD, Linux, Xenix, and probably a few others I've forgotten. This diversity in implementations necessitated standardization. Nowadays, 90% of the systems I use are various Linux distributions, and I often encounter packages that implicitly assume on specific versions like GNU make or GNU tar. Having alternatives encourages better compatibility and flexibility across different systems.
- IntelMiner 2y agoIn the 90's that balkanization was problematic because everyone was reinventing UNIX but with "our one useful feature" (graphics rendering, networked machines, etc) Eventually it boiled down to today where we have the BSD's and Linux. Other than helping force compatibility with the BSD's, who else stands to benefit from breaking away from "GNU-isms"? MacOS?
- account42 2y agoBSDs still exist though and they do have their own make and tar. Of course they tend to also have GNU versions for ports because that's easier than fixing all software. So existence of alternatives is not enough, people also need to care about them.
- wodenokoto 2y agoIf you read the article, it’s a guy playing around on his system to see if hacking this thing with that thing works. If you only read the comments, you think Gentoo was switching out core utils and doing it in the worst technical way possible.
- egberts1 2y agoThis is good stuff, especially when making an embedded system. Good work, Joshua Mcguigan!