4 ms·

An interesting aspect of this effort is that it undoubtesly makes modifying these tools much easier. While GNU and the FSF are all about legally ensuring the ab

An interesting aspect of this effort is that it undoubtesly makes modifying these tools much easier. While GNU and the FSF are all about legally ensuring the ability to read and modify their code, they have, in practice, made it almost impossible to do so for entirely technical reasons. I’m a pretty good C programmer and whenever I’ve looked at even simple GNU tools like `cat` or god forbid GNU libc’s `printf` code (this one is actually kind of delightful it’s so nuts), I have despaired of understanding, let alone modifying, the code. GNU coreutils are such a morass of preprocessor defines and bonkers C code to support every legacy system GNU has ever run on, that it might as well only be distributed in binary form. I think it would literally be easier to decompile most GNU tools and modify the decompiled platform-specific source than to make the same modification to the source as shipped. Rust has a steep learning curve, but learning Rust is a picnic compared to figuring out how to modify just one GNU tool, and of course they’re all different. Rust puts the onus of portability where it belongs: in the language and compiler, instead of forcing it on every single programmer and application.

melissalobos 5y ago
I have seen it argued many times that the GNU codebases are deliberately written to be unintelligible to avoid claims of being derived from UNIX sources. If they were written in the most straightforward ways, then they would have to show that no-one working on it has seen the original sources. I believe there is even a section in the glibc manual discussing this idea.
- StefanKarpinski 5y ago
  Awful but plausible.
- smoldesu 5y ago
  Sometimes it's obscenely complicated because the GNU devs like showing off with even the simplest of programs. My favorite example is GNU yes, a coreutil that's so ludicrously fast[0] that it can cause OOM states in mere seconds if you're not careful where you're redirecting output. Other HN users have compared the GNU version[1] with the BSD one[2], and I think it's a pretty good way to grok the difference in style between the two. In any case, both tools work perfectly fine for pretty much every use case. [0] https://www.reddit.com/r/unix/comments/6gxduc/how_is_gnu_yes_so_fast/ https://www.reddit.com/r/unix/comments/6gxduc/how_is_gnu_yes... [1] https://github.com/coreutils/coreutils/blob/master/src/yes.c https://github.com/coreutils/coreutils/blob/master/src/yes.c [2] https://github.com/openbsd/src/blob/master/usr.bin/yes/yes.c https://github.com/openbsd/src/blob/master/usr.bin/yes/yes.c
  - duckerude 5y ago
    One more data point here is that uutils yes can be even faster than GNU yes by taking advantage of Linux's vmsplice syscall: https://github.com/uutils/coreutils/blob/main/src/uu/yes/src/splice.rs https://github.com/uutils/coreutils/blob/main/src/uu/yes/src... On my system it manages 20GB/s vs 6 GB/s (piping through pv into /dev/null).
yjftsjthsd-h 5y ago
This sounds right on the edge of the traditional situation in rewrites where everyone says how awful the old code is, so they rewrite it, but then the new code doesn't handle edge cases (doesn't run on AIX/Illumos/NetBSD, yes(1) turns out to be really slow without that weirdness[0], file semantics have really weird edge cases that you only find after losing data), so the new code is adjusted to actually be as comprehensive as the original, at which point the new code is as ugly as the original. Of course, it depends on why the old code was ugly; if, as suggested below, some of it was being intentionally obtuse to avoid claims of copying, there might be room for actual improvement, and we do have more perspective and better tooling and new algorithms - real improvement is 100% possible, I would just be very cautious in assuming that based on an incomplete comparison. [0] https://news.ycombinator.com/item?id=14542938 https://news.ycombinator.com/item?id=14542938
- shakow 5y ago
  > doesn't run on AIX/Illumos/NetBSD These OSs distribute their own coreutils. > yes(1) turns out to be really slow without that weirdness[0] I'm not sure there is a workflow where yes writing slower than 10.2GB/s is the bottleneck. Being able to go so fast is a nifty engineering feat, but hardly a necessity IMHO.
  - Spivak 5y ago
    Yes but scripts are written assuming GNU coreutils behavior and this makes them easier to port. No reason they shouldn’t work everywhere. I think a nontrivial portion of Mac users on HN probably use the GNU utilities on macOS.
    - rodgerd 5y ago
      > No reason they shouldn’t work everywhere. No reason except the complexity of maintaining huge amounts of code to support dead processors and operating systems, or ones that are so rare that they might as well be dead.
    - KptMarchewa 5y ago
Anthony-G 5y ago
Thanks for sharing this perspective. A few years ago, I started learning C with the aim of contributing to GNU coreutils and other utilities that come with a GNU/Linux system. I read most of the K&R book and thought I had a decent grasp of the C language but when I cloned the coreutils source code, I couldn’t figure it out and gave up, thinking I’d need a lot more experience before I could understand production-quality code. It’s reassuring to find out it wasn’t just me that struggled with the complexity of the code base. Since then, I’ve discovered (thanks to Hacker News) Decoded: GNU coreutils¹, a “resource for novice programmers exploring the design of command-line utilities” but unfortunately, I no longer have the capacity or free time to spend on coding. ¹ https://www.maizure.org/projects/decoded-gnu-coreutils/index.html https://www.maizure.org/projects/decoded-gnu-coreutils/index...

melissalobos 5y ago

I have seen it argued many times that the GNU codebases are deliberately written to be unintelligible to avoid claims of being derived from UNIX sources. If they were written in the most straightforward ways, then they would have to show that no-one working on it has seen the original sources. I believe there is even a section in the glibc manual discussing this idea.

StefanKarpinski 5y ago
Awful but plausible.
smoldesu 5y ago
Sometimes it's obscenely complicated because the GNU devs like showing off with even the simplest of programs. My favorite example is GNU yes, a coreutil that's so ludicrously fast[0] that it can cause OOM states in mere seconds if you're not careful where you're redirecting output. Other HN users have compared the GNU version[1] with the BSD one[2], and I think it's a pretty good way to grok the difference in style between the two. In any case, both tools work perfectly fine for pretty much every use case. [0] https://www.reddit.com/r/unix/comments/6gxduc/how_is_gnu_yes_so_fast/ https://www.reddit.com/r/unix/comments/6gxduc/how_is_gnu_yes... [1] https://github.com/coreutils/coreutils/blob/master/src/yes.c https://github.com/coreutils/coreutils/blob/master/src/yes.c [2] https://github.com/openbsd/src/blob/master/usr.bin/yes/yes.c https://github.com/openbsd/src/blob/master/usr.bin/yes/yes.c
- duckerude 5y ago
  One more data point here is that uutils yes can be even faster than GNU yes by taking advantage of Linux's vmsplice syscall: https://github.com/uutils/coreutils/blob/main/src/uu/yes/src/splice.rs https://github.com/uutils/coreutils/blob/main/src/uu/yes/src... On my system it manages 20GB/s vs 6 GB/s (piping through pv into /dev/null).

yjftsjthsd-h 5y ago

This sounds right on the edge of the traditional situation in rewrites where everyone says how awful the old code is, so they rewrite it, but then the new code doesn't handle edge cases (doesn't run on AIX/Illumos/NetBSD, yes(1) turns out to be really slow without that weirdness[0], file semantics have really weird edge cases that you only find after losing data), so the new code is adjusted to actually be as comprehensive as the original, at which point the new code is as ugly as the original. Of course, it depends on why the old code was ugly; if, as suggested below, some of it was being intentionally obtuse to avoid claims of copying, there might be room for actual improvement, and we do have more perspective and better tooling and new algorithms - real improvement is 100% possible, I would just be very cautious in assuming that based on an incomplete comparison. [0] https://news.ycombinator.com/item?id=14542938 https://news.ycombinator.com/item?id=14542938

shakow 5y ago
> doesn't run on AIX/Illumos/NetBSD These OSs distribute their own coreutils. > yes(1) turns out to be really slow without that weirdness[0] I'm not sure there is a workflow where yes writing slower than 10.2GB/s is the bottleneck. Being able to go so fast is a nifty engineering feat, but hardly a necessity IMHO.
- Spivak 5y ago
  Yes but scripts are written assuming GNU coreutils behavior and this makes them easier to port. No reason they shouldn’t work everywhere. I think a nontrivial portion of Mac users on HN probably use the GNU utilities on macOS.
  - rodgerd 5y ago
    > No reason they shouldn’t work everywhere. No reason except the complexity of maintaining huge amounts of code to support dead processors and operating systems, or ones that are so rare that they might as well be dead.
  - KptMarchewa 5y ago

Anthony-G 5y ago

Thanks for sharing this perspective. A few years ago, I started learning C with the aim of contributing to GNU coreutils and other utilities that come with a GNU/Linux system. I read most of the K&R book and thought I had a decent grasp of the C language but when I cloned the coreutils source code, I couldn’t figure it out and gave up, thinking I’d need a lot more experience before I could understand production-quality code. It’s reassuring to find out it wasn’t just me that struggled with the complexity of the code base. Since then, I’ve discovered (thanks to Hacker News) Decoded: GNU coreutils¹, a “resource for novice programmers exploring the design of command-line utilities” but unfortunately, I no longer have the capacity or free time to spend on coding. ¹ https://www.maizure.org/projects/decoded-gnu-coreutils/index.html https://www.maizure.org/projects/decoded-gnu-coreutils/index...