10 ms·

An update on rust/coreutils

187 points by marcodiego 5y ago

kkoncevicius 5y ago
According to the graph more than half of the tests are still failing. Can the speed be (to some extent) attributed to some missing functionality ?
- sylvestre 5y ago
  Good question! Probably not. In general, functionalities are enabling or disabling a behavior when doing an operation. In the code, it translates most of the time by a simple if else. For example, adding new options usually looks like this PR: https://github.com/uutils/coreutils/pull/2880/files https://github.com/uutils/coreutils/pull/2880/files The performance wins are usually produced by using some fancy Rust features.
  - alerighi 5y ago
    Well, they could be doing some simplifications and assumptions that can improve performance. Also performance is not always the only factor to consider: you can optimize a program for speed, but also for RAM usage, or the size of the binary itself. In program like Coreutils to me is more important that the programs are small than the rest. Typically you use a lot of commands in a script, to do some trivial operations (the input of the program is usually small), thus simpler programs (that have less startup time) are usually better.
    - nix23 5y ago
      >Coreutils to me is more important that the programs are small than the rest Then use BusyBox/Matchbox....because..well that's the whole point of those projects. https://busybox.net/about.html https://busybox.net/about.html
    - zaarn 5y ago
      After the first time the binary is executed, it should be comfortably in the memory disk cache. And frankly, on a modern SSD, the sector size is large enough that the only difference is reading 3 instead of 1 sectors to call "ls". Barely matters if it gets batched.
  - laumars 5y ago
    That’s only correct if you focus exclusively on optional features enabled by feature flags or environmental variables. Some features might be around support for things like Unicode where a tool could work 90% of the time but that extra 10% of support requires implementing a lot of additional logic that slows the routine down for the other 90% of use cases too. Also ‘if’ causes branching which costs a small amount of CPU overhead. So it’s not a free operation and can quickly add up if you needed inside a hot path.
    - devit 5y ago
      For new flags, this is easily solved by generating and compiling (via generic monomorphization, macros or build-time code generation) two versions of all the relevant code (including of course any loops calling the relevant code) and switching the one that is executed depending on the value of the flag.
      - tomsmeding 5y ago
        This is true, but also results in larger binaries. Elsewhere in this thread, it was noted that the coreutils compiled from the Rust code are already quite a bit bigger than the GNU counterparts. On many systems this might not matter much (and I guess it's fine if a software suite explicitly makes binary size a non-goal), but on some systems it does.
    - nicoburns 5y ago
      > Some features might be around support for things like Unicode where a tool could work 90% of the time but that extra 10% of support requires implementing a lot of additional logic that slows the routine down for the other 90% of use cases too. This kind of thing is where Rust really shines. The ecosystem was built post-unicode, so things tend to support it by default. Ripgrep for example has been unicode aware from the beginning, and you have to opt-out if you don't want that.
      - laumars 5y ago
        I’m aware of Rusts support for Unicode, I was only using that as an example because it’s easy to visualise since most people who’ve written any kind of text parsing will understand the additional computational overhead correctly supporting Unicode costs. But while the example doesn’t directly apply to Rust, I guarantee you that there will be other edge cases in a similar vein that might cause issues.
- tialaramex 5y ago
  Slightly less than half, but yes. That could certainly be relevant. On the other hand inevitably some tests will be fragile or even outright wrong and so while your solution passes a different (possibly better/ faster) solution fails because the test is bad. For example suppose you're implementing case-insensitive sort, you write a test and tweak it slightly so that it passes as you expected. I come along and write a slightly faster case-insensitive sort, and mine fails. Upon examining the test I discover it thinks I ought to sort (rat, doG, cat, DOG, dog, DOg) into (cat, doG, dog, DOG, DOg, rat) but I get (cat, doG, DOG, dog, DOg, rat) my answer seems, if anything better and certainly not wrong but it fails your test.
  - AndrewDucker 5y ago
    214 passes, 298 fails. That seems rather more than 50% failing, even before the errors and skipped tests.
    - tialaramex 5y ago
      There are 611 tests, 298 is less than half.
      - mlindner 5y ago
        There are tests that are erroring as well. 214 passing out of 611 is less than 50% passing (in fact it's almost only 1/3 passing).
  - tharkun__ 5y ago
    The test is right because it tests for the fact that lower case letters have a smaller representation as an ASCII character position. There is probably a ton of scripting and other software out there that assumes that this is how things are sorted. You also want a stable sort meaning sorting the same sequence twice you want it to be sorted the exact same way again. So you need this sort of definition of how things are supposed to be sorted. So while I agree that without historical context and compatibility for human consumption your way of sorting is probably fine, you could wreak major havoc when trying your sort as a drop in replacement. If you only have a few users change management for something like this is relatively easy. The user base of coreutils? Think twice if you want to try that change management.
- cedilla 5y ago
  It is possible, but with benchmarks like "head -n 1000000 wikidata.xml" I doubt it. A comment in that PR says "the difference to GNU head is mostly in user time, not in system time. I suspect this is due to GNU head not using SIMD to detect newlines". Unfortunately I couldn't find a list of failed/successful tests, if that's available I'd be happy if someone linked it
  - sylvestre 5y ago
    https://github.com/uutils/coreutils/actions https://github.com/uutils/coreutils/actions click on a CI job on main For example: https://github.com/uutils/coreutils/runs/4990891225?check_suite_focus=true https://github.com/uutils/coreutils/runs/4990891225?check_su... The "Run GNU tests" is probably what you are looking for.
The_rationalist 5y ago
How about making all ELF binaries symbol processing/retrieving faster? The gnu hashmap is very much obscolete and should be replaced by Swisstable. Key observations like this will keep being ignored for the decades to come.
cosmiccatnap 5y ago
MailNerd 5y ago
Great work! Looking forward to a Linux system where the majority of user land is written in a safe and performant language!
- pjmlp 5y ago
  It is not the first effort to do so, hopefully this one has more wind to come through.
Aldo_MX 5y ago
Instead of 30 to 60 patches per month, we jumped to 400 to 472 patches every month. Similarly, we saw an increase in the number of contributors (20 to 50 per month from 3 to 8). More contributors = more hands to fix issues and tune performance. I believe this is an absolute win.
hobofan 5y ago
Am I missing something, or how does a PR that bumps the version of an existing brew formula translate to "Brew is proposing coreutils"?
agumonkey 5y ago
Anybody has stats on which coreutils programs are the most used ? I wonder the impact of gnu userland on system perf.
- sylvestre 5y ago
  I don't have data but I guess ls, cp, mv, ln, chown, chmod, sort and cut are the most popular. Overall, I don't think it is a key part of the OS regular work. Most of the servers and system are busy with services, db, browsers, etc...
jacquesm 5y ago
An old mantra that has served me well: first make it work, then make it fast. So first get it to 100% compatibility, then, and only then concentrate on performance. Because if you don't do it that way you will end up foregoing compatibility because you'd have to say goodbye to your beautifully tuned code that unfortunately can't be 100% compatible. As long as it does not pass all the tests: it does not work. Even if it works for some selected cases, the devil is in the details, and those details can eat up performance like there is no tomorrow.
- sva_ 5y ago
  The coreutils consist of many individual binaries ('utils') which are more or less stand-alone[0], so I think this mantra doesn't apply here. [0] https://github.com/uutils/coreutils/tree/main/src/uu https://github.com/uutils/coreutils/tree/main/src/uu
  - pjmlp 5y ago
    It does, as many times as there are binaries.
  - btdmaster 5y ago
    It might not apply for the whole project, but it certainly does apply for each individual per-command functionality (cp, mv, or even the general cross-command features like the filesystem interface).
  - jacquesm 5y ago
    It absolutely does. Each utility is a program all by itself and the compatibility tests apply to each and every one of them. How fast they are is not relevant until they are 100% drop in replacements.
- draw_down 5y ago
- notdonspaulding 5y ago
  I follow this advice with one small modification. Namely, mine goes: - Make it once. - Make it right. - Make it fast. "Make it right" as the first step can trick an unseasoned developer into never finishing a prototype. I don't mind the first iteration of something being sloppy and then pursuing correctness with an existing solution to the problem in hand. I'm curious if you've got a sense of the interplay between prototyping and correctness similar to your sense of the interplay between performance tuning and correctness? Any thoughts?
hiccuphippo 5y ago
If one were to replace coreutils in a GNU+Linux system with this, which is MIT licensed, would it be fair to call it MIT+Linux?
- wiz21c 5y ago
  Yeah, I see the pun. But leaving the GPL license is a very bad idea in my opinion. If we have to rest son shoulder of giant, I'd prefer them being made of concrete, not dust. Coreutils are really a common and should be protected against appropriation. Depending on how much a rewrite it is, GNU's intellectual property may be enforced at some point. But that's a tricky question for a lawyer. If it were me, I'd have asked GNU first.
  - mustache_kimono 5y ago
    Um, GNU coreutils is a reimplementation of various AT&T UNIX utilities. It's irresponsible to say GNU has IP rights to this implementation, especially where GNU has done a similar reimplementation(!), and where EFF, etc., has made contrary arguments, as amici, in cases like Oracle v. Google.
    - wiz21c 5y ago
      Agreed, I was completely off on this one :-/
  - nickysielicki 5y ago
    Nobody is going to steal a coreutils implementation, make a bunch of internal contributions to it, and yield a competitive advantage through that. The whole point of coreutils is that it’s old and stable and coreish. In the real world, corporate engineering middle managers do not understand the GPL and do not understand how it does and doesn’t restrict them. They avoid it rather than taking the time to learn it.
- codeflo 5y ago
  I also think “GNU+Linux” is a terrible name and a worse marketing move, but you might misunderstand what it’s about. There’s no “GNU license”, and nobody ever proposed calling the system “GPL+Linux”. GNU sees itself as a project to replace the entirety Unix (hence the name, GNU’s Not Unix). A kernel is one part of the full OS, so if you combine the Linux kernel with the rest of GNU’s OS, according to this logic you get GNU+Linux. (Or maybe it should be GNU-Hurd+Linux, or GNU=~s/Hurd/Linux/... it's terrible.)
est31 5y ago
This is an extremely amazing project because it's actually trying to come up with a replacement that can be used as such. Nobody is going to rewrite scripts to use exa instead of ls for example. The coreutils are also somewhat the "basis" of the shell userspace. Outside of the test failures and missing features, the only reason one might not want the coreutils is the size. Right now, cat on my system is 44K large, while the default output size for the cat executable in release mode is 4.4 megabytes in release mode. If you enable a bunch of options in Cargo.toml to reduce the default bloaty settings (lto = true, codegen-units = 1, strip = true, debug = false), you get that down to 876K. Which is still 20 times larger than the native cat. For true, it's a similar story. 40K vs 812K.
- sylvestre 5y ago
  I added some CI task and replayed the history to graph the evolution of the size: https://github.com/uutils/coreutils-tracking#binary-size-evolution https://github.com/uutils/coreutils-tracking#binary-size-evo... This will help making sure that we aren't regressing (more) ;)
- bsdetector 5y ago
  It'd also be a great time to establish a better standard for using these programs with a shell or scripting. For example, if ls had a --shell option that would write out file information as quoted eval'able variables, or even JSON, or anything that was easily and reliably parsable it would remove a huge portion of scripting headaches and errors.
- andai 5y ago
  Is that a typical ratio for software ported to Rust? Where is all the extra size coming from?
  - rpeden 5y ago
    As someone else mentioned, shared libraries. If you built ls and statically linked libc and whatever else it needs, ls would be a lot bigger.
  - 5y ago
gravypod 5y ago
If you needed to do low level systems programming in Rust would you be able to use this as an OS abstraction layer by importing these sorts of things: https://github.com/uutils/coreutils/blob/main/src/uu/chmod/src/chmod.rs#L196 https://github.com/uutils/coreutils/blob/main/src/uu/chmod/s... Is this one of the intentions of this team? It sounds like it could make "scripting" in rust very nice if all of the CLI functions you're used to exist as language libraries.
- terts 5y ago
  That is a very interesting idea! It is not something we've discussed before as far as I know. For some utils (like chmod) it might be possible, but any util that is focused on outputting information currently does so by printing directly to stdout. So, ls doesn't give you a Vec or Iterator of listed files for instance, but instead prints them to stdout. Nevertheless, it would be a cool experiment to see if we can create something like a "ulib" crate that would provide the functions some functions for which it is possible.
  - Arcterus 5y ago
    I believe I actually had a proof of concept of this functionality in a PR at some point. This would basically have to work like mesabox, where each utility takes some generic input and output and then runs based on them rather than using stdout and friends directly (e.g. to avoid blowing through all your memory by stuffing many gigabytes of output into a Vec on certain commands).
- loeg 5y ago
  That would be much more elegant than something like libxo.
RcouF1uZ4gsC 5y ago
I can see this project really taking off. It has two advantages on the GNU version apart from just memory safety with Rust. 1) Ease of portably building. Just type ‘cargo build —release’ and it will just work, even on Windows. 2) MIT License. A company can take these and distribute them as part of a commercial offering without having to worry about GPL compliance.
- trelane 5y ago
  I'm not sure how point 2 is and advantage for anyone but the company tbh. It's also going to be _really_ hard to be more portable than GNU coreutils, when it comes to platforms it's available on.
  - nickysielicki 5y ago
    The whole value of coreutils is that it’s old and stable. I just can’t imagine someone making major contributions to rust coreutils and keeping them internal as some sort of competitive advantage. I do embedded linux for work and while OpenEmbedded makes it pretty easy to manage license obligations, it’s always a pain in the ass to deal with GPL code in a larger team where people will find any excuse to bikeshed about their misunderstandings of the GPL. For questionable historical reasons, my current job manually whitelists every GPL package we ship, so it being MIT makes things less abrasive to teams like mine. That’s an advantage for everyone, because it makes me more likely to use it and it makes me more likely to submit bug reports and fixes.
StefanKarpinski 5y ago
An interesting aspect of this effort is that it undoubtesly makes modifying these tools much easier. While GNU and the FSF are all about legally ensuring the ability to read and modify their code, they have, in practice, made it almost impossible to do so for entirely technical reasons. I’m a pretty good C programmer and whenever I’ve looked at even simple GNU tools like `cat` or god forbid GNU libc’s `printf` code (this one is actually kind of delightful it’s so nuts), I have despaired of understanding, let alone modifying, the code. GNU coreutils are such a morass of preprocessor defines and bonkers C code to support every legacy system GNU has ever run on, that it might as well only be distributed in binary form. I think it would literally be easier to decompile most GNU tools and modify the decompiled platform-specific source than to make the same modification to the source as shipped. Rust has a steep learning curve, but learning Rust is a picnic compared to figuring out how to modify just one GNU tool, and of course they’re all different. Rust puts the onus of portability where it belongs: in the language and compiler, instead of forcing it on every single programmer and application.
- melissalobos 5y ago
  I have seen it argued many times that the GNU codebases are deliberately written to be unintelligible to avoid claims of being derived from UNIX sources. If they were written in the most straightforward ways, then they would have to show that no-one working on it has seen the original sources. I believe there is even a section in the glibc manual discussing this idea.
  - StefanKarpinski 5y ago
    Awful but plausible.
  - smoldesu 5y ago
    Sometimes it's obscenely complicated because the GNU devs like showing off with even the simplest of programs. My favorite example is GNU yes, a coreutil that's so ludicrously fast[0] that it can cause OOM states in mere seconds if you're not careful where you're redirecting output. Other HN users have compared the GNU version[1] with the BSD one[2], and I think it's a pretty good way to grok the difference in style between the two. In any case, both tools work perfectly fine for pretty much every use case. [0] https://www.reddit.com/r/unix/comments/6gxduc/how_is_gnu_yes_so_fast/ https://www.reddit.com/r/unix/comments/6gxduc/how_is_gnu_yes... [1] https://github.com/coreutils/coreutils/blob/master/src/yes.c https://github.com/coreutils/coreutils/blob/master/src/yes.c [2] https://github.com/openbsd/src/blob/master/usr.bin/yes/yes.c https://github.com/openbsd/src/blob/master/usr.bin/yes/yes.c
j16sdiz 5y ago
Iirc, most of the gnu coreutil came from its locale handling. It supports lots of different legacy encoding.
tpoacher 5y ago
I do wonder, to what extent is this "increase in speed" the result of: - Refactoring out buggy, convoluted, highly-backwards-compliant code for cleaner more practical code? - Reimplementing code in a manner which simplifies previously complicated bottlenecks that were there in response to bug reports? (and whose simplification potentially risks reintroducing said bugs again) In all honesty, I would expect that "reimplementing coreutils" as above would have resulted in a speedup even if it was written in c again. Am I wrong? Is there something about rust that inherently leads to an increase in speed which one could not ever hope to obtain with clean, performant c code?
- StefanKarpinski 5y ago
  Sometimes I would imagine (I’m no expert in Rust but do know a bit about compilers), Rust’s ability to guarantee unshared access to memory can probably enable optimizations that are hard to coax out of a C compiler. Many libc functions are also much slower than they could be because of POSIX requirements and being a shared library. For example, libc’s fwrite, fread, etc. functions are threadsafe by default and acquire a globally shared lock even when you aren’t using threads (you can opt out, but it’s quite annoying and non-standard) which makes them horribly slow if you’re doing lots of small reads and writes. Because libc is a shared library, calls to its functions won’t get inlined, which can be a major performance issue as well. By comparison Rust’s read and write primitives don’t need to acquire a lock and can be inlined, so a small read or write (for example) could be just a couple of instructions instead of what a C program will do, which is a function call (maybe even an indirect one through the PLT) and then a lock acquisition, only to write a few bytes into a buffer. That’s a lot of overhead! And finally, Rust’s promise of safe multithreading no doubt encourages programmers to write code that utilizes threads in situations where only the truly courageous would attempt it in C.

187 points by marcodiego 5y ago

According to the graph more than half of the tests are still failing. Can the speed be (to some extent) attributed to some missing functionality ?

sylvestre 5y ago
Good question! Probably not. In general, functionalities are enabling or disabling a behavior when doing an operation. In the code, it translates most of the time by a simple if else. For example, adding new options usually looks like this PR: https://github.com/uutils/coreutils/pull/2880/files https://github.com/uutils/coreutils/pull/2880/files The performance wins are usually produced by using some fancy Rust features.
- alerighi 5y ago
  Well, they could be doing some simplifications and assumptions that can improve performance. Also performance is not always the only factor to consider: you can optimize a program for speed, but also for RAM usage, or the size of the binary itself. In program like Coreutils to me is more important that the programs are small than the rest. Typically you use a lot of commands in a script, to do some trivial operations (the input of the program is usually small), thus simpler programs (that have less startup time) are usually better.
  - nix23 5y ago
    >Coreutils to me is more important that the programs are small than the rest Then use BusyBox/Matchbox....because..well that's the whole point of those projects. https://busybox.net/about.html https://busybox.net/about.html
  - zaarn 5y ago
    After the first time the binary is executed, it should be comfortably in the memory disk cache. And frankly, on a modern SSD, the sector size is large enough that the only difference is reading 3 instead of 1 sectors to call "ls". Barely matters if it gets batched.
- laumars 5y ago
  That’s only correct if you focus exclusively on optional features enabled by feature flags or environmental variables. Some features might be around support for things like Unicode where a tool could work 90% of the time but that extra 10% of support requires implementing a lot of additional logic that slows the routine down for the other 90% of use cases too. Also ‘if’ causes branching which costs a small amount of CPU overhead. So it’s not a free operation and can quickly add up if you needed inside a hot path.
  - devit 5y ago
    For new flags, this is easily solved by generating and compiling (via generic monomorphization, macros or build-time code generation) two versions of all the relevant code (including of course any loops calling the relevant code) and switching the one that is executed depending on the value of the flag.
    - tomsmeding 5y ago
      This is true, but also results in larger binaries. Elsewhere in this thread, it was noted that the coreutils compiled from the Rust code are already quite a bit bigger than the GNU counterparts. On many systems this might not matter much (and I guess it's fine if a software suite explicitly makes binary size a non-goal), but on some systems it does.
  - nicoburns 5y ago
    > Some features might be around support for things like Unicode where a tool could work 90% of the time but that extra 10% of support requires implementing a lot of additional logic that slows the routine down for the other 90% of use cases too. This kind of thing is where Rust really shines. The ecosystem was built post-unicode, so things tend to support it by default. Ripgrep for example has been unicode aware from the beginning, and you have to opt-out if you don't want that.
    - laumars 5y ago
      I’m aware of Rusts support for Unicode, I was only using that as an example because it’s easy to visualise since most people who’ve written any kind of text parsing will understand the additional computational overhead correctly supporting Unicode costs. But while the example doesn’t directly apply to Rust, I guarantee you that there will be other edge cases in a similar vein that might cause issues.
tialaramex 5y ago
Slightly less than half, but yes. That could certainly be relevant. On the other hand inevitably some tests will be fragile or even outright wrong and so while your solution passes a different (possibly better/ faster) solution fails because the test is bad. For example suppose you're implementing case-insensitive sort, you write a test and tweak it slightly so that it passes as you expected. I come along and write a slightly faster case-insensitive sort, and mine fails. Upon examining the test I discover it thinks I ought to sort (rat, doG, cat, DOG, dog, DOg) into (cat, doG, dog, DOG, DOg, rat) but I get (cat, doG, DOG, dog, DOg, rat) my answer seems, if anything better and certainly not wrong but it fails your test.
- AndrewDucker 5y ago
  214 passes, 298 fails. That seems rather more than 50% failing, even before the errors and skipped tests.
  - tialaramex 5y ago
    There are 611 tests, 298 is less than half.
    - mlindner 5y ago
      There are tests that are erroring as well. 214 passing out of 611 is less than 50% passing (in fact it's almost only 1/3 passing).
- tharkun__ 5y ago
  The test is right because it tests for the fact that lower case letters have a smaller representation as an ASCII character position. There is probably a ton of scripting and other software out there that assumes that this is how things are sorted. You also want a stable sort meaning sorting the same sequence twice you want it to be sorted the exact same way again. So you need this sort of definition of how things are supposed to be sorted. So while I agree that without historical context and compatibility for human consumption your way of sorting is probably fine, you could wreak major havoc when trying your sort as a drop in replacement. If you only have a few users change management for something like this is relatively easy. The user base of coreutils? Think twice if you want to try that change management.
cedilla 5y ago
It is possible, but with benchmarks like "head -n 1000000 wikidata.xml" I doubt it. A comment in that PR says "the difference to GNU head is mostly in user time, not in system time. I suspect this is due to GNU head not using SIMD to detect newlines". Unfortunately I couldn't find a list of failed/successful tests, if that's available I'd be happy if someone linked it
- sylvestre 5y ago
  https://github.com/uutils/coreutils/actions https://github.com/uutils/coreutils/actions click on a CI job on main For example: https://github.com/uutils/coreutils/runs/4990891225?check_suite_focus=true https://github.com/uutils/coreutils/runs/4990891225?check_su... The "Run GNU tests" is probably what you are looking for.

The_rationalist 5y ago

How about making all ELF binaries symbol processing/retrieving faster? The gnu hashmap is very much obscolete and should be replaced by Swisstable. Key observations like this will keep being ignored for the decades to come.

cosmiccatnap 5y ago

MailNerd 5y ago

Great work! Looking forward to a Linux system where the majority of user land is written in a safe and performant language!

pjmlp 5y ago
It is not the first effort to do so, hopefully this one has more wind to come through.

Aldo_MX 5y ago

Instead of 30 to 60 patches per month, we jumped to 400 to 472 patches every month. Similarly, we saw an increase in the number of contributors (20 to 50 per month from 3 to 8). More contributors = more hands to fix issues and tune performance. I believe this is an absolute win.

hobofan 5y ago

Am I missing something, or how does a PR that bumps the version of an existing brew formula translate to "Brew is proposing coreutils"?

agumonkey 5y ago

Anybody has stats on which coreutils programs are the most used ? I wonder the impact of gnu userland on system perf.

sylvestre 5y ago
I don't have data but I guess ls, cp, mv, ln, chown, chmod, sort and cut are the most popular. Overall, I don't think it is a key part of the OS regular work. Most of the servers and system are busy with services, db, browsers, etc...

jacquesm 5y ago

An old mantra that has served me well: first make it work, then make it fast. So first get it to 100% compatibility, then, and only then concentrate on performance. Because if you don't do it that way you will end up foregoing compatibility because you'd have to say goodbye to your beautifully tuned code that unfortunately can't be 100% compatible. As long as it does not pass all the tests: it does not work. Even if it works for some selected cases, the devil is in the details, and those details can eat up performance like there is no tomorrow.

sva_ 5y ago
The coreutils consist of many individual binaries ('utils') which are more or less stand-alone[0], so I think this mantra doesn't apply here. [0] https://github.com/uutils/coreutils/tree/main/src/uu https://github.com/uutils/coreutils/tree/main/src/uu
- pjmlp 5y ago
  It does, as many times as there are binaries.
- btdmaster 5y ago
  It might not apply for the whole project, but it certainly does apply for each individual per-command functionality (cp, mv, or even the general cross-command features like the filesystem interface).
- jacquesm 5y ago
  It absolutely does. Each utility is a program all by itself and the compatibility tests apply to each and every one of them. How fast they are is not relevant until they are 100% drop in replacements.
draw_down 5y ago
notdonspaulding 5y ago
I follow this advice with one small modification. Namely, mine goes: - Make it once. - Make it right. - Make it fast. "Make it right" as the first step can trick an unseasoned developer into never finishing a prototype. I don't mind the first iteration of something being sloppy and then pursuing correctness with an existing solution to the problem in hand. I'm curious if you've got a sense of the interplay between prototyping and correctness similar to your sense of the interplay between performance tuning and correctness? Any thoughts?

hiccuphippo 5y ago

If one were to replace coreutils in a GNU+Linux system with this, which is MIT licensed, would it be fair to call it MIT+Linux?

wiz21c 5y ago
Yeah, I see the pun. But leaving the GPL license is a very bad idea in my opinion. If we have to rest son shoulder of giant, I'd prefer them being made of concrete, not dust. Coreutils are really a common and should be protected against appropriation. Depending on how much a rewrite it is, GNU's intellectual property may be enforced at some point. But that's a tricky question for a lawyer. If it were me, I'd have asked GNU first.
- mustache_kimono 5y ago
  Um, GNU coreutils is a reimplementation of various AT&T UNIX utilities. It's irresponsible to say GNU has IP rights to this implementation, especially where GNU has done a similar reimplementation(!), and where EFF, etc., has made contrary arguments, as amici, in cases like Oracle v. Google.
  - wiz21c 5y ago
    Agreed, I was completely off on this one :-/
- nickysielicki 5y ago
  Nobody is going to steal a coreutils implementation, make a bunch of internal contributions to it, and yield a competitive advantage through that. The whole point of coreutils is that it’s old and stable and coreish. In the real world, corporate engineering middle managers do not understand the GPL and do not understand how it does and doesn’t restrict them. They avoid it rather than taking the time to learn it.
codeflo 5y ago
I also think “GNU+Linux” is a terrible name and a worse marketing move, but you might misunderstand what it’s about. There’s no “GNU license”, and nobody ever proposed calling the system “GPL+Linux”. GNU sees itself as a project to replace the entirety Unix (hence the name, GNU’s Not Unix). A kernel is one part of the full OS, so if you combine the Linux kernel with the rest of GNU’s OS, according to this logic you get GNU+Linux. (Or maybe it should be GNU-Hurd+Linux, or GNU=~s/Hurd/Linux/... it's terrible.)

est31 5y ago

This is an extremely amazing project because it's actually trying to come up with a replacement that can be used as such. Nobody is going to rewrite scripts to use exa instead of ls for example. The coreutils are also somewhat the "basis" of the shell userspace. Outside of the test failures and missing features, the only reason one might not want the coreutils is the size. Right now, cat on my system is 44K large, while the default output size for the cat executable in release mode is 4.4 megabytes in release mode. If you enable a bunch of options in Cargo.toml to reduce the default bloaty settings (lto = true, codegen-units = 1, strip = true, debug = false), you get that down to 876K. Which is still 20 times larger than the native cat. For true, it's a similar story. 40K vs 812K.

sylvestre 5y ago
I added some CI task and replayed the history to graph the evolution of the size: https://github.com/uutils/coreutils-tracking#binary-size-evolution https://github.com/uutils/coreutils-tracking#binary-size-evo... This will help making sure that we aren't regressing (more) ;)
bsdetector 5y ago
It'd also be a great time to establish a better standard for using these programs with a shell or scripting. For example, if ls had a --shell option that would write out file information as quoted eval'able variables, or even JSON, or anything that was easily and reliably parsable it would remove a huge portion of scripting headaches and errors.
andai 5y ago
Is that a typical ratio for software ported to Rust? Where is all the extra size coming from?
- rpeden 5y ago
  As someone else mentioned, shared libraries. If you built ls and statically linked libc and whatever else it needs, ls would be a lot bigger.
- 5y ago

gravypod 5y ago

If you needed to do low level systems programming in Rust would you be able to use this as an OS abstraction layer by importing these sorts of things: https://github.com/uutils/coreutils/blob/main/src/uu/chmod/src/chmod.rs#L196 https://github.com/uutils/coreutils/blob/main/src/uu/chmod/s... Is this one of the intentions of this team? It sounds like it could make "scripting" in rust very nice if all of the CLI functions you're used to exist as language libraries.

terts 5y ago
That is a very interesting idea! It is not something we've discussed before as far as I know. For some utils (like chmod) it might be possible, but any util that is focused on outputting information currently does so by printing directly to stdout. So, ls doesn't give you a Vec or Iterator of listed files for instance, but instead prints them to stdout. Nevertheless, it would be a cool experiment to see if we can create something like a "ulib" crate that would provide the functions some functions for which it is possible.
- Arcterus 5y ago
  I believe I actually had a proof of concept of this functionality in a PR at some point. This would basically have to work like mesabox, where each utility takes some generic input and output and then runs based on them rather than using stdout and friends directly (e.g. to avoid blowing through all your memory by stuffing many gigabytes of output into a Vec on certain commands).
loeg 5y ago
That would be much more elegant than something like libxo.

RcouF1uZ4gsC 5y ago

I can see this project really taking off. It has two advantages on the GNU version apart from just memory safety with Rust. 1) Ease of portably building. Just type ‘cargo build —release’ and it will just work, even on Windows. 2) MIT License. A company can take these and distribute them as part of a commercial offering without having to worry about GPL compliance.

trelane 5y ago
I'm not sure how point 2 is and advantage for anyone but the company tbh. It's also going to be _really_ hard to be more portable than GNU coreutils, when it comes to platforms it's available on.
- nickysielicki 5y ago
  The whole value of coreutils is that it’s old and stable. I just can’t imagine someone making major contributions to rust coreutils and keeping them internal as some sort of competitive advantage. I do embedded linux for work and while OpenEmbedded makes it pretty easy to manage license obligations, it’s always a pain in the ass to deal with GPL code in a larger team where people will find any excuse to bikeshed about their misunderstandings of the GPL. For questionable historical reasons, my current job manually whitelists every GPL package we ship, so it being MIT makes things less abrasive to teams like mine. That’s an advantage for everyone, because it makes me more likely to use it and it makes me more likely to submit bug reports and fixes.

StefanKarpinski 5y ago

An interesting aspect of this effort is that it undoubtesly makes modifying these tools much easier. While GNU and the FSF are all about legally ensuring the ability to read and modify their code, they have, in practice, made it almost impossible to do so for entirely technical reasons. I’m a pretty good C programmer and whenever I’ve looked at even simple GNU tools like `cat` or god forbid GNU libc’s `printf` code (this one is actually kind of delightful it’s so nuts), I have despaired of understanding, let alone modifying, the code. GNU coreutils are such a morass of preprocessor defines and bonkers C code to support every legacy system GNU has ever run on, that it might as well only be distributed in binary form. I think it would literally be easier to decompile most GNU tools and modify the decompiled platform-specific source than to make the same modification to the source as shipped. Rust has a steep learning curve, but learning Rust is a picnic compared to figuring out how to modify just one GNU tool, and of course they’re all different. Rust puts the onus of portability where it belongs: in the language and compiler, instead of forcing it on every single programmer and application.

melissalobos 5y ago
I have seen it argued many times that the GNU codebases are deliberately written to be unintelligible to avoid claims of being derived from UNIX sources. If they were written in the most straightforward ways, then they would have to show that no-one working on it has seen the original sources. I believe there is even a section in the glibc manual discussing this idea.
- StefanKarpinski 5y ago
  Awful but plausible.
- smoldesu 5y ago
  Sometimes it's obscenely complicated because the GNU devs like showing off with even the simplest of programs. My favorite example is GNU yes, a coreutil that's so ludicrously fast[0] that it can cause OOM states in mere seconds if you're not careful where you're redirecting output. Other HN users have compared the GNU version[1] with the BSD one[2], and I think it's a pretty good way to grok the difference in style between the two. In any case, both tools work perfectly fine for pretty much every use case. [0] https://www.reddit.com/r/unix/comments/6gxduc/how_is_gnu_yes_so_fast/ https://www.reddit.com/r/unix/comments/6gxduc/how_is_gnu_yes... [1] https://github.com/coreutils/coreutils/blob/master/src/yes.c https://github.com/coreutils/coreutils/blob/master/src/yes.c [2] https://github.com/openbsd/src/blob/master/usr.bin/yes/yes.c https://github.com/openbsd/src/blob/master/usr.bin/yes/yes.c

j16sdiz 5y ago

Iirc, most of the gnu coreutil came from its locale handling. It supports lots of different legacy encoding.

tpoacher 5y ago

I do wonder, to what extent is this "increase in speed" the result of: - Refactoring out buggy, convoluted, highly-backwards-compliant code for cleaner more practical code? - Reimplementing code in a manner which simplifies previously complicated bottlenecks that were there in response to bug reports? (and whose simplification potentially risks reintroducing said bugs again) In all honesty, I would expect that "reimplementing coreutils" as above would have resulted in a speedup even if it was written in c again. Am I wrong? Is there something about rust that inherently leads to an increase in speed which one could not ever hope to obtain with clean, performant c code?

StefanKarpinski 5y ago
Sometimes I would imagine (I’m no expert in Rust but do know a bit about compilers), Rust’s ability to guarantee unshared access to memory can probably enable optimizations that are hard to coax out of a C compiler. Many libc functions are also much slower than they could be because of POSIX requirements and being a shared library. For example, libc’s fwrite, fread, etc. functions are threadsafe by default and acquire a globally shared lock even when you aren’t using threads (you can opt out, but it’s quite annoying and non-standard) which makes them horribly slow if you’re doing lots of small reads and writes. Because libc is a shared library, calls to its functions won’t get inlined, which can be a major performance issue as well. By comparison Rust’s read and write primitives don’t need to acquire a lock and can be inlined, so a small read or write (for example) could be just a couple of instructions instead of what a C program will do, which is a function call (maybe even an indirect one through the PLT) and then a lock acquisition, only to write a few bytes into a buffer. That’s a lot of overhead! And finally, Rust’s promise of safe multithreading no doubt encourages programmers to write code that utilizes threads in situations where only the truly courageous would attempt it in C.