37 ms·

The Rust Implementation of GNU Coreutils Is Becoming Remarkably Robust

261 points by sdht0 4y ago

nishs 4y ago
from the slides: > Lots of crates (Rust libraries) available - Don't have to reimplement the wheel: > lscolors, walkdir, tempfile, terminal_size i believe this isn't always quite an advantage that the slides make it out to be when implementing tools as critical as coreutils. you typically would want internal packages that you can precisely control.
- sophacles 4y ago
  Why?
  - SQueeeeeL 4y ago
    Not the original poster, but coreutils are almost given similar design requirements as airplane software. They need to be fast and they need to be perfect. Having a dependency tree which invokes items that (potentially) don't place as much emphasis on being error free can lead to buggy software. Remember, there is no reason for anyone to choose the rust implementation, other than philosophy. They need to match or surpass the C implementations with an extreme degree of consistency to be worth the risk of transferring.
    - Cthulhu_ 4y ago
      > Not the original poster, but coreutils are almost given similar design requirements as airplane software. I mean, a lot probably are used in literal airplane software. It also has to be good from as early as possible; some software still in use uses decades old code and libraries which cannot easily be replaced (think also embedded software).
    - sophacles 4y ago
      So they vetted a library and chose a version. That version is the version they will use until they choose another version - that's how rust works. It's not npm, you specify a specific version and that's the code that's used, no matter how upstream changes things. It's not C where you load some random .so with the right name and hope that it's compatible (or have entire giant systems built around managing library compatabilites ala Linux distros). I've got code in prod that uses pre-async versions of tokio and it still builds and runs just fine with the latest rust nightly. If there does turn out to be a problem with the version of some library I chose, and upstream has become incompatible, nothing stops me from vendoring the upstream and fixing the problem my own way. Until then, cargo/crates/rust guarantee that the code I vetted and chose is the code I'll build with. Why is it so vital if the sequence of bytes is stored one place or another?
      - MrJohz 4y ago
        > It's not npm, you specify a specific version and that's the code that's used, no matter how upstream changes things. Note that that _is_ how NPM behaves. (At least, when using a lock file, which is the default behaviour like with Cargo. If you don't use lock files then neither Cargo not NPM can guarantee this property.)
      - nishs 4y ago
        vetting, and dependency locking, addresses nearly all the issues. there's one other issue: the ability to easily adjust the code as you see fit, so it is vital that you are able to precisely control the code. related reply: https://news.ycombinator.com/item?id=34740666 https://news.ycombinator.com/item?id=34740666
- raverbashing 4y ago
  walkdir and tempfile should be simple enough for what they need to do and if it merits a change then that's totally worth taking upstream Rust should definitely have a good "stdlib" or at least an extended base library to be used for stuff like this.
- rocqua 4y ago
  Version pinning and manual verification of correct implementation can work quite well here.
  - nishs 4y ago
    wholeheartedly yes; this addresses nearly all the concerns i had in my comment. the only other concern is the ability and the added time to make changes to these dependencies. what this sometimes means in practice: in terms of time: you may have to wait for upstream to accept your change. alternatively, one could maintain a fork of the package and replace the dependency to point to the fork while waiting for changes to be accepted, however doing so adds back-and-forth work. in terms of ability: upstream may reject a change. after the change is merged upstream, you are required to vet commits in the dependency between the last previously vetted commit and your currently merged commit, all at once, before you can upgrade the dependency in your original project.
- schemescape 4y ago
  How many transitive dependencies result? And what licenses are those crates released under? Edit: I feel like this is often overlooked, but most licenses require including a copy with binary distributions, and wrangling all those text files can be surprisingly cumbersome. Omitting a license can lead to headaches down the road.
synergy20 4y ago
the coreutil rust flavor is 16MB per its release build. after strip it cuts down to 10MB, that's the smallest size you can have. Comparing to c version coreutils, which totals up to 5.8MB, rust does have a slight size "problem", its size is 70% larger even with the busybox-all-in-one style. you're actually forced to do it the busybox style, otherwise each single small utility of coreutils will be about similar size, say, 6MB for each one, then it will just blow up really fast. I commented somewhere else, the rust stdlib by default(AND by design) is statically linked, which is totally different from the shared stdlibs like C and C++ etc, that will leads to large size when you have a few of rust release binaries. I have never figured out why Rust can not do the shared-stdlib -just-like-c-c++ yet.
- sgerenser 4y ago
  Isn’t the problem Rust not having a stable ABI (at least that’s what I heard)? Unstable ABI doesn’t work well with dynamic linking.
  - synergy20 4y ago
    is the 'unstable abi' by design too? or is it just still evolving? why can't Rust have a stable ABI like libc and libstdc++? this is the major reason I did not use Rust so far indeed, and I don't know Rust enough why it is the way it is as far as stdlib-static-link goes.
    - dagmx 4y ago
      It’s partly by design and partly because it’s evolving. the “by design” part is because Rust allows for packing structs in an order independent way. This is a nice optimization but it’s not something that is currently stable between versions.
      - thayne 4y ago
        More generally, stabilizing the ABI can prevent future optimization. Also, including lifetime information in the ABI is a hard problem thay hasn't been solved yet, and you can't have generics in an exposed ABI.
shrubble 4y ago
Robust in comparison to what? Is the current coreutils lacking in robustness?
- doomrobo 4y ago
  It had some CVEs but not many [0]. I think the better argument is that some of the original code is just really hard to read. Click around the repo [1] [0] https://www.cvedetails.com/vulnerability-list.php?vendor_id=72&product_id=5075&version_id=0&page=1&hasexp=0&opdos=0&opec=0&opov=0&opcsrf=0&opgpriv=0&opsqli=0&opxss=0&opdirt=0&opmemc=0&ophttprs=0&opbyp=0&opfileinc=0&opginf=0&cvssscoremin=0&cvssscoremax=0&year=0&cweid=0&order=1&trc=9&sha=5ac91b3849b7eae5db18caec262ad99560af8864 https://www.cvedetails.com/vulnerability-list.php?vendor_id=... [1] https://github.com/coreutils/coreutils/ https://github.com/coreutils/coreutils/
rurban 4y ago
Nice, with mv/cp -g, which was rejected upstream. Now just unorm from Assaf Gordon's multi-byte-20200607.patch is missing: https://github.com/rurban/coreutils/commit/4008663077be6a740751dfd08f664076d34b036e https://github.com/rurban/coreutils/commit/4008663077be6a740...
pjmlp 4y ago
I remember previous attempts to do this in Ada.
- timbit42 4y ago
  I searched for 'implement coreutils ada' but nothing came up. What happened?
- stevekemp 4y ago
  I remember in 1999 there was a project to reimplement a bunch of these tools in perl: https://perlpowertools.com/ https://perlpowertools.com/ I even contributed a little, back then. I guess writing basic versions of "ls", for example, is trivial. But there's a lot of work getting all the tools done, with all the flags implemented and behaving as expected. I guess there are tools like busybox, toybox, and similar, which also implement a lot of "stuff" to varying degrees of completion. From my side the biggest takeaway from those projects is the sheer convenience of deploying a single binary and installing symlinks to change functionality. I replicated something similar with my sysbox project, collecting tools together in one golang binary with various subcommands: https://github.com/skx/sysbox https://github.com/skx/sysbox I use at least one of those tools on a daily basis, though I suspect they're not so universally useful.
  - anthk 4y ago
    Perl was already reimplementing sed/awk and sh in a weird C-ish way, so doing unix utilities would be a piece of cake in Perl.
josephcsible 4y ago
I just wish it was copyleft like the real coreutils instead of being pushover-licensed. Now the corporations are all going to start making proprietary forks of this.
- jeff_carr 4y ago
  > I just wish it was copyleft Yes, the GPL is very important. The BSD license _is_ a problem. Those of you that were around when FreeBSD was maybe going to 'win' remember why Linux(GPL) ended up capturing everyone's attention. You can not count on your code staying free if it's under a BSD license. If there is a way to make money on it, someone will fork it privately and force you to pay for it. If they don't succeed, you will still be under continual threats that you will have to pay for it.
  - rocqua 4y ago
    Why did Linux end up capturing everyone's attention? I wasn't around back then.
- mustache_kimono 4y ago
  FSF/GPL advocate: "I don't like that someone else is reimplementing this code." Jesus, you think s/he ever wondered how AT&T felt about Linux and the BSDs literally reimplementing all of Unix? Oh, that's right. This is the FSF/GPL conundrum -- you're likely to end up on both sides of every issue. You will literally be both a copyright maximalist re Linux and a copyright minimalist re proprietary code, music and film.
- BirAdam 4y ago
  Corporations use copyleft stuff all the time too. FOSS subsidizes mega corporations, which sucks. Thing is, it’s also a subsidy for all of us.
- mouse_ 4y ago
  > pushover-licensed Huh, never heard the polite form of that term.
hulitu 4y ago
> The Rust Implementation of GNU Coreutils Is Becoming Remarkably Robust And performance ?
- snvzz 4y ago
  There's some benchmarks in the talk. It is significantly faster than coreutils in some situations. They also know cases where it is slower, and thus I doubt it'll remain so.
  - KRAKRISMOTT 4y ago
    Code is cleaner too, fewer hacks than typical Coreutils code. You get the benefits of BSD style clean code while still having high performance because there are clear functional boundaries and specific features are compartmentalized in modules and dependencies. Systems programming has finally caught up to 30 years of advances in software engineering. Long live the Rust Evangelism Strike force. Finally Apache/Linux is now possible.
    - riffraff 4y ago
      Why Apache/Linux? Is uutils part of the Apache foundation?
      - allendoerfer 4y ago
        OP is referring to the license.
        riffraff 4y ago
        But that's MIT not Apache
        earthling8118 4y ago
        It's extremely common for Rust projects to be licensed with both of them
AndrewDavis 4y ago
I'm now imagining a distro using relibc (rust implementation of POSIX libc from RedoxOS) and uutils.
- shaunsingh0207 4y ago
  I currently build my nixOS packages with musl, clang, and uutils, and the difference from gcc/coreutils/glibc is unnoticeable. The uutils project is great
- deleted 4y ago
  [deleted]
- rlpb 4y ago
  How does Rust work with dynamic linking? I thought it didn't?
  - aldonius 4y ago
    Rust can dynamically link to/like a library written in C if you really want it to. https://doc.rust-lang.org/nomicon/ffi.html https://doc.rust-lang.org/nomicon/ffi.html
    - rlpb 4y ago
      That's the inverse of my question. If Rust is going to replace things written in C, other stuff is going to want to dynamically link to it. A statically compiled Rust based replacement for an entire distribution isn't a realistic proposition, unless you fancy downloading a gig or two every time there's a security update and everything has to be rebuilt.
      - tsimionescu 4y ago
        Theoretically, you could dynamically link with other Rust code that exposes the standard C ABI. This used to be common for C++ code, when name mangling was different between different compilers and versions - so a C++ library that wanted to be portable had to expose a C ABI, and C++ apps would dynamically link to it by calling that C ABI. Of course, this meant no exceptions, no destructors, no std:: data structures, but such was the price.
throwaway777845 4y ago
are they fully compatible?
- d12bb 4y ago
  They're on their way: https://uutils.github.io/user/test_coverage.html https://uutils.github.io/user/test_coverage.html
0x008 4y ago
How do you measure robustness except with failures / years of service x times deployed ? You cannot make edge cases and strange unique behavior happen in a controlled environment.
- Cthulhu_ 4y ago
  One tool that (I think?) wasn't around, wasn't used as much, or couldn't be deployed at scale back then was fuzzing tools; modern-day application of fuzzers on existing libraries and tools has revealed thousands of potential bugs. If fuzzing, unit tests and things like mutation tests are written and automatically applied to these libraries/utils, on top of being written in a proven memory-safe language like Rust, I would assert they're pretty robust. Second, there's more known knowns this time around; they can write test cases for the decades of bugs and rare occurrences found in the tools they replace. That said, I'm sure there's still plenty of unknown unknowns that can only be uncovered with, as you said, years of service and times deployed.
- yjftsjthsd-h 4y ago
  > You cannot make edge cases and strange unique behavior happen in a controlled environment. Sure you can. Go iterate through the GNU Coreutils bug tracker, find weird bugs, create test cases, feed them to your implementation. Bonus points if you can find an existing test suite (ex. xfstests turns out to be good for testing arbitrary filesystems). Granted, there will always be edge cases that you don't catch until they show up live in prod, but you can hit a chunk of the space without that.
  - kaba0 4y ago
    Also, there are plenty of package system that uses coreutils for their build instructions. Just build the whole thing with coreutils aliased to the rust implementation and check for errors.
  - retcond 4y ago
    You have just contradicted yourself by saying, there will always be edge cases that you don't catch until they show up live in in prod.." in reply to, "How do you measure robustness except with failures / years of service x times deployed ?" The only viable live testing environment that springs to mind might be running your test code synchronously at the atomic level with production, which I'm convinced only IBM Z/OS on a Parallel Sysplex cluster running CICS can do. Ed.spelling
dezgeg 4y ago
How well is this working nowadays with non-UTF8 input?
unixgoddess 4y ago
I wonder why not reimplement coreutils as library functions, to be used within an ad-hoc REPL. It would be much more flexible and extensible. AFAIK, originally the reason why they were made as programs was that the available programming languages were too cumbersome for such use. Now we have plenty of experience making lightweight scripting languages that are pleasant to use in a live environment (vs premade script), so why give up the flexibility of ad-hoc scripting?
- theamk 4y ago
  because most of the coreutil functionality is already availible in libraries of most languages. Article mentions that there are crates for the logic. The hard part is command line parsing and output formatting, and your library should have neither of those. I've seen plenty of shell scripts rewritten in Python because they grew too big, and most of the time coreutil commands just get replaced with standard library calls. There are exceptions (like sorting files which do not fit in memory) but otherwise standard library is good enough
- cleanchit 4y ago
  Inertia
- lelanthran 4y ago
  > I wonder why not reimplement coreutils as library functions, to be used within an ad-hoc REPL. It would be much more flexible and extensible. Doesn't busybox do something similar - (almost) everything is in a single binary.
  - unixgoddess 4y ago
    nope. it's still a traditional binary meant to be used as traditional binaries from the posix shell. What I mean is, replace both the binaries and the shell with a library equivalent of coreutils running from a REPL.
    - ricardobeat 4y ago
harry8 4y ago
How does this work wen you want to add a flag? Eg once upon a time I thought it would be fun to add a flag to ls to limit its results to a certain kind of file so you could list only directories for example. It came up for me as something I needed so I did it. Somebody on the gnu mailing list for core utils rejected it on the basis of ls "having a high bar" for a new flag. $ man ls suggests this hasn't been a consistent policy. It wasn't clear to me if that person was in charge or it was just a vague notion of theirs or anything useful so obviously I dropped it because I'd have something different from someone if there was any interest. The Rust implementation might come to precisely the same conclusion that it isn't worthwhile. But they also might not on some case if not that one. Do they do what's better or do they do what's Gnu always and everywhere? Do they wait until they have significant traction and only then consider such things? Interesting questions for them to ponder and maybe they have?
- terts 4y ago
  One of the maintainers of uutils here. We have a few flags that are not in GNU for one reason or another. Some were rejected by GNU, others come from other coreutils implementations like FreeBSD. We document those at https://uutils.github.io/user/extensions.html https://uutils.github.io/user/extensions.html We tend to do this sparingly, however, because even just adding new flags might break existing scripts that use abbreviated long options. For example, if the flag you propose is called `--filter` it might break scripts that use `--fi` as a shorthand for `--file-type`, because the prefix is now ambiguous. I personally like the flag you propose!
  - emmelaich 4y ago
    FWIW, this particular behaviour can be achieved with `find . -type d -maxdepth 1 ....`
  - efreak 4y ago
    A solution might be too only allow traditional flags to be shorthand, and require the full long option for new flags.
  - pas 4y ago
vbezhenar 4y ago
I wish someone would build modern lean utils. I don't need bazillion grep flags. There's so much unnecessary repetition in unix tools. Why make separate flag for input and output files when I can just redirect input and output with shell, just for one example. Theoretically most unix tools could be implemented in very few lines of code and that's their beauty.
- glutamate 4y ago
  grep implemented in a few lines of code would not be very fast: https://lists.freebsd.org/pipermail/freebsd-current/2010-August/019310.html https://lists.freebsd.org/pipermail/freebsd-current/2010-Aug...
- devnullbrain 4y ago
  >Theoretically most unix tools could be implemented in very few lines of code. So do it?
  - mrtweetyhack 4y ago
    [dead]
- tsimionescu 4y ago
  > Theoretically most unix tools could be implemented in very few lines of code and that's their beauty. As usual, theory and practice should be the same in theory, but they are different in practice. It turns out that the Unix philosophy of gluing together programs by spitting out and re-parsing text is both extraordinarily brittle for automation purposes, and quite tedious for manual work. So, to make realistic use of these tools, there's lots of flags to force the output to be as structured as possible for various automation tasks, to make parsing the output tractable (and to ask for explicit guarantees on the format). Then, to achieve many common tasks without having to write your own parser and text manipulation in awk or whatever, you need many other flags for transformations which are trivial on the base data types but complex for formatted text, such as sorting files by a last modified (which is `ls -lt` with flags, and a terrible amount of work that I'm not going to go into using Unix pipes - at least if you want to support such complex things like file names including whitespace, or dates in the current locale).
vkaku 4y ago
This is great. Chimera Linux uses the FreeBSD userland version of the utilities. It's refreshing to see new distros taking charge of their development.

261 points by sdht0 4y ago

from the slides: > Lots of crates (Rust libraries) available - Don't have to reimplement the wheel: > lscolors, walkdir, tempfile, terminal_size i believe this isn't always quite an advantage that the slides make it out to be when implementing tools as critical as coreutils. you typically would want internal packages that you can precisely control.

sophacles 4y ago
Why?
- SQueeeeeL 4y ago
  Not the original poster, but coreutils are almost given similar design requirements as airplane software. They need to be fast and they need to be perfect. Having a dependency tree which invokes items that (potentially) don't place as much emphasis on being error free can lead to buggy software. Remember, there is no reason for anyone to choose the rust implementation, other than philosophy. They need to match or surpass the C implementations with an extreme degree of consistency to be worth the risk of transferring.
  - Cthulhu_ 4y ago
    > Not the original poster, but coreutils are almost given similar design requirements as airplane software. I mean, a lot probably are used in literal airplane software. It also has to be good from as early as possible; some software still in use uses decades old code and libraries which cannot easily be replaced (think also embedded software).
  - sophacles 4y ago
    So they vetted a library and chose a version. That version is the version they will use until they choose another version - that's how rust works. It's not npm, you specify a specific version and that's the code that's used, no matter how upstream changes things. It's not C where you load some random .so with the right name and hope that it's compatible (or have entire giant systems built around managing library compatabilites ala Linux distros). I've got code in prod that uses pre-async versions of tokio and it still builds and runs just fine with the latest rust nightly. If there does turn out to be a problem with the version of some library I chose, and upstream has become incompatible, nothing stops me from vendoring the upstream and fixing the problem my own way. Until then, cargo/crates/rust guarantee that the code I vetted and chose is the code I'll build with. Why is it so vital if the sequence of bytes is stored one place or another?
    - MrJohz 4y ago
      > It's not npm, you specify a specific version and that's the code that's used, no matter how upstream changes things. Note that that _is_ how NPM behaves. (At least, when using a lock file, which is the default behaviour like with Cargo. If you don't use lock files then neither Cargo not NPM can guarantee this property.)
    - nishs 4y ago
      vetting, and dependency locking, addresses nearly all the issues. there's one other issue: the ability to easily adjust the code as you see fit, so it is vital that you are able to precisely control the code. related reply: https://news.ycombinator.com/item?id=34740666 https://news.ycombinator.com/item?id=34740666
raverbashing 4y ago
walkdir and tempfile should be simple enough for what they need to do and if it merits a change then that's totally worth taking upstream Rust should definitely have a good "stdlib" or at least an extended base library to be used for stuff like this.
rocqua 4y ago
Version pinning and manual verification of correct implementation can work quite well here.
- nishs 4y ago
  wholeheartedly yes; this addresses nearly all the concerns i had in my comment. the only other concern is the ability and the added time to make changes to these dependencies. what this sometimes means in practice: in terms of time: you may have to wait for upstream to accept your change. alternatively, one could maintain a fork of the package and replace the dependency to point to the fork while waiting for changes to be accepted, however doing so adds back-and-forth work. in terms of ability: upstream may reject a change. after the change is merged upstream, you are required to vet commits in the dependency between the last previously vetted commit and your currently merged commit, all at once, before you can upgrade the dependency in your original project.
schemescape 4y ago
How many transitive dependencies result? And what licenses are those crates released under? Edit: I feel like this is often overlooked, but most licenses require including a copy with binary distributions, and wrangling all those text files can be surprisingly cumbersome. Omitting a license can lead to headaches down the road.

synergy20 4y ago

the coreutil rust flavor is 16MB per its release build. after strip it cuts down to 10MB, that's the smallest size you can have. Comparing to c version coreutils, which totals up to 5.8MB, rust does have a slight size "problem", its size is 70% larger even with the busybox-all-in-one style. you're actually forced to do it the busybox style, otherwise each single small utility of coreutils will be about similar size, say, 6MB for each one, then it will just blow up really fast. I commented somewhere else, the rust stdlib by default(AND by design) is statically linked, which is totally different from the shared stdlibs like C and C++ etc, that will leads to large size when you have a few of rust release binaries. I have never figured out why Rust can not do the shared-stdlib -just-like-c-c++ yet.

sgerenser 4y ago
Isn’t the problem Rust not having a stable ABI (at least that’s what I heard)? Unstable ABI doesn’t work well with dynamic linking.
- synergy20 4y ago
  is the 'unstable abi' by design too? or is it just still evolving? why can't Rust have a stable ABI like libc and libstdc++? this is the major reason I did not use Rust so far indeed, and I don't know Rust enough why it is the way it is as far as stdlib-static-link goes.
  - dagmx 4y ago
    It’s partly by design and partly because it’s evolving. the “by design” part is because Rust allows for packing structs in an order independent way. This is a nice optimization but it’s not something that is currently stable between versions.
    - thayne 4y ago
      More generally, stabilizing the ABI can prevent future optimization. Also, including lifetime information in the ABI is a hard problem thay hasn't been solved yet, and you can't have generics in an exposed ABI.

shrubble 4y ago

Robust in comparison to what? Is the current coreutils lacking in robustness?

doomrobo 4y ago
It had some CVEs but not many [0]. I think the better argument is that some of the original code is just really hard to read. Click around the repo [1] [0] https://www.cvedetails.com/vulnerability-list.php?vendor_id=72&product_id=5075&version_id=0&page=1&hasexp=0&opdos=0&opec=0&opov=0&opcsrf=0&opgpriv=0&opsqli=0&opxss=0&opdirt=0&opmemc=0&ophttprs=0&opbyp=0&opfileinc=0&opginf=0&cvssscoremin=0&cvssscoremax=0&year=0&cweid=0&order=1&trc=9&sha=5ac91b3849b7eae5db18caec262ad99560af8864 https://www.cvedetails.com/vulnerability-list.php?vendor_id=... [1] https://github.com/coreutils/coreutils/ https://github.com/coreutils/coreutils/

rurban 4y ago

Nice, with mv/cp -g, which was rejected upstream. Now just unorm from Assaf Gordon's multi-byte-20200607.patch is missing: https://github.com/rurban/coreutils/commit/4008663077be6a740751dfd08f664076d34b036e https://github.com/rurban/coreutils/commit/4008663077be6a740...

pjmlp 4y ago

I remember previous attempts to do this in Ada.

timbit42 4y ago
I searched for 'implement coreutils ada' but nothing came up. What happened?
stevekemp 4y ago
I remember in 1999 there was a project to reimplement a bunch of these tools in perl: https://perlpowertools.com/ https://perlpowertools.com/ I even contributed a little, back then. I guess writing basic versions of "ls", for example, is trivial. But there's a lot of work getting all the tools done, with all the flags implemented and behaving as expected. I guess there are tools like busybox, toybox, and similar, which also implement a lot of "stuff" to varying degrees of completion. From my side the biggest takeaway from those projects is the sheer convenience of deploying a single binary and installing symlinks to change functionality. I replicated something similar with my sysbox project, collecting tools together in one golang binary with various subcommands: https://github.com/skx/sysbox https://github.com/skx/sysbox I use at least one of those tools on a daily basis, though I suspect they're not so universally useful.
- anthk 4y ago
  Perl was already reimplementing sed/awk and sh in a weird C-ish way, so doing unix utilities would be a piece of cake in Perl.

josephcsible 4y ago

I just wish it was copyleft like the real coreutils instead of being pushover-licensed. Now the corporations are all going to start making proprietary forks of this.

jeff_carr 4y ago
> I just wish it was copyleft Yes, the GPL is very important. The BSD license _is_ a problem. Those of you that were around when FreeBSD was maybe going to 'win' remember why Linux(GPL) ended up capturing everyone's attention. You can not count on your code staying free if it's under a BSD license. If there is a way to make money on it, someone will fork it privately and force you to pay for it. If they don't succeed, you will still be under continual threats that you will have to pay for it.
- rocqua 4y ago
  Why did Linux end up capturing everyone's attention? I wasn't around back then.
mustache_kimono 4y ago
FSF/GPL advocate: "I don't like that someone else is reimplementing this code." Jesus, you think s/he ever wondered how AT&T felt about Linux and the BSDs literally reimplementing all of Unix? Oh, that's right. This is the FSF/GPL conundrum -- you're likely to end up on both sides of every issue. You will literally be both a copyright maximalist re Linux and a copyright minimalist re proprietary code, music and film.
BirAdam 4y ago
Corporations use copyleft stuff all the time too. FOSS subsidizes mega corporations, which sucks. Thing is, it’s also a subsidy for all of us.
mouse_ 4y ago
> pushover-licensed Huh, never heard the polite form of that term.

hulitu 4y ago

> The Rust Implementation of GNU Coreutils Is Becoming Remarkably Robust And performance ?

snvzz 4y ago
There's some benchmarks in the talk. It is significantly faster than coreutils in some situations. They also know cases where it is slower, and thus I doubt it'll remain so.
- KRAKRISMOTT 4y ago
  Code is cleaner too, fewer hacks than typical Coreutils code. You get the benefits of BSD style clean code while still having high performance because there are clear functional boundaries and specific features are compartmentalized in modules and dependencies. Systems programming has finally caught up to 30 years of advances in software engineering. Long live the Rust Evangelism Strike force. Finally Apache/Linux is now possible.
  - riffraff 4y ago
    Why Apache/Linux? Is uutils part of the Apache foundation?
    - allendoerfer 4y ago
      OP is referring to the license.
      - riffraff 4y ago
        But that's MIT not Apache
        earthling8118 4y ago
        It's extremely common for Rust projects to be licensed with both of them

AndrewDavis 4y ago

I'm now imagining a distro using relibc (rust implementation of POSIX libc from RedoxOS) and uutils.

shaunsingh0207 4y ago
I currently build my nixOS packages with musl, clang, and uutils, and the difference from gcc/coreutils/glibc is unnoticeable. The uutils project is great
deleted 4y ago
[deleted]
rlpb 4y ago
How does Rust work with dynamic linking? I thought it didn't?
- aldonius 4y ago
  Rust can dynamically link to/like a library written in C if you really want it to. https://doc.rust-lang.org/nomicon/ffi.html https://doc.rust-lang.org/nomicon/ffi.html
  - rlpb 4y ago
    That's the inverse of my question. If Rust is going to replace things written in C, other stuff is going to want to dynamically link to it. A statically compiled Rust based replacement for an entire distribution isn't a realistic proposition, unless you fancy downloading a gig or two every time there's a security update and everything has to be rebuilt.
    - tsimionescu 4y ago
      Theoretically, you could dynamically link with other Rust code that exposes the standard C ABI. This used to be common for C++ code, when name mangling was different between different compilers and versions - so a C++ library that wanted to be portable had to expose a C ABI, and C++ apps would dynamically link to it by calling that C ABI. Of course, this meant no exceptions, no destructors, no std:: data structures, but such was the price.

throwaway777845 4y ago

are they fully compatible?

d12bb 4y ago
They're on their way: https://uutils.github.io/user/test_coverage.html https://uutils.github.io/user/test_coverage.html

0x008 4y ago

How do you measure robustness except with failures / years of service x times deployed ? You cannot make edge cases and strange unique behavior happen in a controlled environment.

Cthulhu_ 4y ago
One tool that (I think?) wasn't around, wasn't used as much, or couldn't be deployed at scale back then was fuzzing tools; modern-day application of fuzzers on existing libraries and tools has revealed thousands of potential bugs. If fuzzing, unit tests and things like mutation tests are written and automatically applied to these libraries/utils, on top of being written in a proven memory-safe language like Rust, I would assert they're pretty robust. Second, there's more known knowns this time around; they can write test cases for the decades of bugs and rare occurrences found in the tools they replace. That said, I'm sure there's still plenty of unknown unknowns that can only be uncovered with, as you said, years of service and times deployed.
yjftsjthsd-h 4y ago
> You cannot make edge cases and strange unique behavior happen in a controlled environment. Sure you can. Go iterate through the GNU Coreutils bug tracker, find weird bugs, create test cases, feed them to your implementation. Bonus points if you can find an existing test suite (ex. xfstests turns out to be good for testing arbitrary filesystems). Granted, there will always be edge cases that you don't catch until they show up live in prod, but you can hit a chunk of the space without that.
- kaba0 4y ago
  Also, there are plenty of package system that uses coreutils for their build instructions. Just build the whole thing with coreutils aliased to the rust implementation and check for errors.
- retcond 4y ago
  You have just contradicted yourself by saying, there will always be edge cases that you don't catch until they show up live in in prod.." in reply to, "How do you measure robustness except with failures / years of service x times deployed ?" The only viable live testing environment that springs to mind might be running your test code synchronously at the atomic level with production, which I'm convinced only IBM Z/OS on a Parallel Sysplex cluster running CICS can do. Ed.spelling

dezgeg 4y ago

How well is this working nowadays with non-UTF8 input?

unixgoddess 4y ago

I wonder why not reimplement coreutils as library functions, to be used within an ad-hoc REPL. It would be much more flexible and extensible. AFAIK, originally the reason why they were made as programs was that the available programming languages were too cumbersome for such use. Now we have plenty of experience making lightweight scripting languages that are pleasant to use in a live environment (vs premade script), so why give up the flexibility of ad-hoc scripting?

theamk 4y ago
because most of the coreutil functionality is already availible in libraries of most languages. Article mentions that there are crates for the logic. The hard part is command line parsing and output formatting, and your library should have neither of those. I've seen plenty of shell scripts rewritten in Python because they grew too big, and most of the time coreutil commands just get replaced with standard library calls. There are exceptions (like sorting files which do not fit in memory) but otherwise standard library is good enough
cleanchit 4y ago
Inertia
lelanthran 4y ago
> I wonder why not reimplement coreutils as library functions, to be used within an ad-hoc REPL. It would be much more flexible and extensible. Doesn't busybox do something similar - (almost) everything is in a single binary.
- unixgoddess 4y ago
  nope. it's still a traditional binary meant to be used as traditional binaries from the posix shell. What I mean is, replace both the binaries and the shell with a library equivalent of coreutils running from a REPL.
  - ricardobeat 4y ago

harry8 4y ago

How does this work wen you want to add a flag? Eg once upon a time I thought it would be fun to add a flag to ls to limit its results to a certain kind of file so you could list only directories for example. It came up for me as something I needed so I did it. Somebody on the gnu mailing list for core utils rejected it on the basis of ls "having a high bar" for a new flag. $ man ls suggests this hasn't been a consistent policy. It wasn't clear to me if that person was in charge or it was just a vague notion of theirs or anything useful so obviously I dropped it because I'd have something different from someone if there was any interest. The Rust implementation might come to precisely the same conclusion that it isn't worthwhile. But they also might not on some case if not that one. Do they do what's better or do they do what's Gnu always and everywhere? Do they wait until they have significant traction and only then consider such things? Interesting questions for them to ponder and maybe they have?

terts 4y ago
One of the maintainers of uutils here. We have a few flags that are not in GNU for one reason or another. Some were rejected by GNU, others come from other coreutils implementations like FreeBSD. We document those at https://uutils.github.io/user/extensions.html https://uutils.github.io/user/extensions.html We tend to do this sparingly, however, because even just adding new flags might break existing scripts that use abbreviated long options. For example, if the flag you propose is called `--filter` it might break scripts that use `--fi` as a shorthand for `--file-type`, because the prefix is now ambiguous. I personally like the flag you propose!
- emmelaich 4y ago
  FWIW, this particular behaviour can be achieved with `find . -type d -maxdepth 1 ....`
- efreak 4y ago
  A solution might be too only allow traditional flags to be shorthand, and require the full long option for new flags.
- pas 4y ago

vbezhenar 4y ago

I wish someone would build modern lean utils. I don't need bazillion grep flags. There's so much unnecessary repetition in unix tools. Why make separate flag for input and output files when I can just redirect input and output with shell, just for one example. Theoretically most unix tools could be implemented in very few lines of code and that's their beauty.

glutamate 4y ago
grep implemented in a few lines of code would not be very fast: https://lists.freebsd.org/pipermail/freebsd-current/2010-August/019310.html https://lists.freebsd.org/pipermail/freebsd-current/2010-Aug...
devnullbrain 4y ago
>Theoretically most unix tools could be implemented in very few lines of code. So do it?
- mrtweetyhack 4y ago
  [dead]
tsimionescu 4y ago
> Theoretically most unix tools could be implemented in very few lines of code and that's their beauty. As usual, theory and practice should be the same in theory, but they are different in practice. It turns out that the Unix philosophy of gluing together programs by spitting out and re-parsing text is both extraordinarily brittle for automation purposes, and quite tedious for manual work. So, to make realistic use of these tools, there's lots of flags to force the output to be as structured as possible for various automation tasks, to make parsing the output tractable (and to ask for explicit guarantees on the format). Then, to achieve many common tasks without having to write your own parser and text manipulation in awk or whatever, you need many other flags for transformations which are trivial on the base data types but complex for formatted text, such as sorting files by a last modified (which is `ls -lt` with flags, and a terrible amount of work that I'm not going to go into using Unix pipes - at least if you want to support such complex things like file names including whitespace, or dates in the current locale).

vkaku 4y ago

This is great. Chimera Linux uses the FreeBSD userland version of the utilities. It's refreshing to see new distros taking charge of their development.