4 ms·

I'm more so interested in performance and size comparisons. Most different implementations of the coreutils have different goals in mind. GNU's implementation,

I'm more so interested in performance and size comparisons. Most different implementations of the coreutils have different goals in mind. GNU's implementation, as is common of GNU is to treat memory for performance, and busybox's attempts to achieve the converse. — so what is the purpose of this one? simply being written in Rust? How does it compare to the established ones?

dastx 6y ago
I'd love to know too. It seems like it's nearly at feature parity but if performance is an issue, it's not really a replacement.
- Blikkentrekker 6y ago
  GNU coreutils are quite fast by the way. https://lists.freebsd.org/pipermail/freebsd-current/2010-August/019310.html https://lists.freebsd.org/pipermail/freebsd-current/2010-Aug... This is an interesting article that outlines why GNU grep outperforms almost any other grep quite significantly, at the cost of more memory.
  - codethief 6y ago
    That statement surprises me because my impression had always been that GNU grep is rather slow compared to other (more modern) tools, see e.g. the benchmarks on https://github.com/BurntSushi/ripgrep https://github.com/BurntSushi/ripgrep and https://blog.burntsushi.net/ripgrep/ https://blog.burntsushi.net/ripgrep/ .
    - duckerude 6y ago
      GNU grep is so fast that it set the bar for later implementations. In your second link ripgrep is even marketed as having "the raw performance of GNU grep" (though it managed to exceed it).
    - arp242 6y ago
      When that was written in 2010 it probably was the fastest grep. It's not like GNU grep is completely devoid of any performance optimisations.
    - rcxdude 6y ago
      It's a bit out of date (that email predates ripgrep). ripgrep goes even further in terms of skipping work, as well as focusing on parallelism.
    - burntsushi 6y ago
      Look at the dates on the material you're reading. That should resolve most of your confusion. Also, to be clear, in a strict apples to apples comparison, GNU grep and ripgrep will tend to have comparable performance. ripgrep does edge it out a bit in many cases, but it isn't always earth shattering. However, if you don't limit yourself to apples-to-apples comparisons and instead look at the user experience, then yes, ripgrep will likely be "a lot" faster. Primarily because of automatic parallelism and its "smart" filtering.
twic 6y ago
> uutils is an attempt at writing universal (as in cross-platform) CLI utilities in Rust. This repository is intended to aggregate GNU coreutils rewrites. > Why? > Many GNU, Linux and other utilities are useful, and obviously some effort has been spent in the past to port them to Windows. However, those projects are either old and abandoned, are hosted on CVS (which makes it more difficult for new contributors to contribute to them), are written in platform-specific C, or suffer from other issues. > Rust provides a good, platform-agnostic way of writing systems utilities that are easy to compile anywhere, and this is as good a way as any to try and learn it. [1] https://github.com/uutils/coreutils/ https://github.com/uutils/coreutils/
- justinclift 6y ago
  Also note the MIT License, which could mean it gets picked up by places unhappy with GNU in general. Maybe future macOS?
- steerablesafe 6y ago
  > Rust provides a good, platform-agnostic way of writing systems utilities that are easy to compile anywhere, and this is as good a way as any to try and learn it. For some value of "anywhere".
sylvestre 6y ago
Significantly bigger than the C version. Rust: 73M source: https://packages.debian.org/experimental/rust-coreutils https://packages.debian.org/experimental/rust-coreutils C: 17M https://packages.debian.org/unstable/coreutils https://packages.debian.org/unstable/coreutils Note that no size optimization that been done and there is a lot ways to do it in Rust https://github.com/johnthagen/min-sized-rust https://github.com/johnthagen/min-sized-rust In term of performance, well, it depends on the binary. For example for cp, most of the time is spending copying the files. For the factor command, some work happened to make it faster https://github.com/uutils/coreutils/commits/master/src/uu/factor https://github.com/uutils/coreutils/commits/master/src/uu/fa... (getting closer). Using Rust also opens some great capabilities like parallelism. But this makes sense only for complex commands. For example, doing a parallel "df" isn't super interesting (I tried it was too expensive just to start threads to do it). Anyway, performances should be a focus but only after correctness is implemented.
- arp242 6y ago
  You can use it as a single binary that's linked to the actual tool name, like busybox. That binary is just 9.6M (0.0.4 from their GitHub, Linux/amd64). The Debian version ships with separate binaries though (109 of them), and is much larger. There is probably a lot of duplication as many tools use the same Rust libraries for flag parsing and whatnot which are compiled in the binaries. This is why those 73M compresses down to just 8M.
  - alerighi 6y ago
    First, symlinks are not really portable, for example they don't work on Windows, or in ancient UNIX systems. Also, that way every command executes with the same privileges because it's the same binary. And at least in Linux capabilities are bounded to a binary. So for utility that necessitate of special capabilities (e.g. ping, passwd, sudo, etc) this approach will not work. Yes, it works on busybox since everything is owned by root on the typical system so nobody cares about privileges. Finally, you need to load in memory every time you run a single binary all the thing needed by all the other binaries, even for the `true` binary. To me is stupid. The solution exists, and exists since ever, and it's encapsulating common functionalities in shared libraries, and have very small binaries linked to only the libraries they use. Unfortunately Rust choose not to support them for stupid reasons like problems that were resolved decades ago.
felixr 6y ago
From looking at the implementation of yes [0], I would say it follows closely the GNU coreutils implementation [1]. It allocates and re-uses a buffer to speed things up [2] and does not implement the old simple version [3] (or the even simpler OpenBSD [4]) [0]: https://github.com/uutils/coreutils/blob/master/src/uu/yes/src/yes.rs https://github.com/uutils/coreutils/blob/master/src/uu/yes/s... [1]: https://github.com/coreutils/coreutils/blob/master/src/yes.c https://github.com/coreutils/coreutils/blob/master/src/yes.c [2]: http://github.com//coreutils/coreutils/commit/35217221c211f3116f374f305654462195aa634a http://github.com//coreutils/coreutils/commit/35217221c211f3... [3]: https://github.com/coreutils/coreutils/blob/ccbd1d7dc5189f4637468a8136f672e60ee0e531/src/yes.c https://github.com/coreutils/coreutils/blob/ccbd1d7dc5189f46... [4]: https://github.com/openbsd/src/blob/master/usr.bin/yes/yes.c https://github.com/openbsd/src/blob/master/usr.bin/yes/yes.c
Freaky 6y ago
Having written a Rust wc implementation a few years ago (https://github.com/Freaky/cw https://github.com/Freaky/cw), I had a look at theirs. It's pretty naive - a simple linewise read_until loop, a conditional to avoid word splitting and such if it's not needed, and for some reason it collects results into an array and prints when it's done rather than printing as it goes. It doesn't support --files0-from like GNU wc, so isn't a drop-in replacement from that perspective. It also has the sadly common Rust trope of only supporting filenames that are valid UTF-8. It doesn't seem overly slow considering its simplicity - usually trading blows with GNU and BSD wc. Perhaps the most glaring omission is the lack of a fast path for -c, which should reduce to a stat() call. Also unfortunate not to use the excellent bytecount crate to provide a very fast -l/m path. The read_until loop also makes its memory use unpredictable compared with other wc's. If you run it on /dev/zero it will try to eat your computer.
- tavianator 6y ago
  GNU wc doesn't seem to have that -c fast path. I can tell because files in /proc report 0 for st_size even though they're not empty: $ wc -c /proc/self/cmdline 25 /proc/self/cmdline $ stat -c '%s' /proc/self/cmdline 0
  - Freaky 6y ago
    It certainly has a fast path, it just mixes it up with lseek() for filesystems like /proc: https://github.com/coreutils/coreutils/blob/9de1d153f82243aeaaf19b0e5da2345f9b8652e3/src/wc.c#L262-L294 https://github.com/coreutils/coreutils/blob/9de1d153f82243ae... FreeBSD just uses fstat: https://github.com/freebsd/freebsd-src/blob/e4b8deb222278b2a12c9c67021b406625f5be301/usr.bin/wc/wc.c#L243-L261 https://github.com/freebsd/freebsd-src/blob/e4b8deb222278b2a...

dastx 6y ago

I'd love to know too. It seems like it's nearly at feature parity but if performance is an issue, it's not really a replacement.

Blikkentrekker 6y ago
GNU coreutils are quite fast by the way. https://lists.freebsd.org/pipermail/freebsd-current/2010-August/019310.html https://lists.freebsd.org/pipermail/freebsd-current/2010-Aug... This is an interesting article that outlines why GNU grep outperforms almost any other grep quite significantly, at the cost of more memory.
- codethief 6y ago
  That statement surprises me because my impression had always been that GNU grep is rather slow compared to other (more modern) tools, see e.g. the benchmarks on https://github.com/BurntSushi/ripgrep https://github.com/BurntSushi/ripgrep and https://blog.burntsushi.net/ripgrep/ https://blog.burntsushi.net/ripgrep/ .
  - duckerude 6y ago
    GNU grep is so fast that it set the bar for later implementations. In your second link ripgrep is even marketed as having "the raw performance of GNU grep" (though it managed to exceed it).
  - arp242 6y ago
    When that was written in 2010 it probably was the fastest grep. It's not like GNU grep is completely devoid of any performance optimisations.
  - rcxdude 6y ago
    It's a bit out of date (that email predates ripgrep). ripgrep goes even further in terms of skipping work, as well as focusing on parallelism.
  - burntsushi 6y ago
    Look at the dates on the material you're reading. That should resolve most of your confusion. Also, to be clear, in a strict apples to apples comparison, GNU grep and ripgrep will tend to have comparable performance. ripgrep does edge it out a bit in many cases, but it isn't always earth shattering. However, if you don't limit yourself to apples-to-apples comparisons and instead look at the user experience, then yes, ripgrep will likely be "a lot" faster. Primarily because of automatic parallelism and its "smart" filtering.

twic 6y ago

> uutils is an attempt at writing universal (as in cross-platform) CLI utilities in Rust. This repository is intended to aggregate GNU coreutils rewrites. > Why? > Many GNU, Linux and other utilities are useful, and obviously some effort has been spent in the past to port them to Windows. However, those projects are either old and abandoned, are hosted on CVS (which makes it more difficult for new contributors to contribute to them), are written in platform-specific C, or suffer from other issues. > Rust provides a good, platform-agnostic way of writing systems utilities that are easy to compile anywhere, and this is as good a way as any to try and learn it. [1] https://github.com/uutils/coreutils/ https://github.com/uutils/coreutils/

justinclift 6y ago
Also note the MIT License, which could mean it gets picked up by places unhappy with GNU in general. Maybe future macOS?
steerablesafe 6y ago
> Rust provides a good, platform-agnostic way of writing systems utilities that are easy to compile anywhere, and this is as good a way as any to try and learn it. For some value of "anywhere".

sylvestre 6y ago

Significantly bigger than the C version. Rust: 73M source: https://packages.debian.org/experimental/rust-coreutils https://packages.debian.org/experimental/rust-coreutils C: 17M https://packages.debian.org/unstable/coreutils https://packages.debian.org/unstable/coreutils Note that no size optimization that been done and there is a lot ways to do it in Rust https://github.com/johnthagen/min-sized-rust https://github.com/johnthagen/min-sized-rust In term of performance, well, it depends on the binary. For example for cp, most of the time is spending copying the files. For the factor command, some work happened to make it faster https://github.com/uutils/coreutils/commits/master/src/uu/factor https://github.com/uutils/coreutils/commits/master/src/uu/fa... (getting closer). Using Rust also opens some great capabilities like parallelism. But this makes sense only for complex commands. For example, doing a parallel "df" isn't super interesting (I tried it was too expensive just to start threads to do it). Anyway, performances should be a focus but only after correctness is implemented.

arp242 6y ago
You can use it as a single binary that's linked to the actual tool name, like busybox. That binary is just 9.6M (0.0.4 from their GitHub, Linux/amd64). The Debian version ships with separate binaries though (109 of them), and is much larger. There is probably a lot of duplication as many tools use the same Rust libraries for flag parsing and whatnot which are compiled in the binaries. This is why those 73M compresses down to just 8M.
- alerighi 6y ago
  First, symlinks are not really portable, for example they don't work on Windows, or in ancient UNIX systems. Also, that way every command executes with the same privileges because it's the same binary. And at least in Linux capabilities are bounded to a binary. So for utility that necessitate of special capabilities (e.g. ping, passwd, sudo, etc) this approach will not work. Yes, it works on busybox since everything is owned by root on the typical system so nobody cares about privileges. Finally, you need to load in memory every time you run a single binary all the thing needed by all the other binaries, even for the `true` binary. To me is stupid. The solution exists, and exists since ever, and it's encapsulating common functionalities in shared libraries, and have very small binaries linked to only the libraries they use. Unfortunately Rust choose not to support them for stupid reasons like problems that were resolved decades ago.

felixr 6y ago

From looking at the implementation of yes [0], I would say it follows closely the GNU coreutils implementation [1]. It allocates and re-uses a buffer to speed things up [2] and does not implement the old simple version [3] (or the even simpler OpenBSD [4]) [0]: https://github.com/uutils/coreutils/blob/master/src/uu/yes/src/yes.rs https://github.com/uutils/coreutils/blob/master/src/uu/yes/s... [1]: https://github.com/coreutils/coreutils/blob/master/src/yes.c https://github.com/coreutils/coreutils/blob/master/src/yes.c [2]: http://github.com//coreutils/coreutils/commit/35217221c211f3116f374f305654462195aa634a http://github.com//coreutils/coreutils/commit/35217221c211f3... [3]: https://github.com/coreutils/coreutils/blob/ccbd1d7dc5189f4637468a8136f672e60ee0e531/src/yes.c https://github.com/coreutils/coreutils/blob/ccbd1d7dc5189f46... [4]: https://github.com/openbsd/src/blob/master/usr.bin/yes/yes.c https://github.com/openbsd/src/blob/master/usr.bin/yes/yes.c

Freaky 6y ago

Having written a Rust wc implementation a few years ago (https://github.com/Freaky/cw https://github.com/Freaky/cw), I had a look at theirs. It's pretty naive - a simple linewise read_until loop, a conditional to avoid word splitting and such if it's not needed, and for some reason it collects results into an array and prints when it's done rather than printing as it goes. It doesn't support --files0-from like GNU wc, so isn't a drop-in replacement from that perspective. It also has the sadly common Rust trope of only supporting filenames that are valid UTF-8. It doesn't seem overly slow considering its simplicity - usually trading blows with GNU and BSD wc. Perhaps the most glaring omission is the lack of a fast path for -c, which should reduce to a stat() call. Also unfortunate not to use the excellent bytecount crate to provide a very fast -l/m path. The read_until loop also makes its memory use unpredictable compared with other wc's. If you run it on /dev/zero it will try to eat your computer.

tavianator 6y ago
GNU wc doesn't seem to have that -c fast path. I can tell because files in /proc report 0 for st_size even though they're not empty: $ wc -c /proc/self/cmdline 25 /proc/self/cmdline $ stat -c '%s' /proc/self/cmdline 0
- Freaky 6y ago
  It certainly has a fast path, it just mixes it up with lseek() for filesystems like /proc: https://github.com/coreutils/coreutils/blob/9de1d153f82243aeaaf19b0e5da2345f9b8652e3/src/wc.c#L262-L294 https://github.com/coreutils/coreutils/blob/9de1d153f82243ae... FreeBSD just uses fstat: https://github.com/freebsd/freebsd-src/blob/e4b8deb222278b2a12c9c67021b406625f5be301/usr.bin/wc/wc.c#L243-L261 https://github.com/freebsd/freebsd-src/blob/e4b8deb222278b2a...