7 ms·

I looked at the wc implementation, there's no way it's compatible with the GNU tools, how it handles wide characters, which is heavily ties to the idiosyncratic

by Maro 2y ago

I looked at the wc implementation, there's no way it's compatible with the GNU tools, how it handles wide characters, which is heavily ties to the idiosyncratic C implementation. I know because I once wrote a wc implementation in modern C++ for my own education, and that's where most of my time and code complexity went. Also, the C versions are devilishly fast, with all flags, with optimized code branches. https://bytepawn.com/tag/wc.html https://bytepawn.com/tag/wc.html

galangalalgol 2y ago
Poking at the rust coreutils project it seems like they all faster than the c versions. The issue is the decades of features that have accumulated that someone somewhere uses but seem safe to disregard because so few do.
- Maro 2y ago
  There are millions of lines of shell scripts relying on those flags.
  - chongli 2y ago
    I would place the blame for that on the GNU coreutils project. At best, it’s feature creep. At worst, it’s embrace and extend on the POSIX standard.
    - erik_seaberg 2y ago
      POSIX doesn't forbid new flags or options. It's up to the author to read the spec and test his portability, or else willingly rely on certain distros. Some GNU tools have had strict modes as a courtesy (they used to jokingly call it POSIX_ME_HARDER).
  - prmoustache 2y ago
    And those scripts should check they are indeed calling the gnu coreutils version before they proceed.
    - tankenmate 2y ago
      in the real world this would not work; it's putting the cart in front of the horse.
  - wodenokoto 2y ago
    Would they work on BSD/MacOS? It’s not just gnu vs rust out there.
- Arcuru 2y ago
  They're faster? They must have been working on performance then, maybe I should take another look at how that project has been maturing. When I last looked 'dd' was significantly slower, though I did make it a bit closer a while back - https://jackson.dev/post/rust-coreutils-dd/ https://jackson.dev/post/rust-coreutils-dd/ A lot of the Rust coreutils were written by people learning the language, and the code is often _more_ complicated than the corresponding code for the Unix utils. They don't seem to get enough experienced programmers fixing them up or usage for me to actually recommend anybody use them, but maybe that's changing.
  - mustache_kimono 2y ago
    > When I last looked 'dd' was significantly slower, though I did make it a bit closer a while back - https://jackson.dev/post/rust-coreutils-dd/ https://jackson.dev/post/rust-coreutils-dd/ I read your blog. Maybe you should take a look at some of the other utils. I worked on sort and ls and both have long been faster than their GNU equivalents. > They don't seem to get enough experienced programmers fixing them up or usage for me to actually recommend anybody use them, but maybe that's changing. The issue is obviously compatibility and the uutils won't be broadly "usable" or stable until is hits 100% PASS/0% FAIL on the GNU test coverage, although the numbers are getting better and better. See: https://raw.githubusercontent.com/uutils/coreutils-tracking/refs/heads/main/gnu-results.png https://raw.githubusercontent.com/uutils/coreutils-tracking/... > A lot of the Rust coreutils were written by people learning the language, I really, really hate this way of framing the project -- which implicitly sounds like GNU is full of pros. Yes, lots of code was written by people just learning Rust and systems programming. Once it reaches parity on GNU test coverage, which it is rapidly approaching, GNU coreutils would wish it had this kind of enthusiasm behind it. > and the code is often _more_ complicated than the corresponding code for the Unix utils. I think it really depends. Much of the code is much simpler. For instance -- I think it's much easier to generically do hashing in Rust, and it shows.
    - oguz-ismail 2y ago
      Yeah just two more weeks
- diath 2y ago
  ...and that's precisely why they are faster, if you can throw out decades of compatibility flags you can avoid expensive branching and such to make them faster. That's why such projects should be called alternatives, and not replacements, it's not a replacement if it cannot be seamlessly replaced and decades worth of scripts can continue working.
  - 7bit 2y ago
    When is it the time to talk about alternatives replacing the current apps in favor of faster apps? 10 years or rare usage? Twenty? At some point the edge cases should be covered by the scripts not by the app, only to make it slower for everyone. I get what you're saying, but I don't care how it's called. Some things must die. Eg. Python 3 and the depreciation of was a very controversial step, but ultimately the right choice at the right time.
    - kanbankaren 2y ago
      > When is it the time to talk about alternatives When all standards bodies and governments decide that POSIX is a hindrance which might take a few decades. And that is if they decide.
      - galangalalgol 2y ago
        The FBI just said everyone needs a roadmap by 2026 to eliminate c/c++ from critical infrastructure.
        cozzyd 2y ago
        Perhaps by 2126, Linux will no longer have C in it, but that's probably hugely optimistic.
        niobe 2y ago
        We'll all have uploaded to the Metaverse by then, running purely on beams of light and upvotes.
- johnisgood 2y ago
  Faster? I do not know, what I do know is that there has been some really crazy vulnerabilities in one (or some) of these utilities (logic errors).
- fire_lake 2y ago
  This can be solved By Nix - stay on an old version if you need it but the rest of the world can move forward
anardil 2y ago
You're right! Data.ByteString.Lazy is Word8 under the covers, so wide characters are truncated. tr takes a similar short cut. Swapping to Data.Text would fix that. Where simplicity conflicted with compatibility, I've chosen the former so far. Targeting the BSD options and behavior is another example of that. The primary goal is to feel out the data flow for each utility, rather than dig into all the edges.
johnisgood 2y ago
Speaking of fast: https://redman.xyz/git/turbowc/tree/turbowc.c https://redman.xyz/git/turbowc/tree/turbowc.c seems to be really fast (it is not a replacement for wc, however).
- stabbles 2y ago
  AVX2 is pretty much ubiquitous, the implementation above is only SSE2. I wrote a blog post with the main AVX2 tricks [1] which includes how to deal with repeated white space when counting words [1] https://stoppels.ch/2022/11/30/io-is-no-longer-the-bottleneck-part-2.html https://stoppels.ch/2022/11/30/io-is-no-longer-the-bottlenec...
- codedokode 2y ago
  It seems to use hand-written code for SIMD. Cannot one use normal C code using loops so that compiler optimizes it into SIMD? I know that gcc can convert some loops into SIMD code.
  - adgjlsfhk1 2y ago
    autovectorizers are surprisingly bad
MBCook 2y ago
The readme mentions feature parity with the BSD coreutils, so I’m assuming that’s the target. Not GNU.
deleted 2y ago
[deleted]

by Maro 2y ago

galangalalgol 2y ago

Poking at the rust coreutils project it seems like they all faster than the c versions. The issue is the decades of features that have accumulated that someone somewhere uses but seem safe to disregard because so few do.

Maro 2y ago
There are millions of lines of shell scripts relying on those flags.
- chongli 2y ago
  I would place the blame for that on the GNU coreutils project. At best, it’s feature creep. At worst, it’s embrace and extend on the POSIX standard.
  - erik_seaberg 2y ago
    POSIX doesn't forbid new flags or options. It's up to the author to read the spec and test his portability, or else willingly rely on certain distros. Some GNU tools have had strict modes as a courtesy (they used to jokingly call it POSIX_ME_HARDER).
- prmoustache 2y ago
  And those scripts should check they are indeed calling the gnu coreutils version before they proceed.
  - tankenmate 2y ago
    in the real world this would not work; it's putting the cart in front of the horse.
- wodenokoto 2y ago
  Would they work on BSD/MacOS? It’s not just gnu vs rust out there.
Arcuru 2y ago
They're faster? They must have been working on performance then, maybe I should take another look at how that project has been maturing. When I last looked 'dd' was significantly slower, though I did make it a bit closer a while back - https://jackson.dev/post/rust-coreutils-dd/ https://jackson.dev/post/rust-coreutils-dd/ A lot of the Rust coreutils were written by people learning the language, and the code is often _more_ complicated than the corresponding code for the Unix utils. They don't seem to get enough experienced programmers fixing them up or usage for me to actually recommend anybody use them, but maybe that's changing.
- mustache_kimono 2y ago
  > When I last looked 'dd' was significantly slower, though I did make it a bit closer a while back - https://jackson.dev/post/rust-coreutils-dd/ https://jackson.dev/post/rust-coreutils-dd/ I read your blog. Maybe you should take a look at some of the other utils. I worked on sort and ls and both have long been faster than their GNU equivalents. > They don't seem to get enough experienced programmers fixing them up or usage for me to actually recommend anybody use them, but maybe that's changing. The issue is obviously compatibility and the uutils won't be broadly "usable" or stable until is hits 100% PASS/0% FAIL on the GNU test coverage, although the numbers are getting better and better. See: https://raw.githubusercontent.com/uutils/coreutils-tracking/refs/heads/main/gnu-results.png https://raw.githubusercontent.com/uutils/coreutils-tracking/... > A lot of the Rust coreutils were written by people learning the language, I really, really hate this way of framing the project -- which implicitly sounds like GNU is full of pros. Yes, lots of code was written by people just learning Rust and systems programming. Once it reaches parity on GNU test coverage, which it is rapidly approaching, GNU coreutils would wish it had this kind of enthusiasm behind it. > and the code is often _more_ complicated than the corresponding code for the Unix utils. I think it really depends. Much of the code is much simpler. For instance -- I think it's much easier to generically do hashing in Rust, and it shows.
  - oguz-ismail 2y ago
    Yeah just two more weeks
diath 2y ago
...and that's precisely why they are faster, if you can throw out decades of compatibility flags you can avoid expensive branching and such to make them faster. That's why such projects should be called alternatives, and not replacements, it's not a replacement if it cannot be seamlessly replaced and decades worth of scripts can continue working.
- 7bit 2y ago
  When is it the time to talk about alternatives replacing the current apps in favor of faster apps? 10 years or rare usage? Twenty? At some point the edge cases should be covered by the scripts not by the app, only to make it slower for everyone. I get what you're saying, but I don't care how it's called. Some things must die. Eg. Python 3 and the depreciation of was a very controversial step, but ultimately the right choice at the right time.
  - kanbankaren 2y ago
    > When is it the time to talk about alternatives When all standards bodies and governments decide that POSIX is a hindrance which might take a few decades. And that is if they decide.
    - galangalalgol 2y ago
      The FBI just said everyone needs a roadmap by 2026 to eliminate c/c++ from critical infrastructure.
      - cozzyd 2y ago
        Perhaps by 2126, Linux will no longer have C in it, but that's probably hugely optimistic.
        niobe 2y ago
        We'll all have uploaded to the Metaverse by then, running purely on beams of light and upvotes.
johnisgood 2y ago
Faster? I do not know, what I do know is that there has been some really crazy vulnerabilities in one (or some) of these utilities (logic errors).
fire_lake 2y ago
This can be solved By Nix - stay on an old version if you need it but the rest of the world can move forward

anardil 2y ago

You're right! Data.ByteString.Lazy is Word8 under the covers, so wide characters are truncated. tr takes a similar short cut. Swapping to Data.Text would fix that. Where simplicity conflicted with compatibility, I've chosen the former so far. Targeting the BSD options and behavior is another example of that. The primary goal is to feel out the data flow for each utility, rather than dig into all the edges.

johnisgood 2y ago

Speaking of fast: https://redman.xyz/git/turbowc/tree/turbowc.c https://redman.xyz/git/turbowc/tree/turbowc.c seems to be really fast (it is not a replacement for wc, however).

stabbles 2y ago
AVX2 is pretty much ubiquitous, the implementation above is only SSE2. I wrote a blog post with the main AVX2 tricks [1] which includes how to deal with repeated white space when counting words [1] https://stoppels.ch/2022/11/30/io-is-no-longer-the-bottleneck-part-2.html https://stoppels.ch/2022/11/30/io-is-no-longer-the-bottlenec...
codedokode 2y ago
It seems to use hand-written code for SIMD. Cannot one use normal C code using loops so that compiler optimizes it into SIMD? I know that gcc can convert some loops into SIMD code.
- adgjlsfhk1 2y ago
  autovectorizers are surprisingly bad

MBCook 2y ago

The readme mentions feature parity with the BSD coreutils, so I’m assuming that’s the target. Not GNU.

deleted 2y ago

[deleted]