4 ms·

Check the list, performance matters a great deal for many of these utilities, and the GNU project version is often pretty well optimized (often best in class of

Check the list, performance matters a great deal for many of these utilities, and the GNU project version is often pretty well optimized (often best in class of POSIX compliant impls that ship with an OS/distribution). I do not want a "performance will be looked at later" version of m4, awk, grep, cp, find, diff, sort, uniq, etc., right now in my personal or dev environments. I can understand using a not-yet-optimized memory safe language version to compete with OpenBSD though, but not for me right now.

littlestymaar 2y ago
You are already using “performance will be looked at later” versions of grep and find as they are much slower than alternative implementations (ripgrep and fd are way faster) and these are the one for which perf is the most sensitive in your list… The fact that performance don't really matter for these tools in the general case is the reason while most people still stick with those slow POSIX-compliant tools when much faster alternatives exist. (POSIX-compliance only matters when using it in existing scripts, but not at all when using it in the command line or when writing new scripts)
- kazinator 2y ago
  POSIX compliance totally matters for interactive use, too, if POSIX is what you know. That's like saying English only matters if you're revising some existing paper written in English, but if you're writing something new or conversing with somebody face to face, then English doesn't matter at all. Sure unless English is your most proficient or perhaps only language. The GNU project didn't have to implement Unix compatible utilities. They were starting with a blank slate and could have implemented anything. Why did they choose the compatible route? Because it matters. People were able to replace proprietary Unix programs with GNU programs without disrupting their workflows or having to learn anything (unless they were curious about GNU extensions).
  - littlestymaar 2y ago
    > POSIX compliance totally matters for interactive use, too, if POSIX is what you know. Nobody knows the entire spec, implementing it only matters for compatibility with existing scripts that can use every features. For interactive use, keeping the same interface for the most common use-case is enough. Switching to non-POSIX tools is well worth the moderate effort if performance matters, but that's my point: performance doesn't really matter for these tools. Anything that's not orders of magnitude slower is good enough.
  - burntsushi 2y ago
    > POSIX compliance totally matters for interactive use, too, if POSIX is what you know. When you use grep on the CLI, do you specifically limit yourself to options specified by POSIX? How do you even know if you do or not? The `-r/--recursive` flag, for example, isn't even part of POSIX grep. Do you ever use it? What about `-w/--word-regexp`? Not in POSIX either. `-P/--pcre2` isn't. Neither is `-a/--text`. Maybe what grep is, is actually more than what POSIX says it is.
    - dundarious 2y ago
      You're absolutely right on the strict point of POSIX compliance. I think there's a point to be made though about just stability of the functional/CLI interface, aside from what's technically in the POSIX standards. ripgrep is pretty damn good on interface stability AFAIK, but the other tools that similarly don't try to position themselves as POSIX+/POSIXish compatible, are a mixed bag. It's nice to just have the stable tools you know, and have them practically everywhere (if you include WSL/msys2/Git for windows, etc.). And similarly you pretty much know what POSIX+ features you have on a GNU system (most Linuxes), or a BSD one (macOS, OpenBSD, etc.). I like having an excellent set of system tools from GNU/BSD, etc., then I can install/use the SotA stuff -- and I'll still end up using both sets of tools all the time, even though one set is not absolute best in class, because I don't have to worry about how to use sota-tool 1.2 on one system vs sota-tool 2.1 on another, when there may be important interface changes. And to provide the full context going back to my first comment: I won't consider using "performance will be looked at later" tools whose purported benefit is just the use of Rust, pretty much at all.
      - burntsushi 2y ago
        For me, I just find it very dubious when folks point to POSIX as a point of stability. The reality is that POSIX is so barebones in a number of respects, that most "POSIX tooling" actually has more features than what POSIX specifies. And sometimes the behavior of those features differs across implementations, with sed's `-i` flag being an infamous example. As I pointed out, the main problem is that, other than POSIX experts who have memorized the spec, most folks have no idea when they're crossing the line between "POSIX specified" and "extra feature." I agree that stability is really the main important bit here. That is, is that script I wrote using your tool 5 years still going to work the same way today? And that is exactly my point. Because this discussion is totally different if we focus on what actually matters (stability) instead of some flawed idealistic approximation of it (POSIX) that only actually exists in theory. Because even the most spartan of tools, like busybox, implement features beyond what POSIX requires. And people use those not because they are in POSIX, but because they are in practice, in the real world, stable. In other words, folks harping on POSIX as an end have confused it with what it really is: a means to an end.
- burntsushi 2y ago
  > You are already using “performance will be looked at later” versions of grep Are you referring to GNU grep? Because if so, this is wildly untrue. GNU grep has been very significantly optimized. There are huge piles of code, including an entire regex engine, in GNU grep specifically for running searches more quickly. Maybe something like busybox's or BSD's grep would satisfy "performance will be looked at later," but certainly not GNU grep. I can't tell which one you're referring to, but a generic "you are already using" seems to at least include GNU grep.
  - littlestymaar 2y ago
    What I mean is not really that grep hasn't been optimized, but that it kind of looks like it is. It's now far from the state of the art from most perspective (in part because of you). And because there exists alternative that in most workloads are much faster than grep without really endangering grep's prominence, it shows that performance isn't really a criteria in its use nowadays (computers are fast). My point as a whole is that as long as the implementation isn't catastrophically slow, then any port is good even if it's slower than GNU's because workloads that both care about max performance, and must be 100% compatible with the original, is small enough.
    - burntsushi 2y ago
      I didn't really want to engage with your broader point. I do disagree with it, but only in a measure of degree, and that's hard to debate because it's perspective and opinion oriented. If performance wasn't as important as you seem to imply, it's very unlikely that ripgrep would have gotten popular given that performance (and filtering) are its two more critical features. That is, that's what I hear from users the most: the like ripgrep because it's fast or because it filters or because of both. But on the smaller point, it is just factually untrue that GNU grep falls into the "performance will be looked at later" bucket. That was why I responded. This post is a classic about why GNU grep is fast, demonstrating that GNU grep definitely doesn't look like it wasn't optimized: https://lists.freebsd.org/pipermail/freebsd-current/2010-August/019310.html https://lists.freebsd.org/pipermail/freebsd-current/2010-Aug... And then before that, there's this: https://ridiculousfish.com/blog/posts/old-age-and-treachery.html https://ridiculousfish.com/blog/posts/old-age-and-treachery....

littlestymaar 2y ago

You are already using “performance will be looked at later” versions of grep and find as they are much slower than alternative implementations (ripgrep and fd are way faster) and these are the one for which perf is the most sensitive in your list… The fact that performance don't really matter for these tools in the general case is the reason while most people still stick with those slow POSIX-compliant tools when much faster alternatives exist. (POSIX-compliance only matters when using it in existing scripts, but not at all when using it in the command line or when writing new scripts)

kazinator 2y ago
POSIX compliance totally matters for interactive use, too, if POSIX is what you know. That's like saying English only matters if you're revising some existing paper written in English, but if you're writing something new or conversing with somebody face to face, then English doesn't matter at all. Sure unless English is your most proficient or perhaps only language. The GNU project didn't have to implement Unix compatible utilities. They were starting with a blank slate and could have implemented anything. Why did they choose the compatible route? Because it matters. People were able to replace proprietary Unix programs with GNU programs without disrupting their workflows or having to learn anything (unless they were curious about GNU extensions).
- littlestymaar 2y ago
  > POSIX compliance totally matters for interactive use, too, if POSIX is what you know. Nobody knows the entire spec, implementing it only matters for compatibility with existing scripts that can use every features. For interactive use, keeping the same interface for the most common use-case is enough. Switching to non-POSIX tools is well worth the moderate effort if performance matters, but that's my point: performance doesn't really matter for these tools. Anything that's not orders of magnitude slower is good enough.
- burntsushi 2y ago
  > POSIX compliance totally matters for interactive use, too, if POSIX is what you know. When you use grep on the CLI, do you specifically limit yourself to options specified by POSIX? How do you even know if you do or not? The `-r/--recursive` flag, for example, isn't even part of POSIX grep. Do you ever use it? What about `-w/--word-regexp`? Not in POSIX either. `-P/--pcre2` isn't. Neither is `-a/--text`. Maybe what grep is, is actually more than what POSIX says it is.
  - dundarious 2y ago
    You're absolutely right on the strict point of POSIX compliance. I think there's a point to be made though about just stability of the functional/CLI interface, aside from what's technically in the POSIX standards. ripgrep is pretty damn good on interface stability AFAIK, but the other tools that similarly don't try to position themselves as POSIX+/POSIXish compatible, are a mixed bag. It's nice to just have the stable tools you know, and have them practically everywhere (if you include WSL/msys2/Git for windows, etc.). And similarly you pretty much know what POSIX+ features you have on a GNU system (most Linuxes), or a BSD one (macOS, OpenBSD, etc.). I like having an excellent set of system tools from GNU/BSD, etc., then I can install/use the SotA stuff -- and I'll still end up using both sets of tools all the time, even though one set is not absolute best in class, because I don't have to worry about how to use sota-tool 1.2 on one system vs sota-tool 2.1 on another, when there may be important interface changes. And to provide the full context going back to my first comment: I won't consider using "performance will be looked at later" tools whose purported benefit is just the use of Rust, pretty much at all.
    - burntsushi 2y ago
      For me, I just find it very dubious when folks point to POSIX as a point of stability. The reality is that POSIX is so barebones in a number of respects, that most "POSIX tooling" actually has more features than what POSIX specifies. And sometimes the behavior of those features differs across implementations, with sed's `-i` flag being an infamous example. As I pointed out, the main problem is that, other than POSIX experts who have memorized the spec, most folks have no idea when they're crossing the line between "POSIX specified" and "extra feature." I agree that stability is really the main important bit here. That is, is that script I wrote using your tool 5 years still going to work the same way today? And that is exactly my point. Because this discussion is totally different if we focus on what actually matters (stability) instead of some flawed idealistic approximation of it (POSIX) that only actually exists in theory. Because even the most spartan of tools, like busybox, implement features beyond what POSIX requires. And people use those not because they are in POSIX, but because they are in practice, in the real world, stable. In other words, folks harping on POSIX as an end have confused it with what it really is: a means to an end.
burntsushi 2y ago
> You are already using “performance will be looked at later” versions of grep Are you referring to GNU grep? Because if so, this is wildly untrue. GNU grep has been very significantly optimized. There are huge piles of code, including an entire regex engine, in GNU grep specifically for running searches more quickly. Maybe something like busybox's or BSD's grep would satisfy "performance will be looked at later," but certainly not GNU grep. I can't tell which one you're referring to, but a generic "you are already using" seems to at least include GNU grep.
- littlestymaar 2y ago
  What I mean is not really that grep hasn't been optimized, but that it kind of looks like it is. It's now far from the state of the art from most perspective (in part because of you). And because there exists alternative that in most workloads are much faster than grep without really endangering grep's prominence, it shows that performance isn't really a criteria in its use nowadays (computers are fast). My point as a whole is that as long as the implementation isn't catastrophically slow, then any port is good even if it's slower than GNU's because workloads that both care about max performance, and must be 100% compatible with the original, is small enough.
  - burntsushi 2y ago
    I didn't really want to engage with your broader point. I do disagree with it, but only in a measure of degree, and that's hard to debate because it's perspective and opinion oriented. If performance wasn't as important as you seem to imply, it's very unlikely that ripgrep would have gotten popular given that performance (and filtering) are its two more critical features. That is, that's what I hear from users the most: the like ripgrep because it's fast or because it filters or because of both. But on the smaller point, it is just factually untrue that GNU grep falls into the "performance will be looked at later" bucket. That was why I responded. This post is a classic about why GNU grep is fast, demonstrating that GNU grep definitely doesn't look like it wasn't optimized: https://lists.freebsd.org/pipermail/freebsd-current/2010-August/019310.html https://lists.freebsd.org/pipermail/freebsd-current/2010-Aug... And then before that, there's this: https://ridiculousfish.com/blog/posts/old-age-and-treachery.html https://ridiculousfish.com/blog/posts/old-age-and-treachery....