5 ms·

Would definitely be interesting, but from a cursory look at the repository, it doesn't look like squeezing the last percentage points of performance has been a

Would definitely be interesting, but from a cursory look at the repository, it doesn't look like squeezing the last percentage points of performance has been a priority yet. Things that stand out: - The `awk` implementation uses the Pest parser generator (https://pest.rs/ https://pest.rs/), which is known to not generate the fastest possible parsers, but is great for getting up and running. - They are using the `clap` crate for argument parsing, which is also known to not be the fastest, but again is very user friendly (for example, it does Unicode linebreaks in the output of `--help`). It's marginal, but for a tiny utility being invoked many times from a shell script, this can add up. It's very probably "fast enough", and it makes sense to prioritize like this at this point, but people shouldn't use this expecting a performance improvement right now.

littlestymaar 2y ago
Yeah, and I don't think performance matters that much for these utilities (and AFAIK many of the original haven't been particularly optimized for performance anyway).
- dundarious 2y ago
  Check the list, performance matters a great deal for many of these utilities, and the GNU project version is often pretty well optimized (often best in class of POSIX compliant impls that ship with an OS/distribution). I do not want a "performance will be looked at later" version of m4, awk, grep, cp, find, diff, sort, uniq, etc., right now in my personal or dev environments. I can understand using a not-yet-optimized memory safe language version to compete with OpenBSD though, but not for me right now.
  - littlestymaar 2y ago
    You are already using “performance will be looked at later” versions of grep and find as they are much slower than alternative implementations (ripgrep and fd are way faster) and these are the one for which perf is the most sensitive in your list… The fact that performance don't really matter for these tools in the general case is the reason while most people still stick with those slow POSIX-compliant tools when much faster alternatives exist. (POSIX-compliance only matters when using it in existing scripts, but not at all when using it in the command line or when writing new scripts)
    - kazinator 2y ago
      POSIX compliance totally matters for interactive use, too, if POSIX is what you know. That's like saying English only matters if you're revising some existing paper written in English, but if you're writing something new or conversing with somebody face to face, then English doesn't matter at all. Sure unless English is your most proficient or perhaps only language. The GNU project didn't have to implement Unix compatible utilities. They were starting with a blank slate and could have implemented anything. Why did they choose the compatible route? Because it matters. People were able to replace proprietary Unix programs with GNU programs without disrupting their workflows or having to learn anything (unless they were curious about GNU extensions).
      - littlestymaar 2y ago
        > POSIX compliance totally matters for interactive use, too, if POSIX is what you know. Nobody knows the entire spec, implementing it only matters for compatibility with existing scripts that can use every features. For interactive use, keeping the same interface for the most common use-case is enough. Switching to non-POSIX tools is well worth the moderate effort if performance matters, but that's my point: performance doesn't really matter for these tools. Anything that's not orders of magnitude slower is good enough.
      - burntsushi 2y ago
        > POSIX compliance totally matters for interactive use, too, if POSIX is what you know. When you use grep on the CLI, do you specifically limit yourself to options specified by POSIX? How do you even know if you do or not? The `-r/--recursive` flag, for example, isn't even part of POSIX grep. Do you ever use it? What about `-w/--word-regexp`? Not in POSIX either. `-P/--pcre2` isn't. Neither is `-a/--text`. Maybe what grep is, is actually more than what POSIX says it is.
        dundarious 2y ago
        You're absolutely right on the strict point of POSIX compliance. I think there's a point to be made though about just stability of the functional/CLI interface, aside from what's technically in the POSIX standards. ripgrep is pretty damn good on interface stability AFAIK, but the other tools that similarly don't try to position themselves as POSIX+/POSIXish compatible, are a mixed bag. It's nice to just have the stable tools you know, and have them practically everywhere (if you include WSL/msys2/Git for windows, etc.). And similarly you pretty much know what POSIX+ features you have on a GNU system (most Linuxes), or a BSD one (macOS, OpenBSD, etc.). I like having an excellent set of system tools from GNU/BSD, etc., then I can install/use the SotA stuff -- and I'll still end up using both sets of tools all the time, even though one set is not absolute best in class, because I don't have to worry about how to use sota-tool 1.2 on one system vs sota-tool 2.1 on another, when there may be important interface changes. And to provide the full context going back to my first comment: I won't consider using "performance will be looked at later" tools whose purported benefit is just the use of Rust, pretty much at all.
jgarzik 2y ago
I wouldn't assume. The awk implementation likely stands up well for a 1-billion-row challenge, with its thoughtful bytecode-based design. Redditors ran some quick performance tests on parsing, also: https://www.reddit.com/r/rust/comments/1fd7qgl/comment/lmelo6z/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button https://www.reddit.com/r/rust/comments/1fd7qgl/comment/lmelo...
- dundarious 2y ago
  Parsing the awk input? While not an irrelevant concern, that is obviously on the low priority end when considering awk performance in general. My most often used awk program is just '{print $1}', but I use it on enormous files. The performance when operating on the enormous file is the concern wrt performance, not the initial parse of '{print $1}' or of command line arguments. I know you're just directly responding to the concerns of a parent comment though.
  - jgarzik 2y ago
    It is hoped that posixutil's awk's bytecode-based modern design should keep performance high, theoretically higher than ancient C-based awks. Inspired by Ray Gardner's "wak" awk implementation https://github.com/raygard/wak https://github.com/raygard/wak A volunteer benchmarking our awk on a 1-billion line text file would be welcome.
    - dundarious 2y ago
      Yes, that's the most important part to focus on. I have nothing to remark about it, I haven't seen/gathered any numbers on it.

littlestymaar 2y ago

Yeah, and I don't think performance matters that much for these utilities (and AFAIK many of the original haven't been particularly optimized for performance anyway).

dundarious 2y ago
Check the list, performance matters a great deal for many of these utilities, and the GNU project version is often pretty well optimized (often best in class of POSIX compliant impls that ship with an OS/distribution). I do not want a "performance will be looked at later" version of m4, awk, grep, cp, find, diff, sort, uniq, etc., right now in my personal or dev environments. I can understand using a not-yet-optimized memory safe language version to compete with OpenBSD though, but not for me right now.
- littlestymaar 2y ago
  You are already using “performance will be looked at later” versions of grep and find as they are much slower than alternative implementations (ripgrep and fd are way faster) and these are the one for which perf is the most sensitive in your list… The fact that performance don't really matter for these tools in the general case is the reason while most people still stick with those slow POSIX-compliant tools when much faster alternatives exist. (POSIX-compliance only matters when using it in existing scripts, but not at all when using it in the command line or when writing new scripts)
  - kazinator 2y ago
    POSIX compliance totally matters for interactive use, too, if POSIX is what you know. That's like saying English only matters if you're revising some existing paper written in English, but if you're writing something new or conversing with somebody face to face, then English doesn't matter at all. Sure unless English is your most proficient or perhaps only language. The GNU project didn't have to implement Unix compatible utilities. They were starting with a blank slate and could have implemented anything. Why did they choose the compatible route? Because it matters. People were able to replace proprietary Unix programs with GNU programs without disrupting their workflows or having to learn anything (unless they were curious about GNU extensions).
    - littlestymaar 2y ago
      > POSIX compliance totally matters for interactive use, too, if POSIX is what you know. Nobody knows the entire spec, implementing it only matters for compatibility with existing scripts that can use every features. For interactive use, keeping the same interface for the most common use-case is enough. Switching to non-POSIX tools is well worth the moderate effort if performance matters, but that's my point: performance doesn't really matter for these tools. Anything that's not orders of magnitude slower is good enough.
    - burntsushi 2y ago
      > POSIX compliance totally matters for interactive use, too, if POSIX is what you know. When you use grep on the CLI, do you specifically limit yourself to options specified by POSIX? How do you even know if you do or not? The `-r/--recursive` flag, for example, isn't even part of POSIX grep. Do you ever use it? What about `-w/--word-regexp`? Not in POSIX either. `-P/--pcre2` isn't. Neither is `-a/--text`. Maybe what grep is, is actually more than what POSIX says it is.
      - dundarious 2y ago
        You're absolutely right on the strict point of POSIX compliance. I think there's a point to be made though about just stability of the functional/CLI interface, aside from what's technically in the POSIX standards. ripgrep is pretty damn good on interface stability AFAIK, but the other tools that similarly don't try to position themselves as POSIX+/POSIXish compatible, are a mixed bag. It's nice to just have the stable tools you know, and have them practically everywhere (if you include WSL/msys2/Git for windows, etc.). And similarly you pretty much know what POSIX+ features you have on a GNU system (most Linuxes), or a BSD one (macOS, OpenBSD, etc.). I like having an excellent set of system tools from GNU/BSD, etc., then I can install/use the SotA stuff -- and I'll still end up using both sets of tools all the time, even though one set is not absolute best in class, because I don't have to worry about how to use sota-tool 1.2 on one system vs sota-tool 2.1 on another, when there may be important interface changes. And to provide the full context going back to my first comment: I won't consider using "performance will be looked at later" tools whose purported benefit is just the use of Rust, pretty much at all.

jgarzik 2y ago

I wouldn't assume. The awk implementation likely stands up well for a 1-billion-row challenge, with its thoughtful bytecode-based design. Redditors ran some quick performance tests on parsing, also: https://www.reddit.com/r/rust/comments/1fd7qgl/comment/lmelo6z/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button https://www.reddit.com/r/rust/comments/1fd7qgl/comment/lmelo...

dundarious 2y ago
Parsing the awk input? While not an irrelevant concern, that is obviously on the low priority end when considering awk performance in general. My most often used awk program is just '{print $1}', but I use it on enormous files. The performance when operating on the enormous file is the concern wrt performance, not the initial parse of '{print $1}' or of command line arguments. I know you're just directly responding to the concerns of a parent comment though.
- jgarzik 2y ago
  It is hoped that posixutil's awk's bytecode-based modern design should keep performance high, theoretically higher than ancient C-based awks. Inspired by Ray Gardner's "wak" awk implementation https://github.com/raygard/wak https://github.com/raygard/wak A volunteer benchmarking our awk on a 1-billion line text file would be welcome.
  - dundarious 2y ago
    Yes, that's the most important part to focus on. I have nothing to remark about it, I haven't seen/gathered any numbers on it.