6 ms·

Frawk: A fast, JITted, statically-typed AWK written in Rust

145 points by benhoyt 5y ago

yjftsjthsd-h 5y ago
Drat: > To a first approximation, it is an implementation of the AWK language; many common Awk programs produce equivalent output when passed to frawk. So it sounds like it's (intentionally) not a drop in replacement. That's probably reasonable in context but unfortunate in terms of making adoption harder.
- the_duke 5y ago
  The Overview page has plenty of details explaining the differences and why they exist: https://github.com/ezrosent/frawk/blob/master/info/overview.md https://github.com/ezrosent/frawk/blob/master/info/overview....
  - dotancohen 5y ago
    The only real-world difference that I see would be that frawk uses Rust's regex syntax, not awk's regex syntax. I'm actually willing to write that off as an improvement, though it may break scripts. I'd like to see this hidden behind a flag, and the classic awk regex syntax used without the flag. For now, while the classic syntax is unavailable, frawk should error wit ha message that classic syntax is not yet supported. Right now it is too big a footgun. That said, gawk uses PCRE but awk does not. So varying regex implmentations already infect the awk ecosystem. I believe that Debian distros actually install mawk when installing awk via the standard repos. mawk has its own incompatibilities with awk, such as empty string representation.
    - asicsp 5y ago
      >gawk uses PCRE ERE with GNU extensions would be more apt here (PCRE has many more powerful features): https://www.gnu.org/software/gawk/manual/html_node/GNU-Regexp-Operators.html#GNU-Regexp-Operators https://www.gnu.org/software/gawk/manual/html_node/GNU-Regex...
- usr1106 5y ago
  How would a statically typed language be a drop-in for something that isn't?
  - matharmin 5y ago
    This is a different compiler/interpreter for the same language - nothing prevents it from being a drop-in replacement.
    - tialaramex 5y ago
      But the specific design of Frawk is that it's statically typed. Hence both the title of this thread and the description in the linked documentation. If Frawk sees that you've got a variable N, and you add 1 to it, Frawk concludes that's a number. Seems fair enough and it makes Frawk faster. However for example, Frawk may be quite sure that something is a number, and so when there isn't a value for that - Frawk says the value is a zero. But Awk isn't fixated on the idea that it's always a number, if there's no value awk will say it is the empty string, and why not. Lots of Awk programs don't care, but if yours does it will behave differently under Frawk.
  - u320 5y ago
    I guess the assumption is that working AWK programs will typecheck.
asicsp 5y ago
I tried this last year when I started writing a book with Rust cli tools like ripgrep, hck, sd, huniq, etc. Was impressed with the performance. Found a few syntax issues when I simply tried to run gawk one-liners I had with frawk and they were fixed by the author. Hope the project crosses version 1.0 soon.
- deleted 5y ago
  [deleted]
frafra 5y ago
> Lack of support for structured CSV input data. Alternative approach that can be used sometimes: use TSV (tab-separated values), which make parsing trivial and works great with a wide range of tools; conversion from CSV to TSV can be done with csvkit or similar.
scandox 5y ago
> You will need to install Rust Am I right in saying Rust can produce executables that can be run without any Rust runtime etc being on the target system? Why would you need to install Rust in cases like this?
- carlmr 5y ago
  To build and install it, you could have a binary release instead of using cargo.
- chrisseaton 5y ago
  Rust is a compiler, so you need to install it to compile this application, which you can then use without Rust. You need an oven to bake a cake but not to eat one.
  - georgyo 5y ago
    Java is both a compiler and a run time. You can ship someone a compiled Java and they will still need the JVM to actually run it. Sometimes you need the oven to bake a cake and eat it to.
    - chrisseaton 5y ago
      Well you can AOT Java to a stand-alone binary too - but none of this is relevant to Rust.
- OJFord 5y ago
  Yes you are, but the author hasn't provided those executables, so you need to produce one for your platform yourself; for which you need to install rust. (Presumably it's deemed not worthwhile providing them while it's unstable/in active pre-1.0 development.)
mkdirp 5y ago
There is also goAWK[0], a re-implementation of AWK in Go. [0] https://github.com/benhoyt/goawk https://github.com/benhoyt/goawk
- adenozine 5y ago
  GoAWK is a great tool. Added benefit of POSIX-compliance, it builds FAST, and is just generally much more reliable and enjoyable to work with. I'll be honest, I'm still quick to reach for perl/python for most awk-able tasks, but, I'm often impressed with Awk and its ilk.
dontlaugh 5y ago
Interesting (and a bit surprising) that the cranelift backend keeps up with LLVM backend reasonably well, especially with parallelism enabled https://github.com/ezrosent/frawk/blob/master/info/performance.md https://github.com/ezrosent/frawk/blob/master/info/performan...
ogogmad 5y ago
Note: You can do many of these one-liners using tools that understand Python: Xonsh or Pyp. You won't need to learn Awk -- let alone any non-standard dialect of Awk. It won't be as fast as some Awk implementations (maybe), but you'll have a gentle on-ramp and off-ramp to a more featureful programming language. I think the problem with DSLs is when a problem only partially fits into what they can do.
- brigandish 5y ago
  What are the chances that Xonsh or Pyp will be installed on a system I wish to use?
  - ogogmad 5y ago
    The submission is about a non-standard dialect of Awk. There is zero chance you'll have that on any random system you might want to use. If it's between that and Pyp, why not use Pyp? And also, in the technology business, you are supposed to make progress happen. Sometimes, that involves making older technology obsolete. Making things obsolete is sometimes good, especially if the older thing has major problems.
    - brigandish 5y ago
      > The submission is about a non-standard dialect of Awk True, but the comment I responded to also has this: > You won't need to learn Awk Regardless, let's accept the claim that it's only about non-standard frawk. > If it's between that and Pyp, why not use Pyp? Because, as the instructions[0] point out > Run pip install pypyp (note the extra "yp"!) > pyp requires Python 3.6 or above. It's necessary to install pip and Python at least to do so, I expect that I can produce a binary for frawk and drop it in far more easily. I'm not even sure what advantage Pyp has over using a Python REPL (or Ruby or perl etc). > And also, in the technology business, you are supposed to make progress happen. I do that by writing good code and improving processes. As I've pointed out, I fail to see how needing to include the Python ecosystem is "progress" over a binary that is tiny, fast, and works. Calling things obsolete because they're "old" is the argument of a teenager. I apologise if that sounds rude, but I don't know how else to put it, that's what it is. [0] https://github.com/hauntsaninja/pyp https://github.com/hauntsaninja/pyp
deleted 5y ago
[deleted]
htfy96 5y ago
One of my recent project requires a fast awk implementation, and I tried frawk and got surprised by how robust it is. Although eventually we chose https://github.com/noyesno/awka https://github.com/noyesno/awka for better native awk script compatibility, it's still wonderful to see a project incorporating recent advances of programming language implementation into the ancient awk.
prirun 5y ago
There's also this for the Python fans: https://github.com/alecthomas/pawk https://github.com/alecthomas/pawk
hawski 5y ago
A bit offtopic, but has someone implemented a structured regex awk? I'm constantly hoping it will become a thing.
pepproni 5y ago
In the example I noticed a PREPARE block in addition to the well known BEGIN and END blocks. After a few minutes of searching it appears to be frawk specific: https://github.com/ezrosent/frawk/blob/master/info/parallelism.md#aggregations https://github.com/ezrosent/frawk/blob/master/info/paralleli... "Because the repeated map references are both annoying to write and inefficient to execute, frawk has a PREPARE block which executes in the worker threads at the end of its input"
jhallenworld 5y ago
Neat tool! I like this is as a small example of SSA-based dataflow analysis.

145 points by benhoyt 5y ago

Drat: > To a first approximation, it is an implementation of the AWK language; many common Awk programs produce equivalent output when passed to frawk. So it sounds like it's (intentionally) not a drop in replacement. That's probably reasonable in context but unfortunate in terms of making adoption harder.

the_duke 5y ago
The Overview page has plenty of details explaining the differences and why they exist: https://github.com/ezrosent/frawk/blob/master/info/overview.md https://github.com/ezrosent/frawk/blob/master/info/overview....
- dotancohen 5y ago
  The only real-world difference that I see would be that frawk uses Rust's regex syntax, not awk's regex syntax. I'm actually willing to write that off as an improvement, though it may break scripts. I'd like to see this hidden behind a flag, and the classic awk regex syntax used without the flag. For now, while the classic syntax is unavailable, frawk should error wit ha message that classic syntax is not yet supported. Right now it is too big a footgun. That said, gawk uses PCRE but awk does not. So varying regex implmentations already infect the awk ecosystem. I believe that Debian distros actually install mawk when installing awk via the standard repos. mawk has its own incompatibilities with awk, such as empty string representation.
  - asicsp 5y ago
    >gawk uses PCRE ERE with GNU extensions would be more apt here (PCRE has many more powerful features): https://www.gnu.org/software/gawk/manual/html_node/GNU-Regexp-Operators.html#GNU-Regexp-Operators https://www.gnu.org/software/gawk/manual/html_node/GNU-Regex...
usr1106 5y ago
How would a statically typed language be a drop-in for something that isn't?
- matharmin 5y ago
  This is a different compiler/interpreter for the same language - nothing prevents it from being a drop-in replacement.
  - tialaramex 5y ago
    But the specific design of Frawk is that it's statically typed. Hence both the title of this thread and the description in the linked documentation. If Frawk sees that you've got a variable N, and you add 1 to it, Frawk concludes that's a number. Seems fair enough and it makes Frawk faster. However for example, Frawk may be quite sure that something is a number, and so when there isn't a value for that - Frawk says the value is a zero. But Awk isn't fixated on the idea that it's always a number, if there's no value awk will say it is the empty string, and why not. Lots of Awk programs don't care, but if yours does it will behave differently under Frawk.
- u320 5y ago
  I guess the assumption is that working AWK programs will typecheck.

asicsp 5y ago

I tried this last year when I started writing a book with Rust cli tools like ripgrep, hck, sd, huniq, etc. Was impressed with the performance. Found a few syntax issues when I simply tried to run gawk one-liners I had with frawk and they were fixed by the author. Hope the project crosses version 1.0 soon.

deleted 5y ago
[deleted]

frafra 5y ago

> Lack of support for structured CSV input data. Alternative approach that can be used sometimes: use TSV (tab-separated values), which make parsing trivial and works great with a wide range of tools; conversion from CSV to TSV can be done with csvkit or similar.

scandox 5y ago

> You will need to install Rust Am I right in saying Rust can produce executables that can be run without any Rust runtime etc being on the target system? Why would you need to install Rust in cases like this?

carlmr 5y ago
To build and install it, you could have a binary release instead of using cargo.
chrisseaton 5y ago
Rust is a compiler, so you need to install it to compile this application, which you can then use without Rust. You need an oven to bake a cake but not to eat one.
- georgyo 5y ago
  Java is both a compiler and a run time. You can ship someone a compiled Java and they will still need the JVM to actually run it. Sometimes you need the oven to bake a cake and eat it to.
  - chrisseaton 5y ago
    Well you can AOT Java to a stand-alone binary too - but none of this is relevant to Rust.
OJFord 5y ago
Yes you are, but the author hasn't provided those executables, so you need to produce one for your platform yourself; for which you need to install rust. (Presumably it's deemed not worthwhile providing them while it's unstable/in active pre-1.0 development.)

mkdirp 5y ago

There is also goAWK[0], a re-implementation of AWK in Go. [0] https://github.com/benhoyt/goawk https://github.com/benhoyt/goawk

adenozine 5y ago
GoAWK is a great tool. Added benefit of POSIX-compliance, it builds FAST, and is just generally much more reliable and enjoyable to work with. I'll be honest, I'm still quick to reach for perl/python for most awk-able tasks, but, I'm often impressed with Awk and its ilk.

dontlaugh 5y ago

Interesting (and a bit surprising) that the cranelift backend keeps up with LLVM backend reasonably well, especially with parallelism enabled https://github.com/ezrosent/frawk/blob/master/info/performance.md https://github.com/ezrosent/frawk/blob/master/info/performan...

ogogmad 5y ago

Note: You can do many of these one-liners using tools that understand Python: Xonsh or Pyp. You won't need to learn Awk -- let alone any non-standard dialect of Awk. It won't be as fast as some Awk implementations (maybe), but you'll have a gentle on-ramp and off-ramp to a more featureful programming language. I think the problem with DSLs is when a problem only partially fits into what they can do.

brigandish 5y ago
What are the chances that Xonsh or Pyp will be installed on a system I wish to use?
- ogogmad 5y ago
  The submission is about a non-standard dialect of Awk. There is zero chance you'll have that on any random system you might want to use. If it's between that and Pyp, why not use Pyp? And also, in the technology business, you are supposed to make progress happen. Sometimes, that involves making older technology obsolete. Making things obsolete is sometimes good, especially if the older thing has major problems.
  - brigandish 5y ago
    > The submission is about a non-standard dialect of Awk True, but the comment I responded to also has this: > You won't need to learn Awk Regardless, let's accept the claim that it's only about non-standard frawk. > If it's between that and Pyp, why not use Pyp? Because, as the instructions[0] point out > Run pip install pypyp (note the extra "yp"!) > pyp requires Python 3.6 or above. It's necessary to install pip and Python at least to do so, I expect that I can produce a binary for frawk and drop it in far more easily. I'm not even sure what advantage Pyp has over using a Python REPL (or Ruby or perl etc). > And also, in the technology business, you are supposed to make progress happen. I do that by writing good code and improving processes. As I've pointed out, I fail to see how needing to include the Python ecosystem is "progress" over a binary that is tiny, fast, and works. Calling things obsolete because they're "old" is the argument of a teenager. I apologise if that sounds rude, but I don't know how else to put it, that's what it is. [0] https://github.com/hauntsaninja/pyp https://github.com/hauntsaninja/pyp

deleted 5y ago

[deleted]

htfy96 5y ago

One of my recent project requires a fast awk implementation, and I tried frawk and got surprised by how robust it is. Although eventually we chose https://github.com/noyesno/awka https://github.com/noyesno/awka for better native awk script compatibility, it's still wonderful to see a project incorporating recent advances of programming language implementation into the ancient awk.

prirun 5y ago

There's also this for the Python fans: https://github.com/alecthomas/pawk https://github.com/alecthomas/pawk

hawski 5y ago

A bit offtopic, but has someone implemented a structured regex awk? I'm constantly hoping it will become a thing.

pepproni 5y ago

In the example I noticed a PREPARE block in addition to the well known BEGIN and END blocks. After a few minutes of searching it appears to be frawk specific: https://github.com/ezrosent/frawk/blob/master/info/parallelism.md#aggregations https://github.com/ezrosent/frawk/blob/master/info/paralleli... "Because the repeated map references are both annoying to write and inefficient to execute, frawk has a PREPARE block which executes in the worker threads at the end of its input"

jhallenworld 5y ago

Neat tool! I like this is as a small example of SSA-based dataflow analysis.