4 ms·

I’m mostly happy with awk. But splitting strings really sucks. I have to read the docs every time. I think my dream awk-like tool would look something like: 1

I’m mostly happy with awk. But splitting strings really sucks. I have to read the docs every time. I think my dream awk-like tool would look something like: 1. Has an “interactive” mode to see what your script is doing as you write it. Something like a combination of less, fzf, and a Bret Victor style debugger showing matches/values of variables at each line 2. Supports things that aren’t list RS-separated records of of FS-separated fields. Some formats I would like are json, csv, records where the fields are all key=value, and maybe some other formats. Support would mean some way to specify patterns for different formats 3. Extracting marching groups from regex matches.

jiggawatts 6y ago
The thing you have to be really careful about is escaping. Compared to the Object Oriented PowerShell where every column is just an object property, strongly typed and everything, the string-based bash programming to me seems absolutely bonkers. Like... what do you do if some text doesn't fit into the space available? How do you handle Unicode? What about simple escaping of names like O'Toole, embedded double quotes, leading or trailing spaces that are meaningful, embedded line feeds, etc... Eventually you have to use a full parser, not a bunch of regexes. I've found that "eventually" to mean: almost immediately, even for supposedly simple problems. Even seeming trivial things like correctly splitting up an X.500 name as seen in LDAP or PKI is deceptively difficult. Now, if it's a LDAP name that includes quotes embedded in a CSV... err... I don't even know where to begin.
- theamk 6y ago
  I found tab-separated files can get me pretty far. It can handle arbitrary length text, Unicode is no problem at all, leading/trailing spaces are preserved and so on. It is trivially convertible to name=value format, as long as you guarantee the "name" part is well-formed (usually the case). The only tricky parts are embedded line feeds and tabs, and those can be defeated with trivial escape schema, even something as simple as '\r', '\t', '\\'. Bash quoting also works. If you master the difference between "${x[@]}" and ${x[*]}, keep in mind order of expand / split all the time, and never forget the right kind of quotes, you can have a robust system which can handle arbitrary strings. But I would not recommend this to anyone.
  - benibela 6y ago
    Or ASCII control characters No reason to use tabs, when there explicit group/record/unit separators
- majkinetor 6y ago
  Totally. In bash 99% of the time is lost in side effects. PowerShell makes everything possible in 3 lines of code.
  - mickotron 6y ago
    You try doing any reasonable data processing with PowerShell and it will use a lot of resources. You can accomplish a lot with awk. I was a PowerShell advocate but it still has a lot of rough edges which is disappointing. And is very verbose. Sometimes grep | awk can be incredibly succinct and fast.
    - majkinetor 6y ago
      Not sure what you mean because it uses dot.net regex. Sure, its way more generally resource hungry then awk but again, it does tone of things. Its also good to know that you have that power ASAP on Windows and without any extra toolson *nix such as awk or jq or sed. I regularly handle 100MB files with Powershell regex , yeah, I wait few seconds here and there, but nothing except ripgrep handles this fast. Verbosity is nonsensical arugment for anything as you have alises. Bash is equally verbose if you use long parameter names.
tylorr 6y ago
This doesn't have the debugger but it allows for a repl interface for awk https://paweldu.dev/posts/fzf-live-repl/ https://paweldu.dev/posts/fzf-live-repl/ edit: There's also https://github.com/akavel/up https://github.com/akavel/up which could possibly be used in combination with fzf
- hawski 6y ago
  That's useful! Thanks for the pointer. Although one must be careful, because awk still can overwrite files and shell out to rm.
asicsp 6y ago
There's a online website [0] for mawk 1.3.4 that accepts input data, command and shows output as you construct the command There's a handy github repo [1] with libraries, has one for csv. For json, see [2]. But yeah, having official support would be better. Or, use tools like xsv, jq, etc that are built for those formats. You can use match() function to get capture group contents. If you are using GNU awk, then the syntax is much easier with arrays instead of fiddling with substr and RSTART/RLENGTH. For examples, see my repo [3] [0] https://awk.js.org/ https://awk.js.org/ [1] https://github.com/e36freak/awk-libs https://github.com/e36freak/awk-libs [2] https://github.com/step-/JSON.awk https://github.com/step-/JSON.awk [3] https://github.com/learnbyexample/learn_gnuawk/blob/master/gnu_awk.md#match https://github.com/learnbyexample/learn_gnuawk/blob/master/g...
- fomine3 6y ago
  AWK REPL looks useful. TIL
hawski 6y ago
1. I was thinking many times about this - an awk IDE, that would also handle common utility functions (i.e. there is no convenient way to print a range of fields). Maybe an Awk Language Server would be a fun project. 2 & 3. Structural regular expressions awk is something I long for. If someone wants to rewrite awk in Rust or whatever, please add this, that would be a killer feature. http://doc.cat-v.org/bell_labs/structural_regexps/ http://doc.cat-v.org/bell_labs/structural_regexps/ EDIT: There is an Awk Language Server! - https://github.com/fwip/awk-language-server https://github.com/fwip/awk-language-server

jiggawatts 6y ago

The thing you have to be really careful about is escaping. Compared to the Object Oriented PowerShell where every column is just an object property, strongly typed and everything, the string-based bash programming to me seems absolutely bonkers. Like... what do you do if some text doesn't fit into the space available? How do you handle Unicode? What about simple escaping of names like O'Toole, embedded double quotes, leading or trailing spaces that are meaningful, embedded line feeds, etc... Eventually you have to use a full parser, not a bunch of regexes. I've found that "eventually" to mean: almost immediately, even for supposedly simple problems. Even seeming trivial things like correctly splitting up an X.500 name as seen in LDAP or PKI is deceptively difficult. Now, if it's a LDAP name that includes quotes embedded in a CSV... err... I don't even know where to begin.

theamk 6y ago
I found tab-separated files can get me pretty far. It can handle arbitrary length text, Unicode is no problem at all, leading/trailing spaces are preserved and so on. It is trivially convertible to name=value format, as long as you guarantee the "name" part is well-formed (usually the case). The only tricky parts are embedded line feeds and tabs, and those can be defeated with trivial escape schema, even something as simple as '\r', '\t', '\\'. Bash quoting also works. If you master the difference between "${x[@]}" and ${x[*]}, keep in mind order of expand / split all the time, and never forget the right kind of quotes, you can have a robust system which can handle arbitrary strings. But I would not recommend this to anyone.
- benibela 6y ago
  Or ASCII control characters No reason to use tabs, when there explicit group/record/unit separators
majkinetor 6y ago
Totally. In bash 99% of the time is lost in side effects. PowerShell makes everything possible in 3 lines of code.
- mickotron 6y ago
  You try doing any reasonable data processing with PowerShell and it will use a lot of resources. You can accomplish a lot with awk. I was a PowerShell advocate but it still has a lot of rough edges which is disappointing. And is very verbose. Sometimes grep | awk can be incredibly succinct and fast.
  - majkinetor 6y ago
    Not sure what you mean because it uses dot.net regex. Sure, its way more generally resource hungry then awk but again, it does tone of things. Its also good to know that you have that power ASAP on Windows and without any extra toolson *nix such as awk or jq or sed. I regularly handle 100MB files with Powershell regex , yeah, I wait few seconds here and there, but nothing except ripgrep handles this fast. Verbosity is nonsensical arugment for anything as you have alises. Bash is equally verbose if you use long parameter names.

tylorr 6y ago

This doesn't have the debugger but it allows for a repl interface for awk https://paweldu.dev/posts/fzf-live-repl/ https://paweldu.dev/posts/fzf-live-repl/ edit: There's also https://github.com/akavel/up https://github.com/akavel/up which could possibly be used in combination with fzf

hawski 6y ago
That's useful! Thanks for the pointer. Although one must be careful, because awk still can overwrite files and shell out to rm.

asicsp 6y ago

There's a online website [0] for mawk 1.3.4 that accepts input data, command and shows output as you construct the command There's a handy github repo [1] with libraries, has one for csv. For json, see [2]. But yeah, having official support would be better. Or, use tools like xsv, jq, etc that are built for those formats. You can use match() function to get capture group contents. If you are using GNU awk, then the syntax is much easier with arrays instead of fiddling with substr and RSTART/RLENGTH. For examples, see my repo [3] [0] https://awk.js.org/ https://awk.js.org/ [1] https://github.com/e36freak/awk-libs https://github.com/e36freak/awk-libs [2] https://github.com/step-/JSON.awk https://github.com/step-/JSON.awk [3] https://github.com/learnbyexample/learn_gnuawk/blob/master/gnu_awk.md#match https://github.com/learnbyexample/learn_gnuawk/blob/master/g...

fomine3 6y ago
AWK REPL looks useful. TIL

hawski 6y ago

1. I was thinking many times about this - an awk IDE, that would also handle common utility functions (i.e. there is no convenient way to print a range of fields). Maybe an Awk Language Server would be a fun project. 2 & 3. Structural regular expressions awk is something I long for. If someone wants to rewrite awk in Rust or whatever, please add this, that would be a killer feature. http://doc.cat-v.org/bell_labs/structural_regexps/ http://doc.cat-v.org/bell_labs/structural_regexps/ EDIT: There is an Awk Language Server! - https://github.com/fwip/awk-language-server https://github.com/fwip/awk-language-server