3 ms·

> The paragraph right below that mentions that Oil has "read -0", which consumes the find -print0 input. Yeah, I read that ("read -0", for the record, is an ex

by l0b0 6y ago

> The paragraph right below that mentions that Oil has "read -0", which consumes the find -print0 input. Yeah, I read that ("read -0", for the record, is an excellent idea). QSN for filenames still a bad idea. Your wc, awk, grep etc commands will now have to decode the stream after splitting it. Taking it to the extreme, it's like mixing JSON and XML (or CSV and TSV) because some things work better in one or the other. Right now, even in Bash, I can use newline-terminated tokens and accept that newlines are going to screw up everything (which is fine if I control the filenames), or I can use NUL-terminated tokens and force my code to handle absolutely all possible inputs. No extra parsing library necessary. As long as filenames can contain any character except slash and NUL, terminating them by NUL is a simple solution (at least when using GNU tools). > Shell scripts can use both formats, but the advantage to QSN is that it preserves the line-based nature of shell. Except when your tokens can contain newlines, which we're stuck with for the foreseeable future. > Say I want to use wc -l, awk, or grep. Then the QSN-lines format is better than the NUL format. Can't agree, for the reasons above. > Also, you can transmit a series of say SHA256 checksums in binary format with QSN-lines, or some other binary format. > Even entire JPG files, audio clips, wasm files, whatever. It's "8-bit clean" (and so are Oil strings). That's a completely different problem set. You could use existing simple encodings like base64. "Why use QSN over base64" would be an interesting blog post. In any case, I admire the courage to try to improve the state of the art! I would absolutely love to see something better than POSIX/Bash as the baseline. At this point I suspect we'll need to ignore those and go for something radically different like PowerShell to regain sanity.

chubot 6y ago
A bunch of points: (1) Except when your tokens can contain newlines, which we're stuck with for the foreseeable future. -- not sure what you mean here, because QSN strings are defined not to contain literal newlines. They're escaped like '\n'. The invariant of QSN is: EVERY BYTE STRING, including those with newlines and nuls, can be represented on a single line. I guess I should put this in the documentation. (2) A QSN decoder is very easy to write. For example, here's a ~6 line regex that validates all of QSN: https://github.com/oilshell/oil/blob/master/qsn_/qsn.py#L498 https://github.com/oilshell/oil/blob/master/qsn_/qsn.py#L498 True, you need like ~20 lines of code to decode it, but that's very easy too. You can also make a QSN decoder from a JSON string decoder. It's basically changing your tests and moving a few statements around. (And yes, QSN is a regular language [1]) (3) Oil should grow [2] an awk-like dialect [3] that understands QSN and QTSV. (4) Although you also don't have to decode it for it to be useful. 1. I can use wc -l on a stream of QSN strings 2. If I know the strings are QSN-encoded, I can search for NUL bytes with fgrep '\0' It's basically like the UTF-8 philosophy. ASCII is valid utf-8. QSN lines are lines of text. (5) base64 is bad for humans at the terminal, because it makes everything unreadable. QSN preserves all printable ASCII and unicode. ----- I don't think it's going to solve all problems, but it will solve some. It's there if you need it. In typical Unix style, the solutions will be heterogeneous. You can absolutely use the NUL format in Oil, and it now has support for it. [1] http://www.oilshell.org/blog/2020/07/eggex-theory.html http://www.oilshell.org/blog/2020/07/eggex-theory.html [2] That is, if I get help :) [3] http://www.oilshell.org/blog/tags.html?tag=awk#awk http://www.oilshell.org/blog/tags.html?tag=awk#awk
- l0b0 6y ago
  > (1) Except when your tokens can contain newlines, which we're stuck with for the foreseeable future. -- not sure what you mean here, because QSN strings are defined not to contain literal newlines. They're escaped like '\n'. The tokens in question are filenames, which are (and will continue to be) allowed to contain newline characters. Sorry that wasn't clear. This was never meant to be a critique of QSN, I'm just saying QSN is unnecessary and unnecessarily complex for between-process communication: - In case of filenames, just use NUL. It already works, is supported by a bunch of tools, and will never have the overhead of encoding/decoding. - In case of arbitrary binary streams, just send the stream unaltered. Why encode/decode it? What benefit is that? In the very rare case that I'm actually manually inspecting a stream of bytes (debugger/printf) there's always some easy way to encode those bytes for easy readability such as `printf '%q\n'`. > (2) A QSN decoder is very easy to write. For example, here's a ~6 line regex that validates all of QSN: It would take me at least a day to write enough test cases to ensure that that regex in fact does what it says. But the point is moot: encoding and decoding should only happen at the interface with a human, not with another process. > (3) Oil should grow [2] an awk-like dialect [3] that understands QSN and QTSV. I couldn't comment; I don't use awk unless I absolutely have to. Once I need to use awk a shell script is no longer the best tool for the job, except for quick once-only processing. > (4) Although you also don't have to decode it for it to be useful. > 1. I can use wc -l on a stream of QSN strings I can already use `wc --files0-from=-` on a stream of NUL-separated tokens. > 2. If I know the strings are QSN-encoded, I can search for NUL bytes with fgrep '\0' You can already `grep` for NUL bytes[1]. > (5) base64 is bad for humans at the terminal, because it makes everything unreadable. QSN preserves all printable ASCII and unicode. But filenames are neither ASCII nor Unicode - they are bytes, which can include 0x80 through 0xFF (not valid ASCII) and things like 0xFF (not valid UTF-8). I still can't see a single use case where I'd want inter-process communication to use QSN. Human input, sure, I'd love something more convenient that `$''`. Human output, also sure, humans can't read NUL characters or tell the difference between spaces and tabs after all. [1] https://superuser.com/a/612336/2259 https://superuser.com/a/612336/2259
  - chubot 6y ago
    If you've ever used JSON between processes, then QSN is the exact same idea. JSON strings are also encoded (but they can't represent byte strings). There are probably people who never use JSON, and that's fine, but there are a lot more who do! The goal is absolutely for tools like grep to adopt QSN support. [1] Another argument is that in networking framing, you have a few choices: (1) delimiter-based (NUL bytes) (2) escaping (QSN with \) (3) length prefixed (netstrings) So Oil supports #1 and #2 now. Oil doesn't support netstrings but it actually does make sense, and avoids encoding/decoding. Though QSN encoding can be made extremely fast, just like it's been done with JSON [2]. [1] GNU grep actually has the ASCII/binary detection problem that was brought up on HN awhlie ago. Honestly it would be worth making a "slow correct utf-8/QSN grep" that avoids this. https://unix.stackexchange.com/questions/19907/what-makes-grep-consider-a-file-to-be-binary https://unix.stackexchange.com/questions/19907/what-makes-gr... [2] https://github.com/simdjson/simdjson https://github.com/simdjson/simdjson

by l0b0 6y ago

chubot 6y ago

A bunch of points: (1) Except when your tokens can contain newlines, which we're stuck with for the foreseeable future. -- not sure what you mean here, because QSN strings are defined not to contain literal newlines. They're escaped like '\n'. The invariant of QSN is: EVERY BYTE STRING, including those with newlines and nuls, can be represented on a single line. I guess I should put this in the documentation. (2) A QSN decoder is very easy to write. For example, here's a ~6 line regex that validates all of QSN: https://github.com/oilshell/oil/blob/master/qsn_/qsn.py#L498 https://github.com/oilshell/oil/blob/master/qsn_/qsn.py#L498 True, you need like ~20 lines of code to decode it, but that's very easy too. You can also make a QSN decoder from a JSON string decoder. It's basically changing your tests and moving a few statements around. (And yes, QSN is a regular language [1]) (3) Oil should grow [2] an awk-like dialect [3] that understands QSN and QTSV. (4) Although you also don't have to decode it for it to be useful. 1. I can use wc -l on a stream of QSN strings 2. If I know the strings are QSN-encoded, I can search for NUL bytes with fgrep '\0' It's basically like the UTF-8 philosophy. ASCII is valid utf-8. QSN lines are lines of text. (5) base64 is bad for humans at the terminal, because it makes everything unreadable. QSN preserves all printable ASCII and unicode. ----- I don't think it's going to solve all problems, but it will solve some. It's there if you need it. In typical Unix style, the solutions will be heterogeneous. You can absolutely use the NUL format in Oil, and it now has support for it. [1] http://www.oilshell.org/blog/2020/07/eggex-theory.html http://www.oilshell.org/blog/2020/07/eggex-theory.html [2] That is, if I get help :) [3] http://www.oilshell.org/blog/tags.html?tag=awk#awk http://www.oilshell.org/blog/tags.html?tag=awk#awk

l0b0 6y ago
> (1) Except when your tokens can contain newlines, which we're stuck with for the foreseeable future. -- not sure what you mean here, because QSN strings are defined not to contain literal newlines. They're escaped like '\n'. The tokens in question are filenames, which are (and will continue to be) allowed to contain newline characters. Sorry that wasn't clear. This was never meant to be a critique of QSN, I'm just saying QSN is unnecessary and unnecessarily complex for between-process communication: - In case of filenames, just use NUL. It already works, is supported by a bunch of tools, and will never have the overhead of encoding/decoding. - In case of arbitrary binary streams, just send the stream unaltered. Why encode/decode it? What benefit is that? In the very rare case that I'm actually manually inspecting a stream of bytes (debugger/printf) there's always some easy way to encode those bytes for easy readability such as `printf '%q\n'`. > (2) A QSN decoder is very easy to write. For example, here's a ~6 line regex that validates all of QSN: It would take me at least a day to write enough test cases to ensure that that regex in fact does what it says. But the point is moot: encoding and decoding should only happen at the interface with a human, not with another process. > (3) Oil should grow [2] an awk-like dialect [3] that understands QSN and QTSV. I couldn't comment; I don't use awk unless I absolutely have to. Once I need to use awk a shell script is no longer the best tool for the job, except for quick once-only processing. > (4) Although you also don't have to decode it for it to be useful. > 1. I can use wc -l on a stream of QSN strings I can already use `wc --files0-from=-` on a stream of NUL-separated tokens. > 2. If I know the strings are QSN-encoded, I can search for NUL bytes with fgrep '\0' You can already `grep` for NUL bytes[1]. > (5) base64 is bad for humans at the terminal, because it makes everything unreadable. QSN preserves all printable ASCII and unicode. But filenames are neither ASCII nor Unicode - they are bytes, which can include 0x80 through 0xFF (not valid ASCII) and things like 0xFF (not valid UTF-8). I still can't see a single use case where I'd want inter-process communication to use QSN. Human input, sure, I'd love something more convenient that `$''`. Human output, also sure, humans can't read NUL characters or tell the difference between spaces and tabs after all. [1] https://superuser.com/a/612336/2259 https://superuser.com/a/612336/2259
- chubot 6y ago
  If you've ever used JSON between processes, then QSN is the exact same idea. JSON strings are also encoded (but they can't represent byte strings). There are probably people who never use JSON, and that's fine, but there are a lot more who do! The goal is absolutely for tools like grep to adopt QSN support. [1] Another argument is that in networking framing, you have a few choices: (1) delimiter-based (NUL bytes) (2) escaping (QSN with \) (3) length prefixed (netstrings) So Oil supports #1 and #2 now. Oil doesn't support netstrings but it actually does make sense, and avoids encoding/decoding. Though QSN encoding can be made extremely fast, just like it's been done with JSON [2]. [1] GNU grep actually has the ASCII/binary detection problem that was brought up on HN awhlie ago. Honestly it would be worth making a "slow correct utf-8/QSN grep" that avoids this. https://unix.stackexchange.com/questions/19907/what-makes-grep-consider-a-file-to-be-binary https://unix.stackexchange.com/questions/19907/what-makes-gr... [2] https://github.com/simdjson/simdjson https://github.com/simdjson/simdjson