3 ms·

There are some good ideas in here, but `--qsn` is described[1] like this: > Print filenames ONE PER LINE. If a name contains a newline or other special char,

by l0b0 6y ago

There are some good ideas in here, but `--qsn` is described[1] like this: > Print filenames ONE PER LINE. If a name contains a newline or other special char, it's QSN-encoded like 'multi-line \n name with NUL \0 byte' This is a bad idea. There is a solution which already works. Just terminate every filename with NUL and you're done. Trying to make something "sorta kinda human readable" is a mistake when dealing with what is effectively arbitrary byte sequences which can contain anything except for NUL. Consistency makes the consuming code much simpler. That said, QSN is a great idea for human input. [1] https://www.oilshell.org/blog/2020/10/osh-features.html#safe-processing-of-user-supplied-data-like-filenames https://www.oilshell.org/blog/2020/10/osh-features.html#safe...

e12e 6y ago
I think having both is a good idea. Sometimes maybe you get your list of path names from a Here-document, for example.
- l0b0 6y ago
  For sure we need both human- and machine-readable stream formats, but I don't really see how QSN has any advantage in that space over `printf '%q\n'` and NUL-terminated strings, respectively.
  - chubot 6y ago
    I actually considered that, but it doesn't work. https://github.com/oilshell/oil/wiki/Shell-Almost-Has-a-JSON-Analogue https://github.com/oilshell/oil/wiki/Shell-Almost-Has-a-JSON... Single quoted strings in shell can't represent arbitrary strings either. And the %q format is actually different from the format ${x@Q} emits. QSN is a well-specified format.
chubot 6y ago
The paragraph right below that mentions that Oil has "read -0", which consumes the find -print0 input. I also link to my Git Log in HTML post [1] from 3 years ago, which is ENTIRELY about the NUL byte solution :) ----- Shell scripts can use both formats, but the advantage to QSN is that it preserves the line-based nature of shell. Say I want to use wc -l, awk, or grep. Then the QSN-lines format is better than the NUL format. Also, you can transmit a series of say SHA256 checksums in binary format with QSN-lines, or some other binary format. Even entire JPG files, audio clips, wasm files, whatever. It's "8-bit clean" (and so are Oil strings). [1] https://www.oilshell.org/blog/2017/09/29.html https://www.oilshell.org/blog/2017/09/29.html
- l0b0 6y ago
  > The paragraph right below that mentions that Oil has "read -0", which consumes the find -print0 input. Yeah, I read that ("read -0", for the record, is an excellent idea). QSN for filenames still a bad idea. Your wc, awk, grep etc commands will now have to decode the stream after splitting it. Taking it to the extreme, it's like mixing JSON and XML (or CSV and TSV) because some things work better in one or the other. Right now, even in Bash, I can use newline-terminated tokens and accept that newlines are going to screw up everything (which is fine if I control the filenames), or I can use NUL-terminated tokens and force my code to handle absolutely all possible inputs. No extra parsing library necessary. As long as filenames can contain any character except slash and NUL, terminating them by NUL is a simple solution (at least when using GNU tools). > Shell scripts can use both formats, but the advantage to QSN is that it preserves the line-based nature of shell. Except when your tokens can contain newlines, which we're stuck with for the foreseeable future. > Say I want to use wc -l, awk, or grep. Then the QSN-lines format is better than the NUL format. Can't agree, for the reasons above. > Also, you can transmit a series of say SHA256 checksums in binary format with QSN-lines, or some other binary format. > Even entire JPG files, audio clips, wasm files, whatever. It's "8-bit clean" (and so are Oil strings). That's a completely different problem set. You could use existing simple encodings like base64. "Why use QSN over base64" would be an interesting blog post. In any case, I admire the courage to try to improve the state of the art! I would absolutely love to see something better than POSIX/Bash as the baseline. At this point I suspect we'll need to ignore those and go for something radically different like PowerShell to regain sanity.
  - chubot 6y ago
    A bunch of points: (1) Except when your tokens can contain newlines, which we're stuck with for the foreseeable future. -- not sure what you mean here, because QSN strings are defined not to contain literal newlines. They're escaped like '\n'. The invariant of QSN is: EVERY BYTE STRING, including those with newlines and nuls, can be represented on a single line. I guess I should put this in the documentation. (2) A QSN decoder is very easy to write. For example, here's a ~6 line regex that validates all of QSN: https://github.com/oilshell/oil/blob/master/qsn_/qsn.py#L498 https://github.com/oilshell/oil/blob/master/qsn_/qsn.py#L498 True, you need like ~20 lines of code to decode it, but that's very easy too. You can also make a QSN decoder from a JSON string decoder. It's basically changing your tests and moving a few statements around. (And yes, QSN is a regular language [1]) (3) Oil should grow [2] an awk-like dialect [3] that understands QSN and QTSV. (4) Although you also don't have to decode it for it to be useful. 1. I can use wc -l on a stream of QSN strings 2. If I know the strings are QSN-encoded, I can search for NUL bytes with fgrep '\0' It's basically like the UTF-8 philosophy. ASCII is valid utf-8. QSN lines are lines of text. (5) base64 is bad for humans at the terminal, because it makes everything unreadable. QSN preserves all printable ASCII and unicode. ----- I don't think it's going to solve all problems, but it will solve some. It's there if you need it. In typical Unix style, the solutions will be heterogeneous. You can absolutely use the NUL format in Oil, and it now has support for it. [1] http://www.oilshell.org/blog/2020/07/eggex-theory.html http://www.oilshell.org/blog/2020/07/eggex-theory.html [2] That is, if I get help :) [3] http://www.oilshell.org/blog/tags.html?tag=awk#awk http://www.oilshell.org/blog/tags.html?tag=awk#awk

by l0b0 6y ago

e12e 6y ago

I think having both is a good idea. Sometimes maybe you get your list of path names from a Here-document, for example.

l0b0 6y ago
For sure we need both human- and machine-readable stream formats, but I don't really see how QSN has any advantage in that space over `printf '%q\n'` and NUL-terminated strings, respectively.
- chubot 6y ago
  I actually considered that, but it doesn't work. https://github.com/oilshell/oil/wiki/Shell-Almost-Has-a-JSON-Analogue https://github.com/oilshell/oil/wiki/Shell-Almost-Has-a-JSON... Single quoted strings in shell can't represent arbitrary strings either. And the %q format is actually different from the format ${x@Q} emits. QSN is a well-specified format.

chubot 6y ago

The paragraph right below that mentions that Oil has "read -0", which consumes the find -print0 input. I also link to my Git Log in HTML post [1] from 3 years ago, which is ENTIRELY about the NUL byte solution :) ----- Shell scripts can use both formats, but the advantage to QSN is that it preserves the line-based nature of shell. Say I want to use wc -l, awk, or grep. Then the QSN-lines format is better than the NUL format. Also, you can transmit a series of say SHA256 checksums in binary format with QSN-lines, or some other binary format. Even entire JPG files, audio clips, wasm files, whatever. It's "8-bit clean" (and so are Oil strings). [1] https://www.oilshell.org/blog/2017/09/29.html https://www.oilshell.org/blog/2017/09/29.html

l0b0 6y ago
> The paragraph right below that mentions that Oil has "read -0", which consumes the find -print0 input. Yeah, I read that ("read -0", for the record, is an excellent idea). QSN for filenames still a bad idea. Your wc, awk, grep etc commands will now have to decode the stream after splitting it. Taking it to the extreme, it's like mixing JSON and XML (or CSV and TSV) because some things work better in one or the other. Right now, even in Bash, I can use newline-terminated tokens and accept that newlines are going to screw up everything (which is fine if I control the filenames), or I can use NUL-terminated tokens and force my code to handle absolutely all possible inputs. No extra parsing library necessary. As long as filenames can contain any character except slash and NUL, terminating them by NUL is a simple solution (at least when using GNU tools). > Shell scripts can use both formats, but the advantage to QSN is that it preserves the line-based nature of shell. Except when your tokens can contain newlines, which we're stuck with for the foreseeable future. > Say I want to use wc -l, awk, or grep. Then the QSN-lines format is better than the NUL format. Can't agree, for the reasons above. > Also, you can transmit a series of say SHA256 checksums in binary format with QSN-lines, or some other binary format. > Even entire JPG files, audio clips, wasm files, whatever. It's "8-bit clean" (and so are Oil strings). That's a completely different problem set. You could use existing simple encodings like base64. "Why use QSN over base64" would be an interesting blog post. In any case, I admire the courage to try to improve the state of the art! I would absolutely love to see something better than POSIX/Bash as the baseline. At this point I suspect we'll need to ignore those and go for something radically different like PowerShell to regain sanity.
- chubot 6y ago
  A bunch of points: (1) Except when your tokens can contain newlines, which we're stuck with for the foreseeable future. -- not sure what you mean here, because QSN strings are defined not to contain literal newlines. They're escaped like '\n'. The invariant of QSN is: EVERY BYTE STRING, including those with newlines and nuls, can be represented on a single line. I guess I should put this in the documentation. (2) A QSN decoder is very easy to write. For example, here's a ~6 line regex that validates all of QSN: https://github.com/oilshell/oil/blob/master/qsn_/qsn.py#L498 https://github.com/oilshell/oil/blob/master/qsn_/qsn.py#L498 True, you need like ~20 lines of code to decode it, but that's very easy too. You can also make a QSN decoder from a JSON string decoder. It's basically changing your tests and moving a few statements around. (And yes, QSN is a regular language [1]) (3) Oil should grow [2] an awk-like dialect [3] that understands QSN and QTSV. (4) Although you also don't have to decode it for it to be useful. 1. I can use wc -l on a stream of QSN strings 2. If I know the strings are QSN-encoded, I can search for NUL bytes with fgrep '\0' It's basically like the UTF-8 philosophy. ASCII is valid utf-8. QSN lines are lines of text. (5) base64 is bad for humans at the terminal, because it makes everything unreadable. QSN preserves all printable ASCII and unicode. ----- I don't think it's going to solve all problems, but it will solve some. It's there if you need it. In typical Unix style, the solutions will be heterogeneous. You can absolutely use the NUL format in Oil, and it now has support for it. [1] http://www.oilshell.org/blog/2020/07/eggex-theory.html http://www.oilshell.org/blog/2020/07/eggex-theory.html [2] That is, if I get help :) [3] http://www.oilshell.org/blog/tags.html?tag=awk#awk http://www.oilshell.org/blog/tags.html?tag=awk#awk