4 ms·

Note that I'm not arguing for structured binary data. Structured text (e.g. s-expressions, JSON, even the-extensible-structured-text-format-which-shall-not-be-

by zeveb 10y ago

Note that I'm not arguing for structured binary data. Structured text (e.g. s-expressions, JSON, even the-extensible-structured-text-format-which-shall-not-be-named) can last just as long. Indeed, S-expressions have existed since the 1950s. There's no particular reason why /etc/passwd couldn't be: ((root nil 0 0 root /root /bin/bash) (daemon nil 1 1 daemon /usr/sbin /usr/sbin/nologin) …) There are any number of similar dialects which could be used, of course, but the principle is obvious.

setrofim_ 10y ago
What would your example achieve? You're making the format more verbose and error-prone (someone might easily forget to match a paren), without imposting any additional structure over what is already implied by line breaks. Though I do agree with your overarching point that some of the formats/outputs could do with a more consistent structure. Perhaps something like YAML would strike a good balance between structure and conciseness/readability...
- junke 10y ago
  The example is not really interesting, because as you said there is already a simple structure. But programming with text becomes really tiresome after a while. For example: git blame -L ${LINE},+1 --porcelain ${FILE} | sed -n '/^author / {s/^author //; p}' I would rather: (name (author (first (git blame file :line line)))) ... where git blame returns a sequence of "blame" data for which I can retrieve the author easily (if you prefer pipes over function composition syntax, use threading macros). Then I don't have to worry about strange characters crashing my scripts randomly. Suppose I forgot to add the "^" symbol in my regexp (I can assume this, since you assume people forget parentheses), there could be situations where I would match too many lines.
- pherq 10y ago
  > What would your example achieve? You're making the format more verbose and error-prone (someone might easily forget to match a paren), without imposting any additional structure over what is already implied by line breaks. That particular example is fairly straightforward (at a simple level, passwd files aren't complex), but being able to express arbitrary nested structure would make various things a lot more straightforward. Line breaks and some sort of tab/colon/what have you work fine if everything has at most two levels of hierarchy, but it starts being painful after that. Missing matched parens are a bit of a specious argument, since many of the random formats for files are fairly strict about what they parse, and the ones that matter (e.g. passwd, sudoers, crontab) are conventionally edited through tools that check the syntax before committing.
  - dagenleg 10y ago
    Maybe you shouldn't use arbitrary nested structures where you can go without?
- zeveb 10y ago
  > You're making the format more verbose and error-prone (someone might easily forget to match a paren), without imposting any additional structure over what is already implied by line breaks. It's already error-prone (as anyone who's ever incorrectly edited /etc/passwd knows). Ultimately, structured data (which is pretty much all data) should be edited with structure editors. Good text formats make it easy to write such structure editors.
junke 10y ago
Beyond text, structured data can give you actual functions or object instances. You can inspect a path object without dealing with escaped slashes, etc. For example, that's how I understand the "capabilities" security model: once you authenticate, you get an object which allows you to perform some tasks (https://en.wikipedia.org/wiki/Capability-based_security https://en.wikipedia.org/wiki/Capability-based_security).
bluetomcat 10y ago
But column-oriented output is still structured and tools like AWK are meant to be a "programmable filter" on it. Reading or outputting deeply nested structures like JSON or S-exps would make it less practical to pipe programs together and instead have big "monolithic filters" with lots of options.
- pwdisswordfish 10y ago
  That just illustrates the limitations of the pipe abstraction, as something only able to send sequentially-read unstructured data. Also, jq, jshon and similar tools are a thing.
- fnj 10y ago
  Parsing columns of text with awk or any other text tools is vexingly problematic. How do you even define "column"? What if columns contain whitespace? How do you differentiate that from the whitespace between columns?

by zeveb 10y ago

setrofim_ 10y ago

What would your example achieve? You're making the format more verbose and error-prone (someone might easily forget to match a paren), without imposting any additional structure over what is already implied by line breaks. Though I do agree with your overarching point that some of the formats/outputs could do with a more consistent structure. Perhaps something like YAML would strike a good balance between structure and conciseness/readability...

junke 10y ago
The example is not really interesting, because as you said there is already a simple structure. But programming with text becomes really tiresome after a while. For example: git blame -L ${LINE},+1 --porcelain ${FILE} | sed -n '/^author / {s/^author //; p}' I would rather: (name (author (first (git blame file :line line)))) ... where git blame returns a sequence of "blame" data for which I can retrieve the author easily (if you prefer pipes over function composition syntax, use threading macros). Then I don't have to worry about strange characters crashing my scripts randomly. Suppose I forgot to add the "^" symbol in my regexp (I can assume this, since you assume people forget parentheses), there could be situations where I would match too many lines.
pherq 10y ago
> What would your example achieve? You're making the format more verbose and error-prone (someone might easily forget to match a paren), without imposting any additional structure over what is already implied by line breaks. That particular example is fairly straightforward (at a simple level, passwd files aren't complex), but being able to express arbitrary nested structure would make various things a lot more straightforward. Line breaks and some sort of tab/colon/what have you work fine if everything has at most two levels of hierarchy, but it starts being painful after that. Missing matched parens are a bit of a specious argument, since many of the random formats for files are fairly strict about what they parse, and the ones that matter (e.g. passwd, sudoers, crontab) are conventionally edited through tools that check the syntax before committing.
- dagenleg 10y ago
  Maybe you shouldn't use arbitrary nested structures where you can go without?
zeveb 10y ago
> You're making the format more verbose and error-prone (someone might easily forget to match a paren), without imposting any additional structure over what is already implied by line breaks. It's already error-prone (as anyone who's ever incorrectly edited /etc/passwd knows). Ultimately, structured data (which is pretty much all data) should be edited with structure editors. Good text formats make it easy to write such structure editors.

junke 10y ago

Beyond text, structured data can give you actual functions or object instances. You can inspect a path object without dealing with escaped slashes, etc. For example, that's how I understand the "capabilities" security model: once you authenticate, you get an object which allows you to perform some tasks (https://en.wikipedia.org/wiki/Capability-based_security https://en.wikipedia.org/wiki/Capability-based_security).

bluetomcat 10y ago

But column-oriented output is still structured and tools like AWK are meant to be a "programmable filter" on it. Reading or outputting deeply nested structures like JSON or S-exps would make it less practical to pipe programs together and instead have big "monolithic filters" with lots of options.

pwdisswordfish 10y ago
That just illustrates the limitations of the pipe abstraction, as something only able to send sequentially-read unstructured data. Also, jq, jshon and similar tools are a thing.
fnj 10y ago
Parsing columns of text with awk or any other text tools is vexingly problematic. How do you even define "column"? What if columns contain whitespace? How do you differentiate that from the whitespace between columns?