4 ms·

> If structured data was embraced we would have developed appropriate tooling to interact with it in the way that we prefer. What kind of tooling might work wi

by yason 10y ago

> If structured data was embraced we would have developed appropriate tooling to interact with it in the way that we prefer. What kind of tooling might work with ad-hoc structured data and still getting all the tools to talk with each other like in Unix? How would it work without having to write input parsing rules, data processing rules, and output formatting/filtering/composing rules for each tool? I suspect that the reason it's not very popular to pass around structured data is that it's damn difficult to make various tools understand arbitrary data streams. Conversely, the power of text is that the tools understand text, only text, and do not understand the context, i.e. the user decides what the text means and how it needs to be processed. Then the tools become generic and the user can apply them in any number of contexts.

tim333 10y ago
>it's not very popular to pass around structured data There's an awful lot of JSON that gets passed around. That seems a reasonable compromise between readable text and some sort of structure.
- yason 10y ago
  So, how do you parse, read, and process JSON without additional instructions of how to interpret the data and what to look for? All right, maybe you do have a common tool that implements a query language so that you can filter out certain paths and objects from the JSON data into a thinner data set with known formatting expected by the next command in the pipeline. Then you need to write that command and you need more instructions, possibly again in another language, to describe what you actually want to do with the data now that you know where to find it. At this point you typically write a separate script file to do this because it's easier to express in full-blown programming language what you want to do with the tree of hashes and lists and values. On the other hand, programs for lines of text are quite short and fit on the command line. I don't see an immediate value in structured data, and especially none that would outweight loss in general applicability and usability in comparison to text based data processing. Don't get me wrong: I would love to see a good prototype or sketch of how such a thing would work, and then try to imagine how I might be able to apply it to similar things for which I use Unix command line today. But I'm sceptical of "how" and also quite sceptical of "why".
  - jslabovitz 10y ago
    I've started using jq (https://stedolan.github.io/jq/ https://stedolan.github.io/jq/) in pipelines to parse/transform JSON output from commands that support it. It has some XPath-like notations for specifying elements within the JSON data tree. It's not perfect, but it's a good start and useful right now.
- falcolas 10y ago
  JSON is an OK serialization format, and a terrible format for outputting human readable data. Let's take, for example, the output of 'ls -al' and imagine it were presented via JSON: The keys 'mode', 'number of links', 'owner', 'group', 'size', 'last modified' would be repeated over and over again, stacked vertically. Mode would remain an arbitrarily formatted string, or (worse) be broken into its own object for every attribute represented by the string. A reasonably populated directory would fill multiple screens with cruft. The formatting of the timestamp in the last modified field would still be an arbitrary string. Comparing two files would require piping through another utility to extract just the appropriate fields and display them unadorned. Sure, it might be moderately easier to consume in another program since you can simply iterate over a list and reference specific keys, but it's not really that hard to iterate over a list of lines and extract a specific field by number. ls -al | awk '{print $5,$9}' vs. ls -al | jq '.[] | [.size,.name|tostring] | join(" ")'
- Slackwise 10y ago
  YAML is a superset of JSON, but is much more human readable: http://yaml.org http://yaml.org This means you can give it JSON when you have JSON, but otherwise use the nicer format when possible.
  - aloisdg 10y ago
    TOML is far better https://github.com/toml-lang/toml https://github.com/toml-lang/toml
emodendroket 10y ago
But the programs don't really just "understand" text; it has to be munged into exactly the format they expect.

by yason 10y ago

tim333 10y ago

>it's not very popular to pass around structured data There's an awful lot of JSON that gets passed around. That seems a reasonable compromise between readable text and some sort of structure.

yason 10y ago
So, how do you parse, read, and process JSON without additional instructions of how to interpret the data and what to look for? All right, maybe you do have a common tool that implements a query language so that you can filter out certain paths and objects from the JSON data into a thinner data set with known formatting expected by the next command in the pipeline. Then you need to write that command and you need more instructions, possibly again in another language, to describe what you actually want to do with the data now that you know where to find it. At this point you typically write a separate script file to do this because it's easier to express in full-blown programming language what you want to do with the tree of hashes and lists and values. On the other hand, programs for lines of text are quite short and fit on the command line. I don't see an immediate value in structured data, and especially none that would outweight loss in general applicability and usability in comparison to text based data processing. Don't get me wrong: I would love to see a good prototype or sketch of how such a thing would work, and then try to imagine how I might be able to apply it to similar things for which I use Unix command line today. But I'm sceptical of "how" and also quite sceptical of "why".
- jslabovitz 10y ago
  I've started using jq (https://stedolan.github.io/jq/ https://stedolan.github.io/jq/) in pipelines to parse/transform JSON output from commands that support it. It has some XPath-like notations for specifying elements within the JSON data tree. It's not perfect, but it's a good start and useful right now.
falcolas 10y ago
JSON is an OK serialization format, and a terrible format for outputting human readable data. Let's take, for example, the output of 'ls -al' and imagine it were presented via JSON: The keys 'mode', 'number of links', 'owner', 'group', 'size', 'last modified' would be repeated over and over again, stacked vertically. Mode would remain an arbitrarily formatted string, or (worse) be broken into its own object for every attribute represented by the string. A reasonably populated directory would fill multiple screens with cruft. The formatting of the timestamp in the last modified field would still be an arbitrary string. Comparing two files would require piping through another utility to extract just the appropriate fields and display them unadorned. Sure, it might be moderately easier to consume in another program since you can simply iterate over a list and reference specific keys, but it's not really that hard to iterate over a list of lines and extract a specific field by number. ls -al | awk '{print $5,$9}' vs. ls -al | jq '.[] | [.size,.name|tostring] | join(" ")'
Slackwise 10y ago
YAML is a superset of JSON, but is much more human readable: http://yaml.org http://yaml.org This means you can give it JSON when you have JSON, but otherwise use the nicer format when possible.
- aloisdg 10y ago
  TOML is far better https://github.com/toml-lang/toml https://github.com/toml-lang/toml

emodendroket 10y ago

But the programs don't really just "understand" text; it has to be munged into exactly the format they expect.