8 ms·

My structured objects aren't your structured objects. In that use case why not use a dedicated program (like say ae python/js/clojure interpreter) to handle mor

by zdkl 9y ago

My structured objects aren't your structured objects. In that use case why not use a dedicated program (like say ae python/js/clojure interpreter) to handle more complex pipes and keep the shell level primitves... Well, primitive. Text is compatible with all systems, past and present, and can be used to model more complex objects. Let's not add features to the core on a "why not" please.

vog 9y ago
The so-called "plain text formats" are also just representations of complex objects. Often these formats are ad-hoc and hard to parse correctly. Multiple text files in a file system hierarchy (e.g. /etc) are also just nested structures. So in principle, treating those as complex object structures is the right way to go. Also, I believe this is the idea behind D-Bus and similar modern Unix developments. However, what's hard is to provide good tooling with a simple syntax that is simple to understand: * Windows Registry and Power Shell show how not to do it. * The XML toolchain demonstrate that any unnecessary complexity in the meta-structure will haunt you through every bit of the toolchain (querying, validation/schema, etc.). * "jq" goes somewhat into the right direction for JSON files, but still hasn't found wide adoption. This appears to be a really hard design issue. Also, while deep hierarchies are easier to process by scripts, a human overview is mostly achieved by advances searching and tags rather than hierarchies. A shell needs to accomodate for both, but maybe we really just need better command line tools for simple hierarchial processing of arbitrary text file formats (ad-hoc configs, CSV, INI, JSON, YAML, XML, etc.).
- zdkl 9y ago
  > in principle, treating those as complex object structures is the right way to go I do not see this as a given. It's a matter of abstraction vs performance tradeoff and that is highly subjective. Unless you pioneer a new standard form for complex object notation this'll just end up back to a format flamewar. (And if we go that way, I'd argue for s-exprs or -dare I say it- xml)
- int_19h 9y ago
  Powershell does two things right: it uses structured output, and it separates producing the data from rendering the data. It also does one thing wrong: it uses objects (i.e. the stuff that carries behavior, not just state). This ties it to a particular object model, and the framework that supports that model. What's really needed is something simple that's data-centric, like JSON, but with a complete toolchain to define schemas and perform transformations, like XML (but without the warts and overengineering).
  - still_grokking 9y ago
    > What's really needed is something simple that's data-centric, like JSON, but with a complete toolchain to define schemas and perform transformations, like XML (but without the warts and overengineering). What would it be? Is there a real alternative to XML with those features out there? I don't think so. When you would want to have the features of XML and would design it from scratch I'm quite sure it would have the complexity of XML again. Usually implementing some functionality yields every time the same level of complexity regardless of how you implement it (given that none of the implementations isn't out right stupid of curse).
    - int_19h 9y ago
      > When you would want to have the features of XML and would design it from scratch I'm quite sure it would have the complexity of XML again. I don't think so. The problem with the XML stack is that it has been designed with some very "enterprisey" (for the lack of better term) scenarios in mind - stuff like SOAP. Consequently, it was all design by committee in the worst possible sense of the word, and it shows. To see what I mean, take a look at XML Schema W3C specs. That's probably the worst part of it, so it should be readily apparent what I mean: https://www.w3.org/TR/xmlschema11-1/ https://www.w3.org/TR/xmlschema11-1/ https://www.w3.org/TR/xmlschema11-2/ https://www.w3.org/TR/xmlschema11-2/ The other problem with XML is that it's rooted in SGML, and inherited a lot of its syntax and semantics, which were designed for a completely different use case - marking up documents. Consequently, the syntax is overly verbose, and some features are inconsistent for other scenarios - for example, if you use XML to describe structured data, when do you use attributes, and when do you use child elements? Don't forget that attributes are semantically unordered in XDM, while elements are ordered, but also that attributes cannot contain anything but scalar values and arrays thereof. Oh, and then don't forget all the legacy stuff like DTD, which is mostly redundant in the face of XML Schema and XInclude, except it's still a required part of the spec. I guess the TL;DR version of it is that XML today is kinda like Java - it was there for too long, including periods when our ideas of best practices were radically different, and all that was enshrined in the design, and then fossilized in the name of backwards compatibility. One important takeaway from XML - why it was so successful, IMO - is that having a coherent, a tightly bound spec stack is a good thing. For example, with XML, when someone is talking about schemas, you can pretty much assume it's XML Schema by default (yes, there's also RELAX NG, but I think calling it schema is a misnomer, because it doesn't delve much into semantics of what it describes - it's more of a grammar definition language for XML). To transfer XML, you use XSLT. To query it, you use XPath or XQuery (which is a strict superset). And so on. With JSON, there's no such certainty. The other thing that the XML stack didn't quite see fully through, but showed that it could be a nice thing, is its homoiconicity: e.g. XML Schema and XSLT being XML. Less so with XPath and XQuery, but there they had at least defined a canonical XML representation for it, which gives you most of the same advantages. Unfortunately, with XML it was just as often a curse as it was a blessing, because of how verbose and sometimes awkward its syntax is - anyone who wrote large amounts of XSLT especially knows what I'm talking about. On the other hand, at least XML had comments, unlike JSON! Hey, maybe that's actually the test case? A data representation language must be concise enough, powerful enough, and flexible enough to make it possible to use it to define its own schema and transformations, without it being a painful experience, while also being simple enough that a single person can write a parser for it in a reasonable amount of time.
- vidarh 9y ago
  I lean towards wanting all my tools to support at least one common structured format (and JSON generally would seem the best supported) these days. It'd be great if there was a "standard" environment variable to toggle JSON input/output on for apps. If you did that, then e.g. a "structured" shell could just default to turning that on, and try detecting JSON on output and offering according additional functionality. That'd be a tolerable graduated approach where you get benefits even if not every tool support it. With "jq", I'm already increasingly leaning on JSON for this type of role. It'd need to be very non-invasive, though, or all kinds of things are likely to break.
  - codesnik 9y ago
    jq is awesome. "json lines" data format (where each record is json without literal newlines, and newline separates records) nicely upgrades unix pipes to structured output, so you can mix old unix tools, like head, tail etc with jq. maybe what's needed is some set of simple json wrappers around core utils to increase adoption.
    - subnixr 9y ago
      > "json lines" data format (where each record is json without literal newlines, and newline separates records) Good point, but jq handles this already. If the payload is an array of object, simply `jq -c '.[]'` to get an object per line
  - vog 9y ago
    Nice idea! Maybe this issue is similar to that of command completion, but could also result in a similar mess. Every command has its "-h" or "--help", showing the syntax, available options and so on. But since even that is not really standardized, separate command-line completion rules are written for every command. And for every shell.
    - vidarh 9y ago
      I still miss AmigaOS, where command options parsing was largely standardised ca. 1987 via ReadArgs() (barring the occasional poor/direct port from other OS's) [1]. It wasn't perfect (it only provided a very basic usage summary), but it was typed, and it was there. [1] http://www.pjhutchison.org/tutorial/cmdline_arrgs.html http://www.pjhutchison.org/tutorial/cmdline_arrgs.html
- agumonkey 9y ago
  sexp ? ps: I kinda like powershell (the few hours I toyed in it
  - eecc 9y ago
    S-expressions
- still_grokking 9y ago
  For XML there are xml-coreutils[1]. [1] http://www.lbreyer.com/xml-coreutils.html http://www.lbreyer.com/xml-coreutils.html
- merb 9y ago
  btw. one of my favorite config is HOCON. it allows ini style and json style.https://github.com/typesafehub/config/blob/master/HOCON.md https://github.com/typesafehub/config/blob/master/HOCON.md
- fnord123 9y ago
  >A shell needs to accomodate for both, but maybe we really just need better command line tools for simple hierarchial processing of arbitrary text file formats (ad-hoc configs, CSV, INI, JSON, YAML, XML, etc.). Start with record based streams first. Text streams requires [buggy] parsing to be implemented everywhere. It should be possible to have escaped record formats that allow the right side of the pipe to use AWK style $1, $2, $3, etc. After removing the need to parse the fields, the next priority, imo, would be to introduce integral types so the actual data itself doesn't need to be parsed. u32 on the LHS can just be 4 bytes and then read out on the RHS as 4 bytes. This could save a lot of overhead when processing large files. Only then would I want to get into hierarchies, product types, sum types, etc.
  - moe 9y ago
    This is dead on. Half of all parsing work consists of splitting things into records, by lines, delimiters or whitespace. That's where the great escaping headache begins. In a better shell the following command would Just Work™: > find | rm %path/%filename
- mos_basik 9y ago
  On phone so unsure if it's been mentioned in the thread already, but I've been enjoying learning a js library JSONata [0] after finding it included in Node-RED. I remember finding jq a few weeks ago and thinking "wow, this will probably come in handy for a specific kind of situation" and filing it mentally for later use, but I haven't used it for anything yet so I'm not super familiar with the extent of it's features. I have been using a lot of JSONata one liners to replace several procedural functions that were doing data transforms on JSON objects, and I'm very impressed. It's a querying library but it's Turing complete - it has lambdas, it can save references to data and functions as variables, etc. It also seems relatively new/unknown; I've found hardly any blogs or forums mentioning it. The developer is active - he fixed a bug report I submitted in less then a day. I'd love to have that kind of functionality in a CLI tool. Maybe jq is equally powerful, I don't know. I haven't had time to run any performance analysis on JSONata and haven't found anyone else online who's done any yet. I'm very curious how its queries compare to efficiently implemented procedural approaches. 0: http://jsonata.org/ http://jsonata.org/
  - elcritch 9y ago
    Thanks! Will have to try out `jsonata`, as `jq` never clicked for me (too complex). Alternately I've been using a node.js program `json` [1] which has a basic but straightforward cli which covers 95% of my daily needs. I believe I found after trying to figure out the tool Joyent.com uses in there SmartOS sysadmin setup. Not sure if it's the exact same tool, but it's very similar. As an example, basic manipulation is just: `echo '{"age":10}' | json -e 'this.age++' #=> {"age": 11}` 1: https://github.com/trentm/json https://github.com/trentm/json
- skywhopper 9y ago
  The brilliant thing about the status quo is that it already does accomodate both line-based and structured pipelines. There's no reason we should expect to use the same set of commands for both purposes. In fact, we most definitely shouldn't. A tool like jq is a great example of this: it works great for JSON, and it doesn't attempt to do more. When we want to work with JSON in a pipeline, we pull out jq. When we want to work with XML, we pull out xmlstarlet or the like. When we want to work with delimited columnar data, we can use awk or cut. I'm failing to see what's missing from an architectural point of view.
Sir_Cmpwn 9y ago
This hits the mark right on the head. Instead of ruining the shell with unnecessary complexity because it doesn't fit every use case, how about knowing when not to use the shell?
v-yadli 9y ago
Likewise, your text objects aren't my text objects. For a fair comparison I think we should first assume compatibility for both unstructured, and structural data. Otherwise I would challenge that your bash script is not my bash script. The sole point of structural data is to turn the storage into a formal system, and enable more math operations on the entries, which align to the desired semantics. Python/js/clojure scripts extract some structural aspects from text, and then the story becomes the same as structural data processing.

by zdkl 9y ago

vog 9y ago

The so-called "plain text formats" are also just representations of complex objects. Often these formats are ad-hoc and hard to parse correctly. Multiple text files in a file system hierarchy (e.g. /etc) are also just nested structures. So in principle, treating those as complex object structures is the right way to go. Also, I believe this is the idea behind D-Bus and similar modern Unix developments. However, what's hard is to provide good tooling with a simple syntax that is simple to understand: * Windows Registry and Power Shell show how not to do it. * The XML toolchain demonstrate that any unnecessary complexity in the meta-structure will haunt you through every bit of the toolchain (querying, validation/schema, etc.). * "jq" goes somewhat into the right direction for JSON files, but still hasn't found wide adoption. This appears to be a really hard design issue. Also, while deep hierarchies are easier to process by scripts, a human overview is mostly achieved by advances searching and tags rather than hierarchies. A shell needs to accomodate for both, but maybe we really just need better command line tools for simple hierarchial processing of arbitrary text file formats (ad-hoc configs, CSV, INI, JSON, YAML, XML, etc.).

zdkl 9y ago
> in principle, treating those as complex object structures is the right way to go I do not see this as a given. It's a matter of abstraction vs performance tradeoff and that is highly subjective. Unless you pioneer a new standard form for complex object notation this'll just end up back to a format flamewar. (And if we go that way, I'd argue for s-exprs or -dare I say it- xml)
int_19h 9y ago
Powershell does two things right: it uses structured output, and it separates producing the data from rendering the data. It also does one thing wrong: it uses objects (i.e. the stuff that carries behavior, not just state). This ties it to a particular object model, and the framework that supports that model. What's really needed is something simple that's data-centric, like JSON, but with a complete toolchain to define schemas and perform transformations, like XML (but without the warts and overengineering).
- still_grokking 9y ago
  > What's really needed is something simple that's data-centric, like JSON, but with a complete toolchain to define schemas and perform transformations, like XML (but without the warts and overengineering). What would it be? Is there a real alternative to XML with those features out there? I don't think so. When you would want to have the features of XML and would design it from scratch I'm quite sure it would have the complexity of XML again. Usually implementing some functionality yields every time the same level of complexity regardless of how you implement it (given that none of the implementations isn't out right stupid of curse).
  - int_19h 9y ago
    > When you would want to have the features of XML and would design it from scratch I'm quite sure it would have the complexity of XML again. I don't think so. The problem with the XML stack is that it has been designed with some very "enterprisey" (for the lack of better term) scenarios in mind - stuff like SOAP. Consequently, it was all design by committee in the worst possible sense of the word, and it shows. To see what I mean, take a look at XML Schema W3C specs. That's probably the worst part of it, so it should be readily apparent what I mean: https://www.w3.org/TR/xmlschema11-1/ https://www.w3.org/TR/xmlschema11-1/ https://www.w3.org/TR/xmlschema11-2/ https://www.w3.org/TR/xmlschema11-2/ The other problem with XML is that it's rooted in SGML, and inherited a lot of its syntax and semantics, which were designed for a completely different use case - marking up documents. Consequently, the syntax is overly verbose, and some features are inconsistent for other scenarios - for example, if you use XML to describe structured data, when do you use attributes, and when do you use child elements? Don't forget that attributes are semantically unordered in XDM, while elements are ordered, but also that attributes cannot contain anything but scalar values and arrays thereof. Oh, and then don't forget all the legacy stuff like DTD, which is mostly redundant in the face of XML Schema and XInclude, except it's still a required part of the spec. I guess the TL;DR version of it is that XML today is kinda like Java - it was there for too long, including periods when our ideas of best practices were radically different, and all that was enshrined in the design, and then fossilized in the name of backwards compatibility. One important takeaway from XML - why it was so successful, IMO - is that having a coherent, a tightly bound spec stack is a good thing. For example, with XML, when someone is talking about schemas, you can pretty much assume it's XML Schema by default (yes, there's also RELAX NG, but I think calling it schema is a misnomer, because it doesn't delve much into semantics of what it describes - it's more of a grammar definition language for XML). To transfer XML, you use XSLT. To query it, you use XPath or XQuery (which is a strict superset). And so on. With JSON, there's no such certainty. The other thing that the XML stack didn't quite see fully through, but showed that it could be a nice thing, is its homoiconicity: e.g. XML Schema and XSLT being XML. Less so with XPath and XQuery, but there they had at least defined a canonical XML representation for it, which gives you most of the same advantages. Unfortunately, with XML it was just as often a curse as it was a blessing, because of how verbose and sometimes awkward its syntax is - anyone who wrote large amounts of XSLT especially knows what I'm talking about. On the other hand, at least XML had comments, unlike JSON! Hey, maybe that's actually the test case? A data representation language must be concise enough, powerful enough, and flexible enough to make it possible to use it to define its own schema and transformations, without it being a painful experience, while also being simple enough that a single person can write a parser for it in a reasonable amount of time.
vidarh 9y ago
I lean towards wanting all my tools to support at least one common structured format (and JSON generally would seem the best supported) these days. It'd be great if there was a "standard" environment variable to toggle JSON input/output on for apps. If you did that, then e.g. a "structured" shell could just default to turning that on, and try detecting JSON on output and offering according additional functionality. That'd be a tolerable graduated approach where you get benefits even if not every tool support it. With "jq", I'm already increasingly leaning on JSON for this type of role. It'd need to be very non-invasive, though, or all kinds of things are likely to break.
- codesnik 9y ago
  jq is awesome. "json lines" data format (where each record is json without literal newlines, and newline separates records) nicely upgrades unix pipes to structured output, so you can mix old unix tools, like head, tail etc with jq. maybe what's needed is some set of simple json wrappers around core utils to increase adoption.
  - subnixr 9y ago
    > "json lines" data format (where each record is json without literal newlines, and newline separates records) Good point, but jq handles this already. If the payload is an array of object, simply `jq -c '.[]'` to get an object per line
- vog 9y ago
  Nice idea! Maybe this issue is similar to that of command completion, but could also result in a similar mess. Every command has its "-h" or "--help", showing the syntax, available options and so on. But since even that is not really standardized, separate command-line completion rules are written for every command. And for every shell.
  - vidarh 9y ago
    I still miss AmigaOS, where command options parsing was largely standardised ca. 1987 via ReadArgs() (barring the occasional poor/direct port from other OS's) [1]. It wasn't perfect (it only provided a very basic usage summary), but it was typed, and it was there. [1] http://www.pjhutchison.org/tutorial/cmdline_arrgs.html http://www.pjhutchison.org/tutorial/cmdline_arrgs.html
agumonkey 9y ago
sexp ? ps: I kinda like powershell (the few hours I toyed in it
- eecc 9y ago
  S-expressions
still_grokking 9y ago
For XML there are xml-coreutils[1]. [1] http://www.lbreyer.com/xml-coreutils.html http://www.lbreyer.com/xml-coreutils.html
merb 9y ago
btw. one of my favorite config is HOCON. it allows ini style and json style.https://github.com/typesafehub/config/blob/master/HOCON.md https://github.com/typesafehub/config/blob/master/HOCON.md
fnord123 9y ago
>A shell needs to accomodate for both, but maybe we really just need better command line tools for simple hierarchial processing of arbitrary text file formats (ad-hoc configs, CSV, INI, JSON, YAML, XML, etc.). Start with record based streams first. Text streams requires [buggy] parsing to be implemented everywhere. It should be possible to have escaped record formats that allow the right side of the pipe to use AWK style $1, $2, $3, etc. After removing the need to parse the fields, the next priority, imo, would be to introduce integral types so the actual data itself doesn't need to be parsed. u32 on the LHS can just be 4 bytes and then read out on the RHS as 4 bytes. This could save a lot of overhead when processing large files. Only then would I want to get into hierarchies, product types, sum types, etc.
- moe 9y ago
  This is dead on. Half of all parsing work consists of splitting things into records, by lines, delimiters or whitespace. That's where the great escaping headache begins. In a better shell the following command would Just Work™: > find | rm %path/%filename
mos_basik 9y ago
On phone so unsure if it's been mentioned in the thread already, but I've been enjoying learning a js library JSONata [0] after finding it included in Node-RED. I remember finding jq a few weeks ago and thinking "wow, this will probably come in handy for a specific kind of situation" and filing it mentally for later use, but I haven't used it for anything yet so I'm not super familiar with the extent of it's features. I have been using a lot of JSONata one liners to replace several procedural functions that were doing data transforms on JSON objects, and I'm very impressed. It's a querying library but it's Turing complete - it has lambdas, it can save references to data and functions as variables, etc. It also seems relatively new/unknown; I've found hardly any blogs or forums mentioning it. The developer is active - he fixed a bug report I submitted in less then a day. I'd love to have that kind of functionality in a CLI tool. Maybe jq is equally powerful, I don't know. I haven't had time to run any performance analysis on JSONata and haven't found anyone else online who's done any yet. I'm very curious how its queries compare to efficiently implemented procedural approaches. 0: http://jsonata.org/ http://jsonata.org/
- elcritch 9y ago
  Thanks! Will have to try out `jsonata`, as `jq` never clicked for me (too complex). Alternately I've been using a node.js program `json` [1] which has a basic but straightforward cli which covers 95% of my daily needs. I believe I found after trying to figure out the tool Joyent.com uses in there SmartOS sysadmin setup. Not sure if it's the exact same tool, but it's very similar. As an example, basic manipulation is just: `echo '{"age":10}' | json -e 'this.age++' #=> {"age": 11}` 1: https://github.com/trentm/json https://github.com/trentm/json
skywhopper 9y ago
The brilliant thing about the status quo is that it already does accomodate both line-based and structured pipelines. There's no reason we should expect to use the same set of commands for both purposes. In fact, we most definitely shouldn't. A tool like jq is a great example of this: it works great for JSON, and it doesn't attempt to do more. When we want to work with JSON in a pipeline, we pull out jq. When we want to work with XML, we pull out xmlstarlet or the like. When we want to work with delimited columnar data, we can use awk or cut. I'm failing to see what's missing from an architectural point of view.

Sir_Cmpwn 9y ago

This hits the mark right on the head. Instead of ruining the shell with unnecessary complexity because it doesn't fit every use case, how about knowing when not to use the shell?

v-yadli 9y ago

Likewise, your text objects aren't my text objects. For a fair comparison I think we should first assume compatibility for both unstructured, and structural data. Otherwise I would challenge that your bash script is not my bash script. The sole point of structural data is to turn the storage into a formal system, and enable more math operations on the entries, which align to the desired semantics. Python/js/clojure scripts extract some structural aspects from text, and then the story becomes the same as structural data processing.