6 ms·

> The biggest disadvantage of the shell is that, by exchanging data using text, you lose opportunities to check for errors in the output. That's pretty bad, bu

> The biggest disadvantage of the shell is that, by exchanging data using text, you lose opportunities to check for errors in the output. That's pretty bad, but isn't the complete lack of support for structured data an even bigger one? After all, if you can't even represent your data, then throwing errors is kind of moot.

chubot 2y ago
Oils/YSH has structured data and JSON! This was finished earlier this year Garbage Collection Makes YSH Different (than POSIX shell, awk, cmake, make, ...) - https://www.oilshell.org/blog/2024/09/gc.html https://www.oilshell.org/blog/2024/09/gc.html You need GC for arbitrary recursive data structures, and traditionally Unix didn't have those languages. Lisp was the first GC language, and pre-dated Unix, and then Java made GC popular, and Java was not integrated with Unix (it wanted to be its own OS) ---- So now you can do # create some JSON ysh-0.23.0$ echo '{"foo":[1,2,3]}' > x.json # read it into the variable x -- you will get a syntax error if it's malformed ysh-0.23.0$ json read (&x) < x.json # pretty print the resulting data structure, = comes from Lua ysh-0.23.0$ = x (Dict) {foo: [1, 2, 3]} # use it in some computation ysh-0.23.0$ var y = x.foo[1] ysh ysh-0.23.0$ = y (Int) 2
- throw10920 2y ago
  Structured shells are neat and I love them, but the Unix philosophy is explicitly built around plain text - the "structured" part of structured shells isn't Unixy.
  - chubot 2y ago
    It's not either-or -- I'd think of it as LAYERED - JSON denotes a data structure, but it is also text - you can use grep and sed on it, or jq - TSV denotes a data structure [1], but it is also text - you can use grep on it, or xsv or recutils or ... (on the other hand, protobuf or Apache arrow not text, and you can't use grep on them directly. But that doesn't mean they're bad or not useful, just not interoperable in a Unix style. The way you use them with Unix is to "project" onto text) etc. That is the layered philosophy of Oils, as shown in this diagram - https://www.oilshell.org/blog/2022/02/diagrams.html#bytes-flat-files https://www.oilshell.org/blog/2022/02/diagrams.html#bytes-fl... IMO this is significantly different and better than say PowerShell, which is all about objects inside a VM what I call "interior vs. exterior" processes and files/JSON/TSV are "exterior", while cmdlets and objects inside a .NET VM are "interior" Oils Is Exterior-First (Code, Text, and Structured Data) - https://www.oilshell.org/blog/2023/06/ysh-design.html https://www.oilshell.org/blog/2023/06/ysh-design.html --- [1] Oils fixes some flaws in the common text formats with "J8 Notation", an optional and compatible upgrade. Both JSON and TSV have some "text-y" quirks, like UTF-16 legacy and inablity to represent tabs So J8 Notation cleans up those rough edges, and makes them more like "real" data structures with clean / composable semantics
    - throw10920 2y ago
      > It's not either-or -- I'd think of it as LAYERED You can almost always represent one format inside of another. I can put JSON in a text system, or I can hold text in Protobuf. That doesn't mean that the systems are equivalently powerful, or that a text-oriented system supports structured data. > on the other hand, protobuf or Apache arrow not text, and you can't use grep on them directly This is kind of tautological, because of course you can't grep through them, because Unix and grep don't understand structure, because they're built around text. > IMO this is significantly different and better than say PowerShell, which is all about objects inside a VM Why?
enriquto 2y ago
> if you can't even represent your data any data can be represented as text
- throw10920 2y ago
  Only in the least useful, most pedantic sense of the word. I can "represent" image data as base64 text in my terminal and yet have no tools to meaningfully manipulate it with. This is a gotcha with no interesting thought behind it.
  - enriquto 2y ago
    Your example is particularly funny, because one can easily undecode the base64 image, convert it to sixel so that it's displayed in the terminal verbatim, all in half a line of shell pipes. If the image is large, you can also pipe it to an interactive visualizer to look at it. Or, you just "cat" your base64 image as is, together with some recently echoed mail headers and pipe it to mail as an attachement. The tools to do that are standard and come with all unix distributions. The greatness of unix is that none of the intermediary programs need to know that they are dealing with an image. This kind of simplicity is impossible if the type of your data must be known by each intermediary program. "Structured data" is often an unnecessary and cumbersome overkill, especially for simple tasks.
    - throw10920 2y ago
      > Your example is particularly funny, because one can easily undecode the base64 image I don't see how that's relevant - the fact that a very small number of specific terminal programs can decode limited data formats and communicate over terminal control sequences that only work for that data format doesn't seem to matter discussion of whether the shell or OS supports structured data in general or whether it gives you the tools to manipulate that data. Moreover, this is actually an example that further proves my point: you can't use classic Unix tools to process that image data. You can't use cut to select rows or columns, head or tail to get parts of an image, grep to look for pixels - of course you can't, because it's not text, and one of the core tenets of the Unix philosophy is that everything is plain text. Now, this wasn't a great example to begin with, because media is somewhat of a special case (and it's somewhat sequential data, as opposed to hierarchical). Let's take something like an array of transactions for a personal budget in JSON form, where each transaction has fields like "type" (e.g. "credit", "debit"), "amount", and "other party", and "other party" further has "type" (e.g. "business", "person", "bank") and "name" fields. The Unix philosophy does not allow you to directly represent this data - there is no way to represent key-value pairs or nested data and then do things like extract fields or filter on values. Your primitives are lines and characters, by definition. > This kind of simplicity is impossible if the type of your data must be known by each intermediary program. I think you're misunderstanding what "structured data" is, because what you just described isn't a property of structured data. I can write a "filter" program that takes an arbitrary test function and applies it to a blob of data, and that program needs zero information about the data, other than that it's an array. Or, it's possible that you're just not familiar with the concept of abstraction, where programming systems can expose an interface that requires other systems to not understand their internals. > The greatness of unix is that none of the intermediary programs need to know that they are dealing with an image. This has nothing to do with Unix. It's trivial to conceive of a shell or OS where programs pass around structured data and are able to productively operate on it without needing to know the full structure of that data. > "Structured data" is often an unnecessary and cumbersome overkill What does this even mean? Data is structured by definition. If something doesn't have internal structure to it, it is not data. The fact that Unix decides to be willfully ignorant of that structure and force you to ignore it doesn't mean that it doesn't exist. I would hope you didn't mean "representing structured data as such is inconvenient" because that would be a very ignorant statement to make that conflates tooling with data format and that has zero empirical evidence behind it.

chubot 2y ago

Oils/YSH has structured data and JSON! This was finished earlier this year Garbage Collection Makes YSH Different (than POSIX shell, awk, cmake, make, ...) - https://www.oilshell.org/blog/2024/09/gc.html https://www.oilshell.org/blog/2024/09/gc.html You need GC for arbitrary recursive data structures, and traditionally Unix didn't have those languages. Lisp was the first GC language, and pre-dated Unix, and then Java made GC popular, and Java was not integrated with Unix (it wanted to be its own OS) ---- So now you can do # create some JSON ysh-0.23.0$ echo '{"foo":[1,2,3]}' > x.json # read it into the variable x -- you will get a syntax error if it's malformed ysh-0.23.0$ json read (&x) < x.json # pretty print the resulting data structure, = comes from Lua ysh-0.23.0$ = x (Dict) {foo: [1, 2, 3]} # use it in some computation ysh-0.23.0$ var y = x.foo[1] ysh ysh-0.23.0$ = y (Int) 2

throw10920 2y ago
Structured shells are neat and I love them, but the Unix philosophy is explicitly built around plain text - the "structured" part of structured shells isn't Unixy.
- chubot 2y ago
  It's not either-or -- I'd think of it as LAYERED - JSON denotes a data structure, but it is also text - you can use grep and sed on it, or jq - TSV denotes a data structure [1], but it is also text - you can use grep on it, or xsv or recutils or ... (on the other hand, protobuf or Apache arrow not text, and you can't use grep on them directly. But that doesn't mean they're bad or not useful, just not interoperable in a Unix style. The way you use them with Unix is to "project" onto text) etc. That is the layered philosophy of Oils, as shown in this diagram - https://www.oilshell.org/blog/2022/02/diagrams.html#bytes-flat-files https://www.oilshell.org/blog/2022/02/diagrams.html#bytes-fl... IMO this is significantly different and better than say PowerShell, which is all about objects inside a VM what I call "interior vs. exterior" processes and files/JSON/TSV are "exterior", while cmdlets and objects inside a .NET VM are "interior" Oils Is Exterior-First (Code, Text, and Structured Data) - https://www.oilshell.org/blog/2023/06/ysh-design.html https://www.oilshell.org/blog/2023/06/ysh-design.html --- [1] Oils fixes some flaws in the common text formats with "J8 Notation", an optional and compatible upgrade. Both JSON and TSV have some "text-y" quirks, like UTF-16 legacy and inablity to represent tabs So J8 Notation cleans up those rough edges, and makes them more like "real" data structures with clean / composable semantics
  - throw10920 2y ago
    > It's not either-or -- I'd think of it as LAYERED You can almost always represent one format inside of another. I can put JSON in a text system, or I can hold text in Protobuf. That doesn't mean that the systems are equivalently powerful, or that a text-oriented system supports structured data. > on the other hand, protobuf or Apache arrow not text, and you can't use grep on them directly This is kind of tautological, because of course you can't grep through them, because Unix and grep don't understand structure, because they're built around text. > IMO this is significantly different and better than say PowerShell, which is all about objects inside a VM Why?

enriquto 2y ago

> if you can't even represent your data any data can be represented as text

throw10920 2y ago
Only in the least useful, most pedantic sense of the word. I can "represent" image data as base64 text in my terminal and yet have no tools to meaningfully manipulate it with. This is a gotcha with no interesting thought behind it.
- enriquto 2y ago
  Your example is particularly funny, because one can easily undecode the base64 image, convert it to sixel so that it's displayed in the terminal verbatim, all in half a line of shell pipes. If the image is large, you can also pipe it to an interactive visualizer to look at it. Or, you just "cat" your base64 image as is, together with some recently echoed mail headers and pipe it to mail as an attachement. The tools to do that are standard and come with all unix distributions. The greatness of unix is that none of the intermediary programs need to know that they are dealing with an image. This kind of simplicity is impossible if the type of your data must be known by each intermediary program. "Structured data" is often an unnecessary and cumbersome overkill, especially for simple tasks.
  - throw10920 2y ago
    > Your example is particularly funny, because one can easily undecode the base64 image I don't see how that's relevant - the fact that a very small number of specific terminal programs can decode limited data formats and communicate over terminal control sequences that only work for that data format doesn't seem to matter discussion of whether the shell or OS supports structured data in general or whether it gives you the tools to manipulate that data. Moreover, this is actually an example that further proves my point: you can't use classic Unix tools to process that image data. You can't use cut to select rows or columns, head or tail to get parts of an image, grep to look for pixels - of course you can't, because it's not text, and one of the core tenets of the Unix philosophy is that everything is plain text. Now, this wasn't a great example to begin with, because media is somewhat of a special case (and it's somewhat sequential data, as opposed to hierarchical). Let's take something like an array of transactions for a personal budget in JSON form, where each transaction has fields like "type" (e.g. "credit", "debit"), "amount", and "other party", and "other party" further has "type" (e.g. "business", "person", "bank") and "name" fields. The Unix philosophy does not allow you to directly represent this data - there is no way to represent key-value pairs or nested data and then do things like extract fields or filter on values. Your primitives are lines and characters, by definition. > This kind of simplicity is impossible if the type of your data must be known by each intermediary program. I think you're misunderstanding what "structured data" is, because what you just described isn't a property of structured data. I can write a "filter" program that takes an arbitrary test function and applies it to a blob of data, and that program needs zero information about the data, other than that it's an array. Or, it's possible that you're just not familiar with the concept of abstraction, where programming systems can expose an interface that requires other systems to not understand their internals. > The greatness of unix is that none of the intermediary programs need to know that they are dealing with an image. This has nothing to do with Unix. It's trivial to conceive of a shell or OS where programs pass around structured data and are able to productively operate on it without needing to know the full structure of that data. > "Structured data" is often an unnecessary and cumbersome overkill What does this even mean? Data is structured by definition. If something doesn't have internal structure to it, it is not data. The fact that Unix decides to be willfully ignorant of that structure and force you to ignore it doesn't mean that it doesn't exist. I would hope you didn't mean "representing structured data as such is inconvenient" because that would be a very ignorant statement to make that conflates tooling with data format and that has zero empirical evidence behind it.