4 ms·

> if you can't even represent your data any data can be represented as text

throw10920 2y ago
Only in the least useful, most pedantic sense of the word. I can "represent" image data as base64 text in my terminal and yet have no tools to meaningfully manipulate it with. This is a gotcha with no interesting thought behind it.
- enriquto 2y ago
  Your example is particularly funny, because one can easily undecode the base64 image, convert it to sixel so that it's displayed in the terminal verbatim, all in half a line of shell pipes. If the image is large, you can also pipe it to an interactive visualizer to look at it. Or, you just "cat" your base64 image as is, together with some recently echoed mail headers and pipe it to mail as an attachement. The tools to do that are standard and come with all unix distributions. The greatness of unix is that none of the intermediary programs need to know that they are dealing with an image. This kind of simplicity is impossible if the type of your data must be known by each intermediary program. "Structured data" is often an unnecessary and cumbersome overkill, especially for simple tasks.
  - throw10920 2y ago
    > Your example is particularly funny, because one can easily undecode the base64 image I don't see how that's relevant - the fact that a very small number of specific terminal programs can decode limited data formats and communicate over terminal control sequences that only work for that data format doesn't seem to matter discussion of whether the shell or OS supports structured data in general or whether it gives you the tools to manipulate that data. Moreover, this is actually an example that further proves my point: you can't use classic Unix tools to process that image data. You can't use cut to select rows or columns, head or tail to get parts of an image, grep to look for pixels - of course you can't, because it's not text, and one of the core tenets of the Unix philosophy is that everything is plain text. Now, this wasn't a great example to begin with, because media is somewhat of a special case (and it's somewhat sequential data, as opposed to hierarchical). Let's take something like an array of transactions for a personal budget in JSON form, where each transaction has fields like "type" (e.g. "credit", "debit"), "amount", and "other party", and "other party" further has "type" (e.g. "business", "person", "bank") and "name" fields. The Unix philosophy does not allow you to directly represent this data - there is no way to represent key-value pairs or nested data and then do things like extract fields or filter on values. Your primitives are lines and characters, by definition. > This kind of simplicity is impossible if the type of your data must be known by each intermediary program. I think you're misunderstanding what "structured data" is, because what you just described isn't a property of structured data. I can write a "filter" program that takes an arbitrary test function and applies it to a blob of data, and that program needs zero information about the data, other than that it's an array. Or, it's possible that you're just not familiar with the concept of abstraction, where programming systems can expose an interface that requires other systems to not understand their internals. > The greatness of unix is that none of the intermediary programs need to know that they are dealing with an image. This has nothing to do with Unix. It's trivial to conceive of a shell or OS where programs pass around structured data and are able to productively operate on it without needing to know the full structure of that data. > "Structured data" is often an unnecessary and cumbersome overkill What does this even mean? Data is structured by definition. If something doesn't have internal structure to it, it is not data. The fact that Unix decides to be willfully ignorant of that structure and force you to ignore it doesn't mean that it doesn't exist. I would hope you didn't mean "representing structured data as such is inconvenient" because that would be a very ignorant statement to make that conflates tooling with data format and that has zero empirical evidence behind it.
    - enriquto 2y ago
      > there is no way to represent key-value pairs or nested data and then do things like extract fields or filter on values man, ever heard of awk? It was designed exactly for that purpose.
      - throw10920 2y ago
        I'm very well aware of awk. This proves my point - Unix doesn't have a way to represent structure, so the only way that you can manipulate it is with purpose-built tools. There's no way to tell "sort" to sort on a specific field, for instance - sort does not understand structure. Things that are trivial in a structured/typed language like PowerShell or Python require a lot of sed/awk glue in Unix and are far less reliable as a result. Anyone who has a basic understanding of computing theory understands that shell is obviously Turing-complete and can compute anything that another Turing machine has - which means that being able to hack something together does not mean that the system was designed for that paradigm. You can implement Prolog in C, but that does not mean that C is a declarative logic programming language, and if you find yourself repeatedly re-implementing Prologs in your C programs, you'd do very well to have the self-awareness to realize that C is probably not the right choice for the problems you're trying to solve. Similarly, if you have to repeatedly shell out to a tool that understands structure like awk or Perl or jq, you'd do well to consider as to whether the shell doesn't actually fit the problem you're trying to solve. There's a reason why there are precisely zero large pieces of software written in bash - because it's an extremely poor paradigm for computation.
coliveira 2y ago
Yes you can represent anything as text, but then you create another problem: parsing the representation. You always need to balance the advantage of sending data in a free format and the need to parse it back into usable input.
agumonkey 2y ago
reminds me of 'parse, don't validate'

Only in the least useful, most pedantic sense of the word. I can "represent" image data as base64 text in my terminal and yet have no tools to meaningfully manipulate it with. This is a gotcha with no interesting thought behind it.

enriquto 2y ago
Your example is particularly funny, because one can easily undecode the base64 image, convert it to sixel so that it's displayed in the terminal verbatim, all in half a line of shell pipes. If the image is large, you can also pipe it to an interactive visualizer to look at it. Or, you just "cat" your base64 image as is, together with some recently echoed mail headers and pipe it to mail as an attachement. The tools to do that are standard and come with all unix distributions. The greatness of unix is that none of the intermediary programs need to know that they are dealing with an image. This kind of simplicity is impossible if the type of your data must be known by each intermediary program. "Structured data" is often an unnecessary and cumbersome overkill, especially for simple tasks.
- throw10920 2y ago
  > Your example is particularly funny, because one can easily undecode the base64 image I don't see how that's relevant - the fact that a very small number of specific terminal programs can decode limited data formats and communicate over terminal control sequences that only work for that data format doesn't seem to matter discussion of whether the shell or OS supports structured data in general or whether it gives you the tools to manipulate that data. Moreover, this is actually an example that further proves my point: you can't use classic Unix tools to process that image data. You can't use cut to select rows or columns, head or tail to get parts of an image, grep to look for pixels - of course you can't, because it's not text, and one of the core tenets of the Unix philosophy is that everything is plain text. Now, this wasn't a great example to begin with, because media is somewhat of a special case (and it's somewhat sequential data, as opposed to hierarchical). Let's take something like an array of transactions for a personal budget in JSON form, where each transaction has fields like "type" (e.g. "credit", "debit"), "amount", and "other party", and "other party" further has "type" (e.g. "business", "person", "bank") and "name" fields. The Unix philosophy does not allow you to directly represent this data - there is no way to represent key-value pairs or nested data and then do things like extract fields or filter on values. Your primitives are lines and characters, by definition. > This kind of simplicity is impossible if the type of your data must be known by each intermediary program. I think you're misunderstanding what "structured data" is, because what you just described isn't a property of structured data. I can write a "filter" program that takes an arbitrary test function and applies it to a blob of data, and that program needs zero information about the data, other than that it's an array. Or, it's possible that you're just not familiar with the concept of abstraction, where programming systems can expose an interface that requires other systems to not understand their internals. > The greatness of unix is that none of the intermediary programs need to know that they are dealing with an image. This has nothing to do with Unix. It's trivial to conceive of a shell or OS where programs pass around structured data and are able to productively operate on it without needing to know the full structure of that data. > "Structured data" is often an unnecessary and cumbersome overkill What does this even mean? Data is structured by definition. If something doesn't have internal structure to it, it is not data. The fact that Unix decides to be willfully ignorant of that structure and force you to ignore it doesn't mean that it doesn't exist. I would hope you didn't mean "representing structured data as such is inconvenient" because that would be a very ignorant statement to make that conflates tooling with data format and that has zero empirical evidence behind it.
  - enriquto 2y ago
    > there is no way to represent key-value pairs or nested data and then do things like extract fields or filter on values man, ever heard of awk? It was designed exactly for that purpose.
    - throw10920 2y ago
      I'm very well aware of awk. This proves my point - Unix doesn't have a way to represent structure, so the only way that you can manipulate it is with purpose-built tools. There's no way to tell "sort" to sort on a specific field, for instance - sort does not understand structure. Things that are trivial in a structured/typed language like PowerShell or Python require a lot of sed/awk glue in Unix and are far less reliable as a result. Anyone who has a basic understanding of computing theory understands that shell is obviously Turing-complete and can compute anything that another Turing machine has - which means that being able to hack something together does not mean that the system was designed for that paradigm. You can implement Prolog in C, but that does not mean that C is a declarative logic programming language, and if you find yourself repeatedly re-implementing Prologs in your C programs, you'd do very well to have the self-awareness to realize that C is probably not the right choice for the problems you're trying to solve. Similarly, if you have to repeatedly shell out to a tool that understands structure like awk or Perl or jq, you'd do well to consider as to whether the shell doesn't actually fit the problem you're trying to solve. There's a reason why there are precisely zero large pieces of software written in bash - because it's an extremely poor paradigm for computation.

coliveira 2y ago

Yes you can represent anything as text, but then you create another problem: parsing the representation. You always need to balance the advantage of sending data in a free format and the need to parse it back into usable input.

agumonkey 2y ago

reminds me of 'parse, don't validate'