4 ms·

Unfortunately "text is the universal interface" is pretty much a lie, for the exact same reason that "bytes in memory are the universal interface" is a lie. An

Unfortunately "text is the universal interface" is pretty much a lie, for the exact same reason that "bytes in memory are the universal interface" is a lie. An interface has some kind of structure for the data (and operations) that it represents. That structure may be explicitly defined or left implicit, but it is always there. And in the vast majority of cases, that structure is more complex than "a single independent string". That's where the whole "pipe text between processes" approach breaks down - in many cases, you aren't just dealing with a single string, but with some sort of structured data that just happens to be encoded as human-readable text. That it's "text" is entirely immaterial to the structure of the data, or how to work with it. And that's how you end up with every CLI tool implementing its own set of subtly incompatible data parsers and stringifiers. This is strictly worse than having a standardized structural format to communicate in.

KronisLV 4y ago
> That structure may be explicitly defined or left implicit, but it is always there. But this is something that every piece of software out there has to deal with. By this reasoning, claiming that RESTful APIs can evolve over time and don't need something like WSDL (though OpenAPI still exists) is also a similar lie, because there absolutely are assumptions about how things are structured by anyone and everyone who actually needs to integrate with the API. Even "schema-less" data storage solutions like MongoDB are then also built on similar lies, because while you can throw arbitrary data at them to store it, querying the data later will still depend on some assumptions about the data structure. I'm inclined to agree, and yet, RESTful APIs are still very popular and seem to have much success (though some push for more "immutable" APIs that are versioned, which is nice), as do "schema-less" data stores in certain domains. Edit: you can also say that about everything from CLI tools or even dynamic programming languages as a whole. I think there's more to it. > And that's how you end up with every CLI tool implementing its own set of subtly incompatible data parsers and stringifiers. This is strictly worse than having a standardized structural format to communicate in. I suspect that this is where opinions might differ. Half-assing things and counting on vaguely compatible (at a given point in time) implementations has been the approach on which many pieces of software are built to solve numerous problems. On one hand, that is ignorant, but on the other hand - how often is the output from "ls -la" going to change its format, really? The majority of CLI tool writers seem to understand how brittle and bad everything is and thus treat their data formats carefully, adding a whole bunch of different flags as needed, or allowing for custom formats, like: "lsblk --output UUID,LABEL,SIZE" For many out there, that's good enough. At least to the point where something like PowerShell isn't mainstream, despite some great ideas in it (or other shells that would let you work with objects, instead of text).
- joepie91_ 4y ago
  > Even "schema-less" data storage solutions like MongoDB are then also built on similar lies, because while you can throw arbitrary data at them to store it, querying the data later will still depend on some assumptions about the data structure. They are. I have been complaining loudly about this for years :) > I'm inclined to agree, and yet, RESTful APIs are still very popular and seem to have much success (though some push for more "immutable" APIs that are versioned, which is nice), as do "schema-less" data stores in certain domains. Edit: you can also say that about everything from CLI tools or even dynamic programming languages as a whole. I think there's more to it. The part that I think you're missing is the difference between structure and schema. It's entirely valid to leave the schema implicit in many circumstances (but not all - generally not in a database, for example), but you still need the general structure of how data is represented. When you deserialize something from JSON, CBOR, whatever, you will get back a bunch of structured data. You may not know what it means semantically, but it's clear what the correct (memory) representation is for each bit of data, and that's exactly the most fragile part of dealing with data, which is now solved for you. You do not get the same benefit when passing around strings in ad-hoc formats that need custom parsers; it is very easy to mess up that fragile part of data handling, which is why parsers are so infamously difficult to write, whereas working with parsed data structures is generally considered much simpler. Likewise, from a usability perspective, it's fairly trivial to let the user pass eg. a dotpath to some tool to select some nested data from its input; but letting the user pass an entire parsing specification as an argument is not viable, and that's why most tools just don't allow that, and instead come with one or more built-in formats. If those supported formats don't match between two tools you're using, well, sucks to be you. They just won't interoperate now. Essentially, "standardized structure but flexible schema" is the 'happy compromise' where the most fragile part is taken care of for you, but the least predictable and most variable part is still entirely customizable. You can see this work successfully in eg. nushell or Powershell, or even to a more limited degree in tools like `jq`. > I suspect that this is where opinions might differ. Half-assing things and counting on vaguely compatible (at a given point in time) implementations has been the approach on which many pieces of software are built to solve numerous problems. On one hand, that is ignorant, but on the other hand - how often is the output from "ls -la" going to change its format, really? The majority of CLI tool writers seem to understand how brittle and bad everything is and thus treat their data formats carefully, adding a whole bunch of different flags as needed, or allowing for custom formats, like: "lsblk --output UUID,LABEL,SIZE" Speaking as someone who is currently working on a project that needs to parse a lot of CLI tools: the formats still change all the time, often in subtle ways that a human wouldn't notice but a parser would, and there's effectively zero consistency between tools, with a ton of edgecases, some of them with security impact (eg. item separators). It's an absolute nightmare to work with, and interop is just bad. > For many out there, that's good enough. I think there's a strong selection bias here. It's not without reason that so many people have an aversion to terminals today - they just aren't very good. And it's not just their terminal-ness either, because many people who cannot deal with a standard (eg. bash) shell will happily use terminal-like input systems in specialized software. It's certainly true that most people who use terminals on a daily basis, consider this paradigm good enough. But that's probably because those who don't, just stop using terminals. We should be striving to make tech better and more accessible, not just "good enough for a bunch of Linux nerds", and certainly not upholding "good enough" paradigms as some sort of virtue of computing, which is what happens when people say "text is the universal interface". Of course changing this is a long process, and there are very real practical barriers to adoption of other interop models. But that's not a reason to downgrade the 'ideal' to fit the current reality, only a reason to acknowledge that we're just not there yet.

KronisLV 4y ago

> That structure may be explicitly defined or left implicit, but it is always there. But this is something that every piece of software out there has to deal with. By this reasoning, claiming that RESTful APIs can evolve over time and don't need something like WSDL (though OpenAPI still exists) is also a similar lie, because there absolutely are assumptions about how things are structured by anyone and everyone who actually needs to integrate with the API. Even "schema-less" data storage solutions like MongoDB are then also built on similar lies, because while you can throw arbitrary data at them to store it, querying the data later will still depend on some assumptions about the data structure. I'm inclined to agree, and yet, RESTful APIs are still very popular and seem to have much success (though some push for more "immutable" APIs that are versioned, which is nice), as do "schema-less" data stores in certain domains. Edit: you can also say that about everything from CLI tools or even dynamic programming languages as a whole. I think there's more to it. > And that's how you end up with every CLI tool implementing its own set of subtly incompatible data parsers and stringifiers. This is strictly worse than having a standardized structural format to communicate in. I suspect that this is where opinions might differ. Half-assing things and counting on vaguely compatible (at a given point in time) implementations has been the approach on which many pieces of software are built to solve numerous problems. On one hand, that is ignorant, but on the other hand - how often is the output from "ls -la" going to change its format, really? The majority of CLI tool writers seem to understand how brittle and bad everything is and thus treat their data formats carefully, adding a whole bunch of different flags as needed, or allowing for custom formats, like: "lsblk --output UUID,LABEL,SIZE" For many out there, that's good enough. At least to the point where something like PowerShell isn't mainstream, despite some great ideas in it (or other shells that would let you work with objects, instead of text).

joepie91_ 4y ago
> Even "schema-less" data storage solutions like MongoDB are then also built on similar lies, because while you can throw arbitrary data at them to store it, querying the data later will still depend on some assumptions about the data structure. They are. I have been complaining loudly about this for years :) > I'm inclined to agree, and yet, RESTful APIs are still very popular and seem to have much success (though some push for more "immutable" APIs that are versioned, which is nice), as do "schema-less" data stores in certain domains. Edit: you can also say that about everything from CLI tools or even dynamic programming languages as a whole. I think there's more to it. The part that I think you're missing is the difference between structure and schema. It's entirely valid to leave the schema implicit in many circumstances (but not all - generally not in a database, for example), but you still need the general structure of how data is represented. When you deserialize something from JSON, CBOR, whatever, you will get back a bunch of structured data. You may not know what it means semantically, but it's clear what the correct (memory) representation is for each bit of data, and that's exactly the most fragile part of dealing with data, which is now solved for you. You do not get the same benefit when passing around strings in ad-hoc formats that need custom parsers; it is very easy to mess up that fragile part of data handling, which is why parsers are so infamously difficult to write, whereas working with parsed data structures is generally considered much simpler. Likewise, from a usability perspective, it's fairly trivial to let the user pass eg. a dotpath to some tool to select some nested data from its input; but letting the user pass an entire parsing specification as an argument is not viable, and that's why most tools just don't allow that, and instead come with one or more built-in formats. If those supported formats don't match between two tools you're using, well, sucks to be you. They just won't interoperate now. Essentially, "standardized structure but flexible schema" is the 'happy compromise' where the most fragile part is taken care of for you, but the least predictable and most variable part is still entirely customizable. You can see this work successfully in eg. nushell or Powershell, or even to a more limited degree in tools like `jq`. > I suspect that this is where opinions might differ. Half-assing things and counting on vaguely compatible (at a given point in time) implementations has been the approach on which many pieces of software are built to solve numerous problems. On one hand, that is ignorant, but on the other hand - how often is the output from "ls -la" going to change its format, really? The majority of CLI tool writers seem to understand how brittle and bad everything is and thus treat their data formats carefully, adding a whole bunch of different flags as needed, or allowing for custom formats, like: "lsblk --output UUID,LABEL,SIZE" Speaking as someone who is currently working on a project that needs to parse a lot of CLI tools: the formats still change all the time, often in subtle ways that a human wouldn't notice but a parser would, and there's effectively zero consistency between tools, with a ton of edgecases, some of them with security impact (eg. item separators). It's an absolute nightmare to work with, and interop is just bad. > For many out there, that's good enough. I think there's a strong selection bias here. It's not without reason that so many people have an aversion to terminals today - they just aren't very good. And it's not just their terminal-ness either, because many people who cannot deal with a standard (eg. bash) shell will happily use terminal-like input systems in specialized software. It's certainly true that most people who use terminals on a daily basis, consider this paradigm good enough. But that's probably because those who don't, just stop using terminals. We should be striving to make tech better and more accessible, not just "good enough for a bunch of Linux nerds", and certainly not upholding "good enough" paradigms as some sort of virtue of computing, which is what happens when people say "text is the universal interface". Of course changing this is a long process, and there are very real practical barriers to adoption of other interop models. But that's not a reason to downgrade the 'ideal' to fit the current reality, only a reason to acknowledge that we're just not there yet.