5 ms·

> Even "schema-less" data storage solutions like MongoDB are then also built on similar lies, because while you can throw arbitrary data at them to store it, qu

> Even "schema-less" data storage solutions like MongoDB are then also built on similar lies, because while you can throw arbitrary data at them to store it, querying the data later will still depend on some assumptions about the data structure. They are. I have been complaining loudly about this for years :) > I'm inclined to agree, and yet, RESTful APIs are still very popular and seem to have much success (though some push for more "immutable" APIs that are versioned, which is nice), as do "schema-less" data stores in certain domains. Edit: you can also say that about everything from CLI tools or even dynamic programming languages as a whole. I think there's more to it. The part that I think you're missing is the difference between structure and schema. It's entirely valid to leave the schema implicit in many circumstances (but not all - generally not in a database, for example), but you still need the general structure of how data is represented. When you deserialize something from JSON, CBOR, whatever, you will get back a bunch of structured data. You may not know what it means semantically, but it's clear what the correct (memory) representation is for each bit of data, and that's exactly the most fragile part of dealing with data, which is now solved for you. You do not get the same benefit when passing around strings in ad-hoc formats that need custom parsers; it is very easy to mess up that fragile part of data handling, which is why parsers are so infamously difficult to write, whereas working with parsed data structures is generally considered much simpler. Likewise, from a usability perspective, it's fairly trivial to let the user pass eg. a dotpath to some tool to select some nested data from its input; but letting the user pass an entire parsing specification as an argument is not viable, and that's why most tools just don't allow that, and instead come with one or more built-in formats. If those supported formats don't match between two tools you're using, well, sucks to be you. They just won't interoperate now. Essentially, "standardized structure but flexible schema" is the 'happy compromise' where the most fragile part is taken care of for you, but the least predictable and most variable part is still entirely customizable. You can see this work successfully in eg. nushell or Powershell, or even to a more limited degree in tools like `jq`. > I suspect that this is where opinions might differ. Half-assing things and counting on vaguely compatible (at a given point in time) implementations has been the approach on which many pieces of software are built to solve numerous problems. On one hand, that is ignorant, but on the other hand - how often is the output from "ls -la" going to change its format, really? The majority of CLI tool writers seem to understand how brittle and bad everything is and thus treat their data formats carefully, adding a whole bunch of different flags as needed, or allowing for custom formats, like: "lsblk --output UUID,LABEL,SIZE" Speaking as someone who is currently working on a project that needs to parse a lot of CLI tools: the formats still change all the time, often in subtle ways that a human wouldn't notice but a parser would, and there's effectively zero consistency between tools, with a ton of edgecases, some of them with security impact (eg. item separators). It's an absolute nightmare to work with, and interop is just bad. > For many out there, that's good enough. I think there's a strong selection bias here. It's not without reason that so many people have an aversion to terminals today - they just aren't very good. And it's not just their terminal-ness either, because many people who cannot deal with a standard (eg. bash) shell will happily use terminal-like input systems in specialized software. It's certainly true that most people who use terminals on a daily basis, consider this paradigm good enough. But that's probably because those who don't, just stop using terminals. We should be striving to make tech better and more accessible, not just "good enough for a bunch of Linux nerds", and certainly not upholding "good enough" paradigms as some sort of virtue of computing, which is what happens when people say "text is the universal interface". Of course changing this is a long process, and there are very real practical barriers to adoption of other interop models. But that's not a reason to downgrade the 'ideal' to fit the current reality, only a reason to acknowledge that we're just not there yet.

KronisLV 4y ago
> It's entirely valid to leave the schema implicit in many circumstances (but not all - generally not in a database, for example), but you still need the general structure of how data is represented. > ... > Essentially, "standardized structure but flexible schema" is the 'happy compromise' where the most fragile part is taken care of for you, but the least predictable and most variable part is still entirely customizable. You can see this work successfully in eg. nushell or Powershell, or even to a more limited degree in tools like `jq`. That's a fair point, thanks for putting emphasis on this! > I think there's a strong selection bias here. It's not without reason that so many people have an aversion to terminals today - they just aren't very good. And it's not just their terminal-ness either, because many people who cannot deal with a standard (eg. bash) shell will happily use terminal-like input systems in specialized software. The question then becomes what can be reasonably done about it? Or rather, can it even be realistically achieved (in an arbitrary time scale that we care about), given how much of the culture and tooling is currently centered around passing text from one command to another. Change for something so foundational surely wouldn't be quick. > But that's not a reason to downgrade the 'ideal' to fit the current reality, only a reason to acknowledge that we're just not there yet. In the end, this probably sums it up nicely. Even if that current reality, which often will be the lowest common denominator, is what most people will earn their paychecks with (and possibly leave edge cases for someone else to deal with down the line).
- joepie91_ 4y ago
  > The question then becomes what can be reasonably done about it? Or rather, can it even be realistically achieved (in an arbitrary time scale that we care about), given how much of the culture and tooling is currently centered around passing text from one command to another. Change for something so foundational surely wouldn't be quick. It definitely won't be quick, no. I think the tooling is actually not that big of a problem there; the functionality provided by the 'common tools' generally isn't that complex in scope (at least, in the context of modern development tools), and the many RIIR projects have shown that total do-overs for these sorts of tools are viable. The bigger problem is going to be cultural, and particularly that persistent (but often unspoken) belief that "computers peaked with UNIX in the 70s/80s and what we have now is the best it will ever be". It often results in people actively pushing back on improvements that really wouldn't have any downsides for them at all, thereby creating unnecessary friction. That same 'ideology', for lack of a better word, also makes it very difficult to talk about eg. hybrid terminal/graphical systems - because "GUIs are evil, text is better" and similar sentiments. That really gets in the way of advancing technology here. Ultimately, I think the actual interop and/or reimplementation problems are going to be a walk in the park, compared to dealing with the messy human and cultural/ideological factors involved that actively resist change.

KronisLV 4y ago

> It's entirely valid to leave the schema implicit in many circumstances (but not all - generally not in a database, for example), but you still need the general structure of how data is represented. > ... > Essentially, "standardized structure but flexible schema" is the 'happy compromise' where the most fragile part is taken care of for you, but the least predictable and most variable part is still entirely customizable. You can see this work successfully in eg. nushell or Powershell, or even to a more limited degree in tools like `jq`. That's a fair point, thanks for putting emphasis on this! > I think there's a strong selection bias here. It's not without reason that so many people have an aversion to terminals today - they just aren't very good. And it's not just their terminal-ness either, because many people who cannot deal with a standard (eg. bash) shell will happily use terminal-like input systems in specialized software. The question then becomes what can be reasonably done about it? Or rather, can it even be realistically achieved (in an arbitrary time scale that we care about), given how much of the culture and tooling is currently centered around passing text from one command to another. Change for something so foundational surely wouldn't be quick. > But that's not a reason to downgrade the 'ideal' to fit the current reality, only a reason to acknowledge that we're just not there yet. In the end, this probably sums it up nicely. Even if that current reality, which often will be the lowest common denominator, is what most people will earn their paychecks with (and possibly leave edge cases for someone else to deal with down the line).

joepie91_ 4y ago
> The question then becomes what can be reasonably done about it? Or rather, can it even be realistically achieved (in an arbitrary time scale that we care about), given how much of the culture and tooling is currently centered around passing text from one command to another. Change for something so foundational surely wouldn't be quick. It definitely won't be quick, no. I think the tooling is actually not that big of a problem there; the functionality provided by the 'common tools' generally isn't that complex in scope (at least, in the context of modern development tools), and the many RIIR projects have shown that total do-overs for these sorts of tools are viable. The bigger problem is going to be cultural, and particularly that persistent (but often unspoken) belief that "computers peaked with UNIX in the 70s/80s and what we have now is the best it will ever be". It often results in people actively pushing back on improvements that really wouldn't have any downsides for them at all, thereby creating unnecessary friction. That same 'ideology', for lack of a better word, also makes it very difficult to talk about eg. hybrid terminal/graphical systems - because "GUIs are evil, text is better" and similar sentiments. That really gets in the way of advancing technology here. Ultimately, I think the actual interop and/or reimplementation problems are going to be a walk in the park, compared to dealing with the messy human and cultural/ideological factors involved that actively resist change.