5 ms·

> This is the idea that programs will all talk together via unstructured streams of text. Curious. To me this is the worst thing you could ever do. Talking via

> This is the idea that programs will all talk together via unstructured streams of text. Curious. To me this is the worst thing you could ever do. Talking via streams to say `cat file.txt | grep ERROR | wc -l` is cool. But you could do SOOO much more, if programs would actually output structured data streams. You could connect standalone applications much in the same way as Visual Scripting, where you plug inputs and outputs together and mix them with operators (think of Unreal Engine's Blueprint, just for command line tooling). It's a true shame that Linux did not develop a well defined CLI metaformat that defined exactly what parameters are there, what's their documentation, their completion, what outputs does a program produce based on the parameters you provide, etc. You could do true magic with all this information. Right now you kinda still can, but it is very brittle, a lot of work and breaks potentially with each version increment. I think it stems from the design failure to build your app around a CLI. Instead, you should build your app around an API and generate the CLI for that API. Then all properties of structured data streams and auto-explore CLI shells come for free.

ZeteticElench 6y ago
Have you used PowerShell? It passes full objects via pipes and it works really well.
bzb6 6y ago
As my sibling dead comment says you should try powershell.
- liversage 6y ago
  Even the original post said so in the third paragraph: > and now PowerShell where you can load .NET assemblies and pass around structured objects
npsimons 6y ago
> But you could do SOOO much more, if programs would actually output structured data streams. Like even treating code and data the same, and minimizing the syntax required so you're left with clean, parseable code and data. Maybe in some sort of tree, that is abstract. Where have I heard this idea before . . . > I think it stems from the design failure to build your app around a CLI. Instead, you should build your app around an API and generate the CLI for that API. Now this I am fully in favor of, and IMHO, it leads to much better code all around: you can then test via the API, build a GUI via the API, etc, etc, etc.
patrec 6y ago
> Curious. To me this is the worst thing you could ever do. Talking via streams to say `cat file.txt | grep ERROR | wc -l` is cool. But you could do SOOO much more, if programs would actually output structured data streams. A lot of people have had this thought over the decades, but it hasn't really happened -- powershell exists for linux, but who's using it? The genius of the primitive representation (stringly typed tables) is that it has just enough structure to do interesting processing but not enough to cause significant mental overhead in trying to understand, memorize and reference the structure. Case in point of the difficulties of adding more structure without wrecking immediacy of manipulation is json. For anything with more than 1 level of nesting, I do stuff like blah | jq . | grep -C3 ERROR probably a lot more than I do blah | jq $SOME_EXPRESSION because it's just so much less mental overhead -- I don't have to think about indexing into some complex hierarchy and pulling out parts of it. I'm not saying it's not possible to get out of this local optimum, but it appears to be a lot more subtle than many people seem to think. There may be an simple and elegant solution, but it seems it has so far escaped discovery. Almost five decades later, composing pipelines of weakly structured and typed bytes (that, by convention, often are line separated tables, possibly with tab or space seperated columns) is still the only high-level software re-use via composition success story of the whole computing field.
- lenkite 6y ago
  Very few use Powershell for Linux because it doesn't pre-installed on a Linux box. Otherwise you can bet that people would be using it in large numbers. And yes I would prefer your second "mental overhead" way as it involves less typing. Unfortunately powershell is more verbose than bash not less. Powershell is unfortunately not the shining example of a shell that best leverages structured/typed input/output succinctly. But on Windows, sysadmins use powershell heavily. Nearly every IT department that manages windows machines uses Powershell.
  - f1refly 6y ago
    > Very few use Powershell for Linux because it doesn't pre-installed on a Linux box I don't buy that. On a GNU/Linux box, there's few things that are easier than installing a new shell, if you prefer a different shell than bash it's two commands away. Bash does the job people expect it to do and would probably be _very_ alienated it they'd had to start messing around with .net gubbins. >And yes I would prefer your second "mental overhead" way as it involves less typing Maybe for the first time you would. Maybe if you were to accomplish this specific thing. Anything else? Have fun diving into the manpage of your shell _and_ the programs you want to use, and you better hope they share a somewhat common approach to the implemented (object) datatype or well, good luck trying to get them to talk with each other >Powershell is unfortunately not the shining example of a shell that best leverages structured/typed input/output succinctly I would just remove the last part, then agree with you: ">Powershell is unfortunately not the shining example of a shell" > Nearly every IT department that manages windows machines uses Powershell I mean, what other choice do they have there? cmd? Yeah right, if you want to loose your will to live go for it
yourapostasy 6y ago
IMHO, I need both text streams and metaformat. YMMV. Just like GUIs should but usually are not gracefully, responsively scaled to user expertise, the developer experience should but usually are not gracefully, responsively scaled to the appropriate level of scaffolding to fit for purpose to the requirements defining the problem space at hand. I need more representations and abstractions, not less. Metaformats drag in their own logistical long tail that in many use cases are wildly heavyweight for small problems. Demanding metaformats or APIs everywhere and The Only Option trades off against the REPL-like accessibility of lesser scaffolding. API-first comes with its own non-trivial balls of string; version control between caller and callee, argument parsing between versions, impedance mismatch to the kind of generative CLI's you envision, and against other API interfaces, etc. The current unstructured primitives on the CLI, composable into structured primitives presenting as microservices or similar functions landing into a more DevOps-style landscape, etc. represents a pretty flexible toolbox that helps mitigate some of the risks in Big Design Up Front efforts that structure tends to emerge in my experience. I think of it as REPL-in-the-large. As I gained experience I've come to appreciate and tolerate the ragged edge uncouthness of real world solutions, and lose a lot of my fanatical puritanism that veered into astronaut architecture.

ZeteticElench 6y ago

Have you used PowerShell? It passes full objects via pipes and it works really well.

bzb6 6y ago

As my sibling dead comment says you should try powershell.

liversage 6y ago
Even the original post said so in the third paragraph: > and now PowerShell where you can load .NET assemblies and pass around structured objects

npsimons 6y ago

> But you could do SOOO much more, if programs would actually output structured data streams. Like even treating code and data the same, and minimizing the syntax required so you're left with clean, parseable code and data. Maybe in some sort of tree, that is abstract. Where have I heard this idea before . . . > I think it stems from the design failure to build your app around a CLI. Instead, you should build your app around an API and generate the CLI for that API. Now this I am fully in favor of, and IMHO, it leads to much better code all around: you can then test via the API, build a GUI via the API, etc, etc, etc.

patrec 6y ago

> Curious. To me this is the worst thing you could ever do. Talking via streams to say `cat file.txt | grep ERROR | wc -l` is cool. But you could do SOOO much more, if programs would actually output structured data streams. A lot of people have had this thought over the decades, but it hasn't really happened -- powershell exists for linux, but who's using it? The genius of the primitive representation (stringly typed tables) is that it has just enough structure to do interesting processing but not enough to cause significant mental overhead in trying to understand, memorize and reference the structure. Case in point of the difficulties of adding more structure without wrecking immediacy of manipulation is json. For anything with more than 1 level of nesting, I do stuff like blah | jq . | grep -C3 ERROR probably a lot more than I do blah | jq $SOME_EXPRESSION because it's just so much less mental overhead -- I don't have to think about indexing into some complex hierarchy and pulling out parts of it. I'm not saying it's not possible to get out of this local optimum, but it appears to be a lot more subtle than many people seem to think. There may be an simple and elegant solution, but it seems it has so far escaped discovery. Almost five decades later, composing pipelines of weakly structured and typed bytes (that, by convention, often are line separated tables, possibly with tab or space seperated columns) is still the only high-level software re-use via composition success story of the whole computing field.

lenkite 6y ago
Very few use Powershell for Linux because it doesn't pre-installed on a Linux box. Otherwise you can bet that people would be using it in large numbers. And yes I would prefer your second "mental overhead" way as it involves less typing. Unfortunately powershell is more verbose than bash not less. Powershell is unfortunately not the shining example of a shell that best leverages structured/typed input/output succinctly. But on Windows, sysadmins use powershell heavily. Nearly every IT department that manages windows machines uses Powershell.
- f1refly 6y ago
  > Very few use Powershell for Linux because it doesn't pre-installed on a Linux box I don't buy that. On a GNU/Linux box, there's few things that are easier than installing a new shell, if you prefer a different shell than bash it's two commands away. Bash does the job people expect it to do and would probably be _very_ alienated it they'd had to start messing around with .net gubbins. >And yes I would prefer your second "mental overhead" way as it involves less typing Maybe for the first time you would. Maybe if you were to accomplish this specific thing. Anything else? Have fun diving into the manpage of your shell _and_ the programs you want to use, and you better hope they share a somewhat common approach to the implemented (object) datatype or well, good luck trying to get them to talk with each other >Powershell is unfortunately not the shining example of a shell that best leverages structured/typed input/output succinctly I would just remove the last part, then agree with you: ">Powershell is unfortunately not the shining example of a shell" > Nearly every IT department that manages windows machines uses Powershell I mean, what other choice do they have there? cmd? Yeah right, if you want to loose your will to live go for it

yourapostasy 6y ago

IMHO, I need both text streams and metaformat. YMMV. Just like GUIs should but usually are not gracefully, responsively scaled to user expertise, the developer experience should but usually are not gracefully, responsively scaled to the appropriate level of scaffolding to fit for purpose to the requirements defining the problem space at hand. I need more representations and abstractions, not less. Metaformats drag in their own logistical long tail that in many use cases are wildly heavyweight for small problems. Demanding metaformats or APIs everywhere and The Only Option trades off against the REPL-like accessibility of lesser scaffolding. API-first comes with its own non-trivial balls of string; version control between caller and callee, argument parsing between versions, impedance mismatch to the kind of generative CLI's you envision, and against other API interfaces, etc. The current unstructured primitives on the CLI, composable into structured primitives presenting as microservices or similar functions landing into a more DevOps-style landscape, etc. represents a pretty flexible toolbox that helps mitigate some of the risks in Big Design Up Front efforts that structure tends to emerge in my experience. I think of it as REPL-in-the-large. As I gained experience I've come to appreciate and tolerate the ragged edge uncouthness of real world solutions, and lose a lot of my fanatical puritanism that veered into astronaut architecture.