11 ms·

While I do like the new wave of the structured shells(elvish, uxy, nushell, crush, etc...), I don't think anything will actually succeed unless there's a big ch

While I do like the new wave of the structured shells(elvish, uxy, nushell, crush, etc...), I don't think anything will actually succeed unless there's a big change in the operating system. (I'm not suggesting these aren't valuable - these are valuable experiments, but I'm saying for them to succeed, a new OS is needed.) The native OS doesn't really embrace structured communication, and so commands will still have to treat the structured shell as a second-level citizen. In an ideal world, for the structured shell to work, 'npm list' should return a table (or whatever the native data structure is) and you should be able to do 'npm list | where {deduped==false}'. In the real world, 'npm list' won't support the structured shell, so you would be writing 'npm list --json | jq '<obscure commands>' | json:from | where {deduped==false}'. And then 'npm list | grep -v 'deduped' is more simpler, so you don't really feel the advantages of the structured shell. I think for this to be solved, the OS's primary communication system should be structured, so you can guarantee that every command will be a table. Then the programmers for the tools of that OS will output a table, and the users will be able to take advantage of this. Until then... it's basically a fun experiment.

dvh 6y ago
https://github.com/hikikomori82/StructurOS https://github.com/hikikomori82/StructurOS
rootlocus 6y ago
> I think for this to be solved, the OS's primary communication system should be structured You mean the STDOUT format of all processes? I disagree. `npm list --json` already does what you want. Your only argument is that ` | jq '<obscure commands>' | json:from | where {deduped==false}` is too obscure. The solution is (probably) a simplified `jq` that has the query format you want. Problem solved. Keep it simple.
- smitty1e 6y ago
  I kinda like the mystical feeling of jq. As with haskell, it'll feel as though I've arrived somehow, should I achieve real understanding.
  - frou_dh 6y ago
    To me it's a good example of Opportunity Cost. We could spend the time and effort memorising jq's special snowflake language, but that time and effort would almost certainly be better spent on something else. I deleted it a while back. Can't be bothered with it.
    - knome 6y ago
      >jq's special snowflake language If you'll bother condescending to such, you'll find one of the most interesting languages I've encountered in years. jq's syntax and execution model is beautiful. It has some rough edges, leave no doubt. Some function names, date handling, and the obtuseness of some of the errors it returns can be frustrating. But its data streaming layers of nested closures are an extremely terse, readable, and powerful way of expressing operations over structured data.
    - gen220 6y ago
      I usually use jq just to format the blob, and then some unholy combination of grep -A/-B, tr, and sed to get the output I want. It’s usually faster (for me) than relearning jq’s query language.
    - deleted 6y ago
      [deleted]
    - klibertp 6y ago
      > would almost certainly be better spent on something else. I'd would suggest to spend that time on learning how to effectively learn new languages. There's more than one way, and I guarantee that afterwards you'll conquer any given "snowflake" in no time at all.
      - frou_dh 6y ago
        I deliberately wrote "memorise" rather than learn in the usual sense, because there's a higher than normal degree of familiarity needed to get working code flowing from the fingers, when the venue is between a pair of single quotes in a shell.
schemy 6y ago
The problem, as always, is that a table isn't the natural output for a computer. It's either serialized text, or a graph. Unix tools work ok-ish for serialized text and you need general recursive functions to deal with graphs. Something that all shell like languages are extremely bad at.
cnity 6y ago
It doesn't have to be this black and white, in my opinion. The most common data structures I've interacted with on the command line are _newline separated_ or _JSON_. What if your shell allowed parsing of JSON natively? npm list --json | ({ json }) => json.somePackage.author.name What if it allowed parsing of XML natively, allowing instant scrapers to be written in a single command? curl www.example.com | ({ xml }) => xml.find('.importantInformation') > scraped_info.txt
- guerrilla 6y ago
  Yeah this is probably it right here. Just being able to parse various common formats would give a lot of the structure people are looking for. The builtins can just operate on that. Maybe it could optionally detect the formats too. If such a shell became popular, it would also incentivize developers adding common formats as outputs to more programs too.
  - kbrazil 6y ago
    This is what jc[0] does. [0] https://github.com/kellyjonbrazil/jc https://github.com/kellyjonbrazil/jc (disclaimer: I am the author)
- trenchgun 6y ago
  Ooh that would be pretty damn sweet tbh.
- jodrellblank 6y ago
  Newline separated what? Basically all Linux config files are newline separated lines, but that won’t get you the equals separated data pairs, indented blocks, pretend-xml sections, ad-hoc boolean representations, INI-style sections, ad-hoc list-of-values style data out of them. > What if it allowed parsing of XML natively, allowing instant scrapers to be written in a single command? Powershell’s Invoke-RestMethod parses xml natively to a structured output; it’s useful but not enough to make XML fun or trivial to work with beyond root.node.foo.title simple scraping.
  - 6y ago
david_draco 6y ago
> The native OS doesn't really embrace structured communication OS system calls are highly structured, down to the argument position and data type.
- smitty1e 6y ago
  Right, but the unix philosophy is that everything consumes and emits text. Having something more than a blob is great. Of course, there will be a performance tax.
hhas01 6y ago
What you want is tagged and/or typed pipes. Once a pipe declares how its data is encoded, you can build all kinds of modern tooling and high-level automation on top of that. The whole “But everything’s a text file!” is an outrageous lie, and always has been: just an excuse to cover up a massive omission in the design of the Unix file system (untyped, untagged “file” resources). Such egregious corner-cutting might have been understandable—even forgivable—back in K&R’s day, when they were bootstrapping C and Unix on hardware with less processing power than a modern microwave oven controller. But we’re decades on from that now and there’s no excuse for perpetuating it: we know how to build safe powerful modern static and dynamic type systems, and we’re spoiled rotten for the hardware resources to support it. Fifty years ago, K&R built C and Unix, in a cave, with scraps. Nowadays you can bootstrap a complete hardware stack for less than the cost of a 360, and there are modern kernels (L4) and systems language (Rust) already available for free off the shelf. So pick a target market and build the userland for that. All that’s missing is a modern-day Kernighan and Ritchie to step up and get on with it, so it’d be a sorry indictment of this generation if they do not exist.
- gen220 6y ago
  > untyped, untagged “file” resources I think the problem then, and the problem today, is that these “types” are a very very mobile target. How many implementations of JSON are there? XML? HTML? CSV? It doesn’t make sense to put these types in the operating system, or else you’ll start TypeError’s every time you download anything. Not to mention the Computational costs of parsing the entire structure of a document every time you edit it to verify that it’s “still of type X”. I’m not at all sold on the idea that categorizing files and streams in this way is a step forward (in the shell, at least). This is what applications are for: to define their own formats, parsers, and generators. The point of the operating system is not to be one of these applications, but to facilitate their development and use. IMO, typed files and streams would actually impede more than enable this work.
  - hhas01 6y ago
    “It doesn’t make sense to put these types in the operating system” The OS doesn’t need to hardcode individual types, and to do so would defeat growth. It only has to provide a standard mechanism by which files can declare the type of data they contain, and executables can declare the type(s) of data they consume and/or produce. BeOS, for instance, tagged files with MIME types. The original Mac OS used “four-char codes”. Even DOS had file name extensions, as awful as that particular design choice was. Storing arbitrary data with no way to indicate how that data is encoded is rank insanity. Saying “it’s the app’s job to deal with encodings” is not wrong, but if apps have no way to know what encoding a given file was written in, how can they know how to decode it? Please do not rationalize away the longstanding and well-known deficiencies of Unix. These deficiencies do not aid progress; they hinder it. As do apologetics.
throwaway894345 6y ago
I don’t think you need an OS to pass structured objects, but rather simply a runtime that supports it, a la dotnet. Basically anything that can allow different programs to run and share memory as well as information about the shape of the objects in that shared memory. This can run on existing OSes and in the case of powershell, it already does. I think Go could probably do this as well via its plugin interface and runtime type system except that you aren’t allowed to unload plugins (so every program you ever run hangs out in memory).
hughw 6y ago
Microsoft went far down this road in the early 2000s with WinFS [1] but abandoned it. [Edit: WinFS and Longhorn together] [1] https://en.wikipedia.org/wiki/WinFS https://en.wikipedia.org/wiki/WinFS
gbear0 6y ago
I keep having the same thoughts, but I'm not sure it needs to be a full OS solution. Also a related problem that always frustrates me with input is what do I use for the help command? Is it '-h', '-help', '--help', or just 'help', or even man pages only. Each script/bin has its own way of doing things and this seems like something that should be standardized, but I realize there's no great way to do that currently when its just input to the program. I wonder if instead of an OS solution we can just add more metadata to the actual binary files we compile. For example, if we have a new ELF format that has an optional list of commands, options, full input/output types, etc, any one of the shells can start to provide better tooling. Output format types can then be passed along with the stream, and potentially checked that they work as the proper inputs for another command. Let the shell auto generate the simple cli help docs, or generate the bash completions, or figure out how to better convert data to desired types.
gregmac 6y ago
Something relatively simple would be borrowing the idea of the HTTP "accepts" header, and standardizing on an environment variable. For example: STDOUT_ACCEPT=application/json,application/csv,text/plain I'm not sure how the application could easily communicate back the understanding of this, though -- basically the equivalent of HTTP Content-Type header. It would need to be done in a way that doesn't break other applications that aren't expecting it, perhaps such as being sent back on STDERR with an escape sequence of sorts? The other trick is knowing what an application will accept via STDIN - ideally at runtime, but before actually writing anything. If the shell understands all this, it could automatically format JSON/csv output for display, integrate jq for formatting, or even convert between formats (eg: when one app writes in yaml, the other understands JSON input).

dvh 6y ago

https://github.com/hikikomori82/StructurOS https://github.com/hikikomori82/StructurOS

rootlocus 6y ago

> I think for this to be solved, the OS's primary communication system should be structured You mean the STDOUT format of all processes? I disagree. `npm list --json` already does what you want. Your only argument is that ` | jq '<obscure commands>' | json:from | where {deduped==false}` is too obscure. The solution is (probably) a simplified `jq` that has the query format you want. Problem solved. Keep it simple.

smitty1e 6y ago
I kinda like the mystical feeling of jq. As with haskell, it'll feel as though I've arrived somehow, should I achieve real understanding.
- frou_dh 6y ago
  To me it's a good example of Opportunity Cost. We could spend the time and effort memorising jq's special snowflake language, but that time and effort would almost certainly be better spent on something else. I deleted it a while back. Can't be bothered with it.
  - knome 6y ago
    >jq's special snowflake language If you'll bother condescending to such, you'll find one of the most interesting languages I've encountered in years. jq's syntax and execution model is beautiful. It has some rough edges, leave no doubt. Some function names, date handling, and the obtuseness of some of the errors it returns can be frustrating. But its data streaming layers of nested closures are an extremely terse, readable, and powerful way of expressing operations over structured data.
  - gen220 6y ago
    I usually use jq just to format the blob, and then some unholy combination of grep -A/-B, tr, and sed to get the output I want. It’s usually faster (for me) than relearning jq’s query language.
  - deleted 6y ago
    [deleted]
  - klibertp 6y ago
    > would almost certainly be better spent on something else. I'd would suggest to spend that time on learning how to effectively learn new languages. There's more than one way, and I guarantee that afterwards you'll conquer any given "snowflake" in no time at all.
    - frou_dh 6y ago
      I deliberately wrote "memorise" rather than learn in the usual sense, because there's a higher than normal degree of familiarity needed to get working code flowing from the fingers, when the venue is between a pair of single quotes in a shell.

schemy 6y ago

The problem, as always, is that a table isn't the natural output for a computer. It's either serialized text, or a graph. Unix tools work ok-ish for serialized text and you need general recursive functions to deal with graphs. Something that all shell like languages are extremely bad at.

cnity 6y ago

It doesn't have to be this black and white, in my opinion. The most common data structures I've interacted with on the command line are _newline separated_ or _JSON_. What if your shell allowed parsing of JSON natively? npm list --json | ({ json }) => json.somePackage.author.name What if it allowed parsing of XML natively, allowing instant scrapers to be written in a single command? curl www.example.com | ({ xml }) => xml.find('.importantInformation') > scraped_info.txt

guerrilla 6y ago
Yeah this is probably it right here. Just being able to parse various common formats would give a lot of the structure people are looking for. The builtins can just operate on that. Maybe it could optionally detect the formats too. If such a shell became popular, it would also incentivize developers adding common formats as outputs to more programs too.
- kbrazil 6y ago
  This is what jc[0] does. [0] https://github.com/kellyjonbrazil/jc https://github.com/kellyjonbrazil/jc (disclaimer: I am the author)
trenchgun 6y ago
Ooh that would be pretty damn sweet tbh.
jodrellblank 6y ago
Newline separated what? Basically all Linux config files are newline separated lines, but that won’t get you the equals separated data pairs, indented blocks, pretend-xml sections, ad-hoc boolean representations, INI-style sections, ad-hoc list-of-values style data out of them. > What if it allowed parsing of XML natively, allowing instant scrapers to be written in a single command? Powershell’s Invoke-RestMethod parses xml natively to a structured output; it’s useful but not enough to make XML fun or trivial to work with beyond root.node.foo.title simple scraping.
- 6y ago

david_draco 6y ago

> The native OS doesn't really embrace structured communication OS system calls are highly structured, down to the argument position and data type.

smitty1e 6y ago
Right, but the unix philosophy is that everything consumes and emits text. Having something more than a blob is great. Of course, there will be a performance tax.

hhas01 6y ago

What you want is tagged and/or typed pipes. Once a pipe declares how its data is encoded, you can build all kinds of modern tooling and high-level automation on top of that. The whole “But everything’s a text file!” is an outrageous lie, and always has been: just an excuse to cover up a massive omission in the design of the Unix file system (untyped, untagged “file” resources). Such egregious corner-cutting might have been understandable—even forgivable—back in K&R’s day, when they were bootstrapping C and Unix on hardware with less processing power than a modern microwave oven controller. But we’re decades on from that now and there’s no excuse for perpetuating it: we know how to build safe powerful modern static and dynamic type systems, and we’re spoiled rotten for the hardware resources to support it. Fifty years ago, K&R built C and Unix, in a cave, with scraps. Nowadays you can bootstrap a complete hardware stack for less than the cost of a 360, and there are modern kernels (L4) and systems language (Rust) already available for free off the shelf. So pick a target market and build the userland for that. All that’s missing is a modern-day Kernighan and Ritchie to step up and get on with it, so it’d be a sorry indictment of this generation if they do not exist.

gen220 6y ago
> untyped, untagged “file” resources I think the problem then, and the problem today, is that these “types” are a very very mobile target. How many implementations of JSON are there? XML? HTML? CSV? It doesn’t make sense to put these types in the operating system, or else you’ll start TypeError’s every time you download anything. Not to mention the Computational costs of parsing the entire structure of a document every time you edit it to verify that it’s “still of type X”. I’m not at all sold on the idea that categorizing files and streams in this way is a step forward (in the shell, at least). This is what applications are for: to define their own formats, parsers, and generators. The point of the operating system is not to be one of these applications, but to facilitate their development and use. IMO, typed files and streams would actually impede more than enable this work.
- hhas01 6y ago
  “It doesn’t make sense to put these types in the operating system” The OS doesn’t need to hardcode individual types, and to do so would defeat growth. It only has to provide a standard mechanism by which files can declare the type of data they contain, and executables can declare the type(s) of data they consume and/or produce. BeOS, for instance, tagged files with MIME types. The original Mac OS used “four-char codes”. Even DOS had file name extensions, as awful as that particular design choice was. Storing arbitrary data with no way to indicate how that data is encoded is rank insanity. Saying “it’s the app’s job to deal with encodings” is not wrong, but if apps have no way to know what encoding a given file was written in, how can they know how to decode it? Please do not rationalize away the longstanding and well-known deficiencies of Unix. These deficiencies do not aid progress; they hinder it. As do apologetics.

throwaway894345 6y ago

I don’t think you need an OS to pass structured objects, but rather simply a runtime that supports it, a la dotnet. Basically anything that can allow different programs to run and share memory as well as information about the shape of the objects in that shared memory. This can run on existing OSes and in the case of powershell, it already does. I think Go could probably do this as well via its plugin interface and runtime type system except that you aren’t allowed to unload plugins (so every program you ever run hangs out in memory).

hughw 6y ago

Microsoft went far down this road in the early 2000s with WinFS [1] but abandoned it. [Edit: WinFS and Longhorn together] [1] https://en.wikipedia.org/wiki/WinFS https://en.wikipedia.org/wiki/WinFS

gbear0 6y ago

I keep having the same thoughts, but I'm not sure it needs to be a full OS solution. Also a related problem that always frustrates me with input is what do I use for the help command? Is it '-h', '-help', '--help', or just 'help', or even man pages only. Each script/bin has its own way of doing things and this seems like something that should be standardized, but I realize there's no great way to do that currently when its just input to the program. I wonder if instead of an OS solution we can just add more metadata to the actual binary files we compile. For example, if we have a new ELF format that has an optional list of commands, options, full input/output types, etc, any one of the shells can start to provide better tooling. Output format types can then be passed along with the stream, and potentially checked that they work as the proper inputs for another command. Let the shell auto generate the simple cli help docs, or generate the bash completions, or figure out how to better convert data to desired types.

gregmac 6y ago

Something relatively simple would be borrowing the idea of the HTTP "accepts" header, and standardizing on an environment variable. For example: STDOUT_ACCEPT=application/json,application/csv,text/plain I'm not sure how the application could easily communicate back the understanding of this, though -- basically the equivalent of HTTP Content-Type header. It would need to be done in a way that doesn't break other applications that aren't expecting it, perhaps such as being sent back on STDERR with an escape sequence of sorts? The other trick is knowing what an application will accept via STDIN - ideally at runtime, but before actually writing anything. If the shell understands all this, it could automatically format JSON/csv output for display, integrate jq for formatting, or even convert between formats (eg: when one app writes in yaml, the other understands JSON input).