11 ms·

Amazing work! Between this, Alacritty [1], and coreutils [2], we're getting pretty close to a plausible all-Rust CLI stack. While Cicada is pretty clearly mode

Amazing work! Between this, Alacritty [1], and coreutils [2], we're getting pretty close to a plausible all-Rust CLI stack. While Cicada is pretty clearly modeled on the "old" generation of shells (sh, Bash, Zsh, etc.), one has to wonder what a more modern shell might look to address some of the problems of its predecessors like pipes that are essentially text-only, poor composability (see `cut` or `xargs`), and terribly obscure syntax. PowerShell seems to have been a flop outside of the Windows ecosystem, and between overly elaborate syntax and COM baggage, was probably rightfully passed over, but it probably had the right idea. Imagine a future where we're piping structured objects between programs, scripting in a modern language, and have IntelliSense quality auto-completion for every command. --- [1] https://github.com/jwilm/alacritty https://github.com/jwilm/alacritty [2] https://github.com/uutils/coreutils https://github.com/uutils/coreutils

zdkl 9y ago
My structured objects aren't your structured objects. In that use case why not use a dedicated program (like say ae python/js/clojure interpreter) to handle more complex pipes and keep the shell level primitves... Well, primitive. Text is compatible with all systems, past and present, and can be used to model more complex objects. Let's not add features to the core on a "why not" please.
- vog 9y ago
  The so-called "plain text formats" are also just representations of complex objects. Often these formats are ad-hoc and hard to parse correctly. Multiple text files in a file system hierarchy (e.g. /etc) are also just nested structures. So in principle, treating those as complex object structures is the right way to go. Also, I believe this is the idea behind D-Bus and similar modern Unix developments. However, what's hard is to provide good tooling with a simple syntax that is simple to understand: * Windows Registry and Power Shell show how not to do it. * The XML toolchain demonstrate that any unnecessary complexity in the meta-structure will haunt you through every bit of the toolchain (querying, validation/schema, etc.). * "jq" goes somewhat into the right direction for JSON files, but still hasn't found wide adoption. This appears to be a really hard design issue. Also, while deep hierarchies are easier to process by scripts, a human overview is mostly achieved by advances searching and tags rather than hierarchies. A shell needs to accomodate for both, but maybe we really just need better command line tools for simple hierarchial processing of arbitrary text file formats (ad-hoc configs, CSV, INI, JSON, YAML, XML, etc.).
  - zdkl 9y ago
    > in principle, treating those as complex object structures is the right way to go I do not see this as a given. It's a matter of abstraction vs performance tradeoff and that is highly subjective. Unless you pioneer a new standard form for complex object notation this'll just end up back to a format flamewar. (And if we go that way, I'd argue for s-exprs or -dare I say it- xml)
  - int_19h 9y ago
    Powershell does two things right: it uses structured output, and it separates producing the data from rendering the data. It also does one thing wrong: it uses objects (i.e. the stuff that carries behavior, not just state). This ties it to a particular object model, and the framework that supports that model. What's really needed is something simple that's data-centric, like JSON, but with a complete toolchain to define schemas and perform transformations, like XML (but without the warts and overengineering).
    - still_grokking 9y ago
      > What's really needed is something simple that's data-centric, like JSON, but with a complete toolchain to define schemas and perform transformations, like XML (but without the warts and overengineering). What would it be? Is there a real alternative to XML with those features out there? I don't think so. When you would want to have the features of XML and would design it from scratch I'm quite sure it would have the complexity of XML again. Usually implementing some functionality yields every time the same level of complexity regardless of how you implement it (given that none of the implementations isn't out right stupid of curse).
      - int_19h 9y ago
        > When you would want to have the features of XML and would design it from scratch I'm quite sure it would have the complexity of XML again. I don't think so. The problem with the XML stack is that it has been designed with some very "enterprisey" (for the lack of better term) scenarios in mind - stuff like SOAP. Consequently, it was all design by committee in the worst possible sense of the word, and it shows. To see what I mean, take a look at XML Schema W3C specs. That's probably the worst part of it, so it should be readily apparent what I mean: https://www.w3.org/TR/xmlschema11-1/ https://www.w3.org/TR/xmlschema11-1/ https://www.w3.org/TR/xmlschema11-2/ https://www.w3.org/TR/xmlschema11-2/ The other problem with XML is that it's rooted in SGML, and inherited a lot of its syntax and semantics, which were designed for a completely different use case - marking up documents. Consequently, the syntax is overly verbose, and some features are inconsistent for other scenarios - for example, if you use XML to describe structured data, when do you use attributes, and when do you use child elements? Don't forget that attributes are semantically unordered in XDM, while elements are ordered, but also that attributes cannot contain anything but scalar values and arrays thereof. Oh, and then don't forget all the legacy stuff like DTD, which is mostly redundant in the face of XML Schema and XInclude, except it's still a required part of the spec. I guess the TL;DR version of it is that XML today is kinda like Java - it was there for too long, including periods when our ideas of best practices were radically different, and all that was enshrined in the design, and then fossilized in the name of backwards compatibility. One important takeaway from XML - why it was so successful, IMO - is that having a coherent, a tightly bound spec stack is a good thing. For example, with XML, when someone is talking about schemas, you can pretty much assume it's XML Schema by default (yes, there's also RELAX NG, but I think calling it schema is a misnomer, because it doesn't delve much into semantics of what it describes - it's more of a grammar definition language for XML). To transfer XML, you use XSLT. To query it, you use XPath or XQuery (which is a strict superset). And so on. With JSON, there's no such certainty. The other thing that the XML stack didn't quite see fully through, but showed that it could be a nice thing, is its homoiconicity: e.g. XML Schema and XSLT being XML. Less so with XPath and XQuery, but there they had at least defined a canonical XML representation for it, which gives you most of the same advantages. Unfortunately, with XML it was just as often a curse as it was a blessing, because of how verbose and sometimes awkward its syntax is - anyone who wrote large amounts of XSLT especially knows what I'm talking about. On the other hand, at least XML had comments, unlike JSON! Hey, maybe that's actually the test case? A data representation language must be concise enough, powerful enough, and flexible enough to make it possible to use it to define its own schema and transformations, without it being a painful experience, while also being simple enough that a single person can write a parser for it in a reasonable amount of time.
- Sir_Cmpwn 9y ago
  This hits the mark right on the head. Instead of ruining the shell with unnecessary complexity because it doesn't fit every use case, how about knowing when not to use the shell?
- v-yadli 9y ago
  Likewise, your text objects aren't my text objects. For a fair comparison I think we should first assume compatibility for both unstructured, and structural data. Otherwise I would challenge that your bash script is not my bash script. The sole point of structural data is to turn the storage into a formal system, and enable more math operations on the entries, which align to the desired semantics. Python/js/clojure scripts extract some structural aspects from text, and then the story becomes the same as structural data processing.
Whil- 9y ago
It seems a bit premature to say that PowerShell was passed over. The journey outside Windows has just started. Indeed, PowerShell is onto something. Such delight it is to work with. Especially when you start to build your own tools for it.
- whatnotests 9y ago
  It's been 16 years.
  - coldtea 9y ago
    16 years that it has only been available for Windows, the parent says.
- TokenDiversity 9y ago
  Powershell made me actually switch to Windows. I was playing with WSL when it came out and learn of Powershell. WSL has some warts but Powershell means I cannot use Linux anymore :( I'm hoping Powershell on Linux will soon be at par with the Windows counterpart (right now it's pretty buggy)
vmarsy 9y ago
Powershell is available on Linux these days, it was in 'alpha' version for a long time, apparently it's now 'beta' https://github.com/PowerShell/PowerShell https://github.com/PowerShell/PowerShell
deleted 9y ago
[deleted]
mitnk 9y ago
You're right, I consider Cicada is the "old" generation shell. I intend to keep it simple (for speed etc). Like in readme, it won't introduce functions or other complex stuff. But still can add some feature for my own needs. For modern shell, please check out xonsh: http://xon.sh/ http://xon.sh/ - powered by Python - It's super cool!
vidarh 9y ago
The problem with modernising the shell stack is that there are a lot of things that depends on the current stack, be it the shell (e.g. see the amount of work it's taken for Ubuntu and Debian to switch to dash as /bin/sh; and dash has a pedigree going back to the 80's), or the upper layers (try to alias "ls" to something that acts differently, and see how many tools depend on parsing its output). You'd either have to be very careful (e.g. feature flags guarding every change) or expect a surprising amount of things to fail.
- sirn 9y ago
  This matches my experience trying to switch the login shell from bash to fish: a surprisingly amount of things failed. macOS users won't have this problem, as the login shell is only started when Terminal/iTerm is started, but in Linux pretty much every process rely on the fact that your $SHELL is POSIX-complaint. It seems like the way to go to use non-POSIX shell as the default shell without changing login shell, is to have .bashrc `exec`'d that shell when it is running in interactive mode. Regardless, I would really love to see bash-like shell that provides out of the box experience in the same level as fish.
  - gkya 9y ago
    How come? Having your user's shell sth. non-compliant should not affect other scripts, as they are supposed to specify what interpreter they wanna use via shebang lines; and as long as programs are run with the correct variant of the exec* functions they should be fine.
    - sirn 9y ago
      I meant the login shell, the one that get executed during console login (i.e. the "-bash" process). The login shell need to source /etc/profile and /etc/profile.d during logins to populate environment variables. Some apps that expect those environment variable to be present wrote to /etc/profile.d with an expectation that a POSIX-compatible shell and will read the directory (by source-ing those files). The environment variable getting set there ranging from PATH, LANG, to application/distro-specific like MOZ_PLUGIN_PATH. So when you changed the login shell to something that no longer reads /etc/profile.d (like fish, which actually say it's "interactive shell" in the man page) then application that rely on those variables will start to behave in some funny way, which was the point I was trying to make.
imron 9y ago
> like pipes that are essentially text-only Pipes work fine with binary data, you just need to be piping to a tool that works with binary data rather than text e.g. cat <somebinaryfile> | openssl base64
- twic 9y ago
  The trouble with that example is that it could more simply be written: openssl base64 <somebinaryfile On the plus side, you do get a prize for suggesting it [1] Something involving bulk binary data moving over a pipe which isn't an indirect redirection would be: gzip somefile | ssh user@host "gunzip >somefile" Although, again, there's a perfectly good -c flag to SSH that would do the same thing. Probably the most common example of tool-to-tool binary communication over pipes is: find -print0 | xargs -0 [1] http://porkmail.org/era/unix/award.html http://porkmail.org/era/unix/award.html
  - imron 9y ago
    > On the plus side, you do get a prize for suggesting it Good catch! I actually use another command that generates binary output that I pipe to OpenSSL to convert to base64 but rather than type out the full command I thought 'what's the easiest way to get some binary data on stdout' and 'cat binaryfile' was it.
ori_b 9y ago
> what a more modern shell might look to address some of the problems of its predecessors like pipes that are essentially text-only, poor composability (see `cut` or `xargs`) A new system and set of utilities. None of the Unix environment works in that world, and shoehorning it in would feel incredibly clunky.
- panic 9y ago
  If you use bytes that are invalid in UTF-8 (e.g. 0xF5-0xFF) as delimiters / structure characters, you can use text-only tools to do structured operations on UTF-8 strings without ever having to escape anything -- the structural "characters" you would need to escape can never appear in the encoded bytes.
  - ori_b 9y ago
    The Unix shell operates on binary streams, not on text.
    - panic 9y ago
      Ah, I thought this was about structured text (like JSON) vs. plain text -- binary content is a bigger issue!
      - ori_b 9y ago
        If you want to replace the shell with a structured one, the change needs to be pervasive, down to replacing the typical file formats, imo.
  - carussell 9y ago
    > If you use bytes that are invalid in UTF-8 (e.g. 0xF5-0xFF) as delimiters / structure characters No need for that. UTF-8 is a superset of ASCII, and ASCII already includes a handful of control codes that could be used here.
laumars 9y ago
The approach I took to my shell was to pass JSON objects about as the primary data type but fall back to text streams when the data doesn't look like JSON. The problem with passing objects around though is it can cause issues with the parallel nature of pipes. ie the programs in a chain of pipes can run concurrently since the data is being passed is just a stream. But with objects you need to read the object in its entirety to ensure it is a valid object. This means each process is waiting for the previous process to finish outputting it's object before the next can start processing it. While this is less of an issue with smaller chunks of data in a chain, it would become really noticeable really quickly as your data scales up. I'm sure will be workarounds for the above problem but my project is still young so there are other bugs I'm more focused on (for me I'm more interested in getting an IDE-level auto-complete implemented so writing one liners in the Shell is as natural as writing your favourite language in your preferred development tool.
- still_grokking 9y ago
  The problems could be tackled nicely by a clean functional solution like Iteratees[1], maybe with a hight-level API like Reactive Streams[2]. [1] https://github.com/playframework/play-iteratees https://github.com/playframework/play-iteratees [2] http://www.reactive-streams.org/ http://www.reactive-streams.org/
  - laumars 9y ago
    Thank you, I will take a read of them.
ofek 9y ago
You may be interested in https://github.com/xonsh/xonsh https://github.com/xonsh/xonsh
zokier 9y ago
What sort of features do you feel that shell needs to support structured data? I've toyed with this idea, and in my view the shell itself has relatively minor role here. What is needed are utilities that produce and process structured data, and a terminal that can display it. But shell, it just glues the pieces together and doesn't really need to be aware of the data and the structure of it.
- brandur 9y ago
  Here's a very simple example that I'd consider the holy grail. Say I want run `ls -lh` on a directory: -rw-r--r-- 1 brandur staff 6.5K Mar 25 15:37 Makefile -rw-r--r-- 1 brandur staff 63B Jul 10 2016 Procfile -rw-r--r-- 1 brandur staff 1.6K Mar 11 07:20 README.md drwxr-xr-x 4 brandur staff 136B Oct 16 2016 assets/ drwxr-xr-x 4 brandur staff 136B Jul 15 2016 atom/ drwxr-xr-x 4 brandur staff 136B Apr 26 2016 cmd/ I'm interested in getting the date fields out of this structure. Currently that's possible with some disgusting use of `awk` or maybe `cut`. What if I could do something like `ls -lh | select 6` (select the 6th column of data): Mar 25 15:37 Jul 10 2016 Mar 11 07:20 Oct 16 2016 Jul 15 2016 Apr 26 2016 It doesn't matter that the date comes in three parts and would act as three separate tokens for `awk` and `cut` because in my structured object backend, it's all just one logical timestamp. Now say I want to sort these dates and pick the earliest. Once again, shells make this extremely difficult. Because the dates are not lexigraphically orderable, I'd have to use some fancy parsing, or try to step back to `ls` and have it somehow order by date. Imagine if I could instead to something like `ls -lh | select 6 | sort | first`: Apr 26 2016 In this case, although the shell is still displaying a string to me that's not lexigraphically orderable, the data is being passed around in the backend in a structured way and these timestamps are actually real timestamps. It's then trivial for the next utility to apply ordering. The part that the shell would have to provide (as opposed to the programs) is a "smart pipe" that understands more than just a basic text stream. It would also have to know when there is no pipe connected, and print as pretty strings when it knows the output is going to `STDOUT`.
  - lihaoyi 9y ago
    Not quite a "Unix" shell, but the [Ammonite Scala Shell](http://ammonite.io/#Ammonite-Shell http://ammonite.io/#Ammonite-Shell) lets you do this trivially: lihaoyi Ammonite$ amm @ import ammonite.ops._ import ammonite.ops._ @ ls! pwd res1: LsSeq = ".git" 'LICENSE 'ci 'project 'sshd ".gitignore" 'amm 'integration 'readme 'target ".idea" "appveyor.yml" "internals-docs" "readme.md" 'terminal ".travis.yml" "build.sbt" 'ops 'shell @ ls! pwd | (_.mtime) res2: Seq[java.nio.file.attribute.FileTime] = List( 2017-06-21T12:24:41Z, 2017-06-11T13:48:45Z, 2017-06-21T12:29:05Z, 2017-06-11T13:48:45Z, 2017-05-01T03:41:14Z, 2017-06-18T14:06:33Z, 2017-06-11T13:48:45Z, 2017-06-18T08:01:20Z, 2017-06-18T12:34:00Z, 2017-06-19T06:05:51Z, 2017-06-18T09:06:07Z, 2017-06-18T14:06:33Z, 2017-06-18T14:06:15Z, 2017-06-18T14:07:20Z, 2017-06-11T13:48:45Z, 2017-06-18T14:06:33Z, 2017-06-18T14:06:33Z, 2017-06-19T06:06:03Z, 2017-06-18T14:06:33Z ) @ ls! pwd | (_.mtime) sorted res3: Seq[java.nio.file.attribute.FileTime] = List( 2017-05-01T03:41:14Z, 2017-06-11T13:48:45Z, 2017-06-11T13:48:45Z, 2017-06-11T13:48:45Z, 2017-06-11T13:48:45Z, 2017-06-18T08:01:20Z, 2017-06-18T09:06:07Z, 2017-06-18T12:34:00Z, 2017-06-18T14:06:15Z, 2017-06-18T14:06:33Z, 2017-06-18T14:06:33Z, 2017-06-18T14:06:33Z, 2017-06-18T14:06:33Z, 2017-06-18T14:06:33Z, 2017-06-18T14:07:20Z, 2017-06-19T06:05:51Z, 2017-06-19T06:06:03Z, 2017-06-21T12:24:41Z, 2017-06-21T12:29:05Z ) @ ls! pwd | (_.mtime) min res4: java.nio.file.attribute.FileTime = 2017-05-01T03:41:14Z @ ls! pwd | (_.mtime) max res5: java.nio.file.attribute.FileTime = 2017-06-21T12:29:05Z @ ls! pwd maxBy (_.mtime) res6: Path = root/'Users/'lihaoyi/'Dropbox/'Github/'Ammonite/".idea" Ammonite certainly has problems around JVM memory usage and startup times, but it has a sweet spot for this sort of not-quite-trivial filesystem operations that are a pain to do in Bash but too small to be worth futzing with a Python script
naasking 9y ago
A modern shell with properly structured data streams would look like caml-shcaml: http://users.eecs.northwestern.edu/~jesse/pubs/caml-shcaml/ http://users.eecs.northwestern.edu/~jesse/pubs/caml-shcaml/ PowerShell definitely made a lot of mistakes they should have known needed fixing. Like the need for signed scripts.
raarts 9y ago
Not to be condescending, but looking at the feature set, this has a long way to go before becoming a viable bash replacement. I don't see the amazing aspect to be frank.
- Drdrdrq 9y ago
  Security? I would love to see whole classes of bug eradicated (stack overflow, use after free, memory leaks,...)
vram22 9y ago
>Imagine a future where we're piping structured objects between programs, scripting in a modern language May not be quite the same as what you mean, but check out osh (a Python tool, a sort of shell, for doing distributed command-line and scripting operations on clusters of nodes). I had blogged about osh briefly here: Some ways of doing UNIX-style pipes in Python: https://jugad2.blogspot.in/2011/09/some-ways-of-doing-unix-style-pipes-in.html https://jugad2.blogspot.in/2011/09/some-ways-of-doing-unix-s... Interestingly, later on, when I blogged this: Akiban, new database, supported by SQLAlchemy: https://jugad2.blogspot.in/2012/10/akiban-new-database-supported-by.html https://jugad2.blogspot.in/2012/10/akiban-new-database-suppo... the creator of osh, Jack Orenstein, who was then at Akiban (the company behind the Akiban product), commented on that post, giving more details about both osh and what it was used for at Archivas, later acquired by Hitachi Data Systems, and also said something about Akiban. And now I just saw by googling that it (Akiban) was acquired by FoundationDB.
throwaway91111 9y ago
Personally the only structured data i have an urge to work with pipes are json via jq. However, you should check out fish; it improves on bash/zsh/whatever syntax in sane ways.
gcr 9y ago
In terms of interactive session,s I’ve always wanted an asynchronous shell thta gave me a prompt back as soon as it started executing the command. It would have real job control, letting me see what was running and page through the output separately. Idk.
skywhopper 9y ago
I certainly did like the idea behind Powershell's structured-object pipelines, but it relies on that "COM baggage" and "elaborate syntax" that you say are its downsides. I don't think you can achieve the goal behind Powershell of a universal standard binary object format without an elaborate infrastructure like COM and without making a bunch of assumptions and requirements on how those objects must look and behave, which will vastly limit the usefulness of the pipeline. All of that said... nothing is stopping you from piping binary objects between tools in Unix shells. In fact, it works great, and I'm curious what functionality you are looking for that doesn't exist. The standard tools like cut and grep don't work great with binary data, but they aren't meant to. There are format-specific tools aplenty, and stuff like bbe exist for generic work. And for structured text, tools like `jq` are phenomenal.
elcritch 9y ago
One thing lacking in traditional shell utilities is difficulty having a universally understood way of passing metadata about the content of the pipes/streams. Other commentators have (rightly!) pointed out the difficulties of requiring a single "structured object" in the core of the ecosystem... but if there were a way to gracefully support multiple structured object formats and let tooling build on top of that. Got me wondering what would happen if we embedded modified form of HTTP response headers into the beginning of every UNIX command line stream. Let's tag this version `HTTP/UNIX.1`. This is to differentiate that this HTTP isn't due to a request, but as part of a standard 'UNIX' command response. Otherwise programs could follow RFC2616 section for HTTP response formats [1]. Speaking to the existing ecosystem of tooling, it'd be straightforward (trivial?) to add native support into existing shells to support this. For example now when you run `ls` your shell settings would be `SHELL_DISPLAY_HTTP_UNIX_PRETTY="terminal/text:terminal/json"`. Programs without native support built in could be handled by adding a "http_strip" program. Otherwise adding support into command line programs would be simple as libraries to handle HTTP/1.1 exist in most any language/platform and only a the "Response" header section would be needed of those. Alternately a generic wrapper could created to handle the http response info. Examples: HTTP/UNIX.1 200 OK Date: Mon, 27 Jul 2009 12:28:53 GMT Server: ifconfig lo0 Content-Length: 248 Content-Type: terminal/text Connection: Closed lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384 options=1203<RXCSUM,TXCSUM,TXSTATUS,SW_TIMESTAMP> inet 127.0.0.1 netmask 0xff000000 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 nd6 options=201<PERFORMNUD,DAD> Next: HTTP/UNIX.1 200 OK Date: Mon, 27 Jul 2009 12:28:53 GMT Server: sysinfo Content-Length: 107 Content-Type: terminal/json Connection: Closed { "aggr0": { "LACP mode": "active", "Interfaces": [ "e1000g1", "e1000g0" ] } } Given this system could be in place, many tools could start adopting "json" or "messagepack" or other structured formats and generic tools could be built to translate in between formats. This is exacerbated now in that there's no way for programs to say "I'm outputting JSON" and "My json is using this json-schema://unix/some/random/filedescription/system". Or tools like `xargs` and `find` could attach header metadata of "Null-Terminator: \0" or "Null-Terminator: \n". Currently shell tools rely on command flags to set that, but if that feature flag could also be set by "Null-Terminator" header flags in the input/output streams it'd make the shell a lot more intuitive without reduce visibility. For example a common `find -print0 /etc | header_inspect` could be used to override shell settings and print out the header. 1: https://www.w3.org/Protocols/rfc2616/rfc2616-sec6.html https://www.w3.org/Protocols/rfc2616/rfc2616-sec6.html edit: formatting whitespace?
johnny22 9y ago
hotwire tried it too. it's been dead for awhile though. https://code.google.com/archive/p/hotwire-shell/wikis https://code.google.com/archive/p/hotwire-shell/wikis
slowmovintarget 9y ago
You've remindeded me of this talk: https://www.destroyallsoftware.com/talks/a-whole-new-world https://www.destroyallsoftware.com/talks/a-whole-new-world Near the end, he starts talking about terminals and making a new terminal standard (~ 17:30 mins in).

zdkl 9y ago

My structured objects aren't your structured objects. In that use case why not use a dedicated program (like say ae python/js/clojure interpreter) to handle more complex pipes and keep the shell level primitves... Well, primitive. Text is compatible with all systems, past and present, and can be used to model more complex objects. Let's not add features to the core on a "why not" please.

vog 9y ago
The so-called "plain text formats" are also just representations of complex objects. Often these formats are ad-hoc and hard to parse correctly. Multiple text files in a file system hierarchy (e.g. /etc) are also just nested structures. So in principle, treating those as complex object structures is the right way to go. Also, I believe this is the idea behind D-Bus and similar modern Unix developments. However, what's hard is to provide good tooling with a simple syntax that is simple to understand: * Windows Registry and Power Shell show how not to do it. * The XML toolchain demonstrate that any unnecessary complexity in the meta-structure will haunt you through every bit of the toolchain (querying, validation/schema, etc.). * "jq" goes somewhat into the right direction for JSON files, but still hasn't found wide adoption. This appears to be a really hard design issue. Also, while deep hierarchies are easier to process by scripts, a human overview is mostly achieved by advances searching and tags rather than hierarchies. A shell needs to accomodate for both, but maybe we really just need better command line tools for simple hierarchial processing of arbitrary text file formats (ad-hoc configs, CSV, INI, JSON, YAML, XML, etc.).
- zdkl 9y ago
  > in principle, treating those as complex object structures is the right way to go I do not see this as a given. It's a matter of abstraction vs performance tradeoff and that is highly subjective. Unless you pioneer a new standard form for complex object notation this'll just end up back to a format flamewar. (And if we go that way, I'd argue for s-exprs or -dare I say it- xml)
- int_19h 9y ago
  Powershell does two things right: it uses structured output, and it separates producing the data from rendering the data. It also does one thing wrong: it uses objects (i.e. the stuff that carries behavior, not just state). This ties it to a particular object model, and the framework that supports that model. What's really needed is something simple that's data-centric, like JSON, but with a complete toolchain to define schemas and perform transformations, like XML (but without the warts and overengineering).
  - still_grokking 9y ago
    > What's really needed is something simple that's data-centric, like JSON, but with a complete toolchain to define schemas and perform transformations, like XML (but without the warts and overengineering). What would it be? Is there a real alternative to XML with those features out there? I don't think so. When you would want to have the features of XML and would design it from scratch I'm quite sure it would have the complexity of XML again. Usually implementing some functionality yields every time the same level of complexity regardless of how you implement it (given that none of the implementations isn't out right stupid of curse).
    - int_19h 9y ago
      > When you would want to have the features of XML and would design it from scratch I'm quite sure it would have the complexity of XML again. I don't think so. The problem with the XML stack is that it has been designed with some very "enterprisey" (for the lack of better term) scenarios in mind - stuff like SOAP. Consequently, it was all design by committee in the worst possible sense of the word, and it shows. To see what I mean, take a look at XML Schema W3C specs. That's probably the worst part of it, so it should be readily apparent what I mean: https://www.w3.org/TR/xmlschema11-1/ https://www.w3.org/TR/xmlschema11-1/ https://www.w3.org/TR/xmlschema11-2/ https://www.w3.org/TR/xmlschema11-2/ The other problem with XML is that it's rooted in SGML, and inherited a lot of its syntax and semantics, which were designed for a completely different use case - marking up documents. Consequently, the syntax is overly verbose, and some features are inconsistent for other scenarios - for example, if you use XML to describe structured data, when do you use attributes, and when do you use child elements? Don't forget that attributes are semantically unordered in XDM, while elements are ordered, but also that attributes cannot contain anything but scalar values and arrays thereof. Oh, and then don't forget all the legacy stuff like DTD, which is mostly redundant in the face of XML Schema and XInclude, except it's still a required part of the spec. I guess the TL;DR version of it is that XML today is kinda like Java - it was there for too long, including periods when our ideas of best practices were radically different, and all that was enshrined in the design, and then fossilized in the name of backwards compatibility. One important takeaway from XML - why it was so successful, IMO - is that having a coherent, a tightly bound spec stack is a good thing. For example, with XML, when someone is talking about schemas, you can pretty much assume it's XML Schema by default (yes, there's also RELAX NG, but I think calling it schema is a misnomer, because it doesn't delve much into semantics of what it describes - it's more of a grammar definition language for XML). To transfer XML, you use XSLT. To query it, you use XPath or XQuery (which is a strict superset). And so on. With JSON, there's no such certainty. The other thing that the XML stack didn't quite see fully through, but showed that it could be a nice thing, is its homoiconicity: e.g. XML Schema and XSLT being XML. Less so with XPath and XQuery, but there they had at least defined a canonical XML representation for it, which gives you most of the same advantages. Unfortunately, with XML it was just as often a curse as it was a blessing, because of how verbose and sometimes awkward its syntax is - anyone who wrote large amounts of XSLT especially knows what I'm talking about. On the other hand, at least XML had comments, unlike JSON! Hey, maybe that's actually the test case? A data representation language must be concise enough, powerful enough, and flexible enough to make it possible to use it to define its own schema and transformations, without it being a painful experience, while also being simple enough that a single person can write a parser for it in a reasonable amount of time.
Sir_Cmpwn 9y ago
This hits the mark right on the head. Instead of ruining the shell with unnecessary complexity because it doesn't fit every use case, how about knowing when not to use the shell?
v-yadli 9y ago
Likewise, your text objects aren't my text objects. For a fair comparison I think we should first assume compatibility for both unstructured, and structural data. Otherwise I would challenge that your bash script is not my bash script. The sole point of structural data is to turn the storage into a formal system, and enable more math operations on the entries, which align to the desired semantics. Python/js/clojure scripts extract some structural aspects from text, and then the story becomes the same as structural data processing.

Whil- 9y ago

It seems a bit premature to say that PowerShell was passed over. The journey outside Windows has just started. Indeed, PowerShell is onto something. Such delight it is to work with. Especially when you start to build your own tools for it.

whatnotests 9y ago
It's been 16 years.
- coldtea 9y ago
  16 years that it has only been available for Windows, the parent says.
TokenDiversity 9y ago
Powershell made me actually switch to Windows. I was playing with WSL when it came out and learn of Powershell. WSL has some warts but Powershell means I cannot use Linux anymore :( I'm hoping Powershell on Linux will soon be at par with the Windows counterpart (right now it's pretty buggy)

vmarsy 9y ago

Powershell is available on Linux these days, it was in 'alpha' version for a long time, apparently it's now 'beta' https://github.com/PowerShell/PowerShell https://github.com/PowerShell/PowerShell

deleted 9y ago

[deleted]

mitnk 9y ago

You're right, I consider Cicada is the "old" generation shell. I intend to keep it simple (for speed etc). Like in readme, it won't introduce functions or other complex stuff. But still can add some feature for my own needs. For modern shell, please check out xonsh: http://xon.sh/ http://xon.sh/ - powered by Python - It's super cool!

vidarh 9y ago

The problem with modernising the shell stack is that there are a lot of things that depends on the current stack, be it the shell (e.g. see the amount of work it's taken for Ubuntu and Debian to switch to dash as /bin/sh; and dash has a pedigree going back to the 80's), or the upper layers (try to alias "ls" to something that acts differently, and see how many tools depend on parsing its output). You'd either have to be very careful (e.g. feature flags guarding every change) or expect a surprising amount of things to fail.

sirn 9y ago
This matches my experience trying to switch the login shell from bash to fish: a surprisingly amount of things failed. macOS users won't have this problem, as the login shell is only started when Terminal/iTerm is started, but in Linux pretty much every process rely on the fact that your $SHELL is POSIX-complaint. It seems like the way to go to use non-POSIX shell as the default shell without changing login shell, is to have .bashrc `exec`'d that shell when it is running in interactive mode. Regardless, I would really love to see bash-like shell that provides out of the box experience in the same level as fish.
- gkya 9y ago
  How come? Having your user's shell sth. non-compliant should not affect other scripts, as they are supposed to specify what interpreter they wanna use via shebang lines; and as long as programs are run with the correct variant of the exec* functions they should be fine.
  - sirn 9y ago
    I meant the login shell, the one that get executed during console login (i.e. the "-bash" process). The login shell need to source /etc/profile and /etc/profile.d during logins to populate environment variables. Some apps that expect those environment variable to be present wrote to /etc/profile.d with an expectation that a POSIX-compatible shell and will read the directory (by source-ing those files). The environment variable getting set there ranging from PATH, LANG, to application/distro-specific like MOZ_PLUGIN_PATH. So when you changed the login shell to something that no longer reads /etc/profile.d (like fish, which actually say it's "interactive shell" in the man page) then application that rely on those variables will start to behave in some funny way, which was the point I was trying to make.

imron 9y ago

> like pipes that are essentially text-only Pipes work fine with binary data, you just need to be piping to a tool that works with binary data rather than text e.g. cat <somebinaryfile> | openssl base64

twic 9y ago
The trouble with that example is that it could more simply be written: openssl base64 <somebinaryfile On the plus side, you do get a prize for suggesting it [1] Something involving bulk binary data moving over a pipe which isn't an indirect redirection would be: gzip somefile | ssh user@host "gunzip >somefile" Although, again, there's a perfectly good -c flag to SSH that would do the same thing. Probably the most common example of tool-to-tool binary communication over pipes is: find -print0 | xargs -0 [1] http://porkmail.org/era/unix/award.html http://porkmail.org/era/unix/award.html
- imron 9y ago
  > On the plus side, you do get a prize for suggesting it Good catch! I actually use another command that generates binary output that I pipe to OpenSSL to convert to base64 but rather than type out the full command I thought 'what's the easiest way to get some binary data on stdout' and 'cat binaryfile' was it.

ori_b 9y ago

> what a more modern shell might look to address some of the problems of its predecessors like pipes that are essentially text-only, poor composability (see `cut` or `xargs`) A new system and set of utilities. None of the Unix environment works in that world, and shoehorning it in would feel incredibly clunky.

panic 9y ago
If you use bytes that are invalid in UTF-8 (e.g. 0xF5-0xFF) as delimiters / structure characters, you can use text-only tools to do structured operations on UTF-8 strings without ever having to escape anything -- the structural "characters" you would need to escape can never appear in the encoded bytes.
- ori_b 9y ago
  The Unix shell operates on binary streams, not on text.
  - panic 9y ago
    Ah, I thought this was about structured text (like JSON) vs. plain text -- binary content is a bigger issue!
    - ori_b 9y ago
      If you want to replace the shell with a structured one, the change needs to be pervasive, down to replacing the typical file formats, imo.
- carussell 9y ago
  > If you use bytes that are invalid in UTF-8 (e.g. 0xF5-0xFF) as delimiters / structure characters No need for that. UTF-8 is a superset of ASCII, and ASCII already includes a handful of control codes that could be used here.

laumars 9y ago

The approach I took to my shell was to pass JSON objects about as the primary data type but fall back to text streams when the data doesn't look like JSON. The problem with passing objects around though is it can cause issues with the parallel nature of pipes. ie the programs in a chain of pipes can run concurrently since the data is being passed is just a stream. But with objects you need to read the object in its entirety to ensure it is a valid object. This means each process is waiting for the previous process to finish outputting it's object before the next can start processing it. While this is less of an issue with smaller chunks of data in a chain, it would become really noticeable really quickly as your data scales up. I'm sure will be workarounds for the above problem but my project is still young so there are other bugs I'm more focused on (for me I'm more interested in getting an IDE-level auto-complete implemented so writing one liners in the Shell is as natural as writing your favourite language in your preferred development tool.

still_grokking 9y ago
The problems could be tackled nicely by a clean functional solution like Iteratees[1], maybe with a hight-level API like Reactive Streams[2]. [1] https://github.com/playframework/play-iteratees https://github.com/playframework/play-iteratees [2] http://www.reactive-streams.org/ http://www.reactive-streams.org/
- laumars 9y ago
  Thank you, I will take a read of them.

ofek 9y ago

You may be interested in https://github.com/xonsh/xonsh https://github.com/xonsh/xonsh

zokier 9y ago

What sort of features do you feel that shell needs to support structured data? I've toyed with this idea, and in my view the shell itself has relatively minor role here. What is needed are utilities that produce and process structured data, and a terminal that can display it. But shell, it just glues the pieces together and doesn't really need to be aware of the data and the structure of it.

brandur 9y ago
Here's a very simple example that I'd consider the holy grail. Say I want run `ls -lh` on a directory: -rw-r--r-- 1 brandur staff 6.5K Mar 25 15:37 Makefile -rw-r--r-- 1 brandur staff 63B Jul 10 2016 Procfile -rw-r--r-- 1 brandur staff 1.6K Mar 11 07:20 README.md drwxr-xr-x 4 brandur staff 136B Oct 16 2016 assets/ drwxr-xr-x 4 brandur staff 136B Jul 15 2016 atom/ drwxr-xr-x 4 brandur staff 136B Apr 26 2016 cmd/ I'm interested in getting the date fields out of this structure. Currently that's possible with some disgusting use of `awk` or maybe `cut`. What if I could do something like `ls -lh | select 6` (select the 6th column of data): Mar 25 15:37 Jul 10 2016 Mar 11 07:20 Oct 16 2016 Jul 15 2016 Apr 26 2016 It doesn't matter that the date comes in three parts and would act as three separate tokens for `awk` and `cut` because in my structured object backend, it's all just one logical timestamp. Now say I want to sort these dates and pick the earliest. Once again, shells make this extremely difficult. Because the dates are not lexigraphically orderable, I'd have to use some fancy parsing, or try to step back to `ls` and have it somehow order by date. Imagine if I could instead to something like `ls -lh | select 6 | sort | first`: Apr 26 2016 In this case, although the shell is still displaying a string to me that's not lexigraphically orderable, the data is being passed around in the backend in a structured way and these timestamps are actually real timestamps. It's then trivial for the next utility to apply ordering. The part that the shell would have to provide (as opposed to the programs) is a "smart pipe" that understands more than just a basic text stream. It would also have to know when there is no pipe connected, and print as pretty strings when it knows the output is going to `STDOUT`.
- lihaoyi 9y ago
  Not quite a "Unix" shell, but the [Ammonite Scala Shell](http://ammonite.io/#Ammonite-Shell http://ammonite.io/#Ammonite-Shell) lets you do this trivially: lihaoyi Ammonite$ amm @ import ammonite.ops._ import ammonite.ops._ @ ls! pwd res1: LsSeq = ".git" 'LICENSE 'ci 'project 'sshd ".gitignore" 'amm 'integration 'readme 'target ".idea" "appveyor.yml" "internals-docs" "readme.md" 'terminal ".travis.yml" "build.sbt" 'ops 'shell @ ls! pwd | (_.mtime) res2: Seq[java.nio.file.attribute.FileTime] = List( 2017-06-21T12:24:41Z, 2017-06-11T13:48:45Z, 2017-06-21T12:29:05Z, 2017-06-11T13:48:45Z, 2017-05-01T03:41:14Z, 2017-06-18T14:06:33Z, 2017-06-11T13:48:45Z, 2017-06-18T08:01:20Z, 2017-06-18T12:34:00Z, 2017-06-19T06:05:51Z, 2017-06-18T09:06:07Z, 2017-06-18T14:06:33Z, 2017-06-18T14:06:15Z, 2017-06-18T14:07:20Z, 2017-06-11T13:48:45Z, 2017-06-18T14:06:33Z, 2017-06-18T14:06:33Z, 2017-06-19T06:06:03Z, 2017-06-18T14:06:33Z ) @ ls! pwd | (_.mtime) sorted res3: Seq[java.nio.file.attribute.FileTime] = List( 2017-05-01T03:41:14Z, 2017-06-11T13:48:45Z, 2017-06-11T13:48:45Z, 2017-06-11T13:48:45Z, 2017-06-11T13:48:45Z, 2017-06-18T08:01:20Z, 2017-06-18T09:06:07Z, 2017-06-18T12:34:00Z, 2017-06-18T14:06:15Z, 2017-06-18T14:06:33Z, 2017-06-18T14:06:33Z, 2017-06-18T14:06:33Z, 2017-06-18T14:06:33Z, 2017-06-18T14:06:33Z, 2017-06-18T14:07:20Z, 2017-06-19T06:05:51Z, 2017-06-19T06:06:03Z, 2017-06-21T12:24:41Z, 2017-06-21T12:29:05Z ) @ ls! pwd | (_.mtime) min res4: java.nio.file.attribute.FileTime = 2017-05-01T03:41:14Z @ ls! pwd | (_.mtime) max res5: java.nio.file.attribute.FileTime = 2017-06-21T12:29:05Z @ ls! pwd maxBy (_.mtime) res6: Path = root/'Users/'lihaoyi/'Dropbox/'Github/'Ammonite/".idea" Ammonite certainly has problems around JVM memory usage and startup times, but it has a sweet spot for this sort of not-quite-trivial filesystem operations that are a pain to do in Bash but too small to be worth futzing with a Python script

naasking 9y ago

A modern shell with properly structured data streams would look like caml-shcaml: http://users.eecs.northwestern.edu/~jesse/pubs/caml-shcaml/ http://users.eecs.northwestern.edu/~jesse/pubs/caml-shcaml/ PowerShell definitely made a lot of mistakes they should have known needed fixing. Like the need for signed scripts.

raarts 9y ago

Not to be condescending, but looking at the feature set, this has a long way to go before becoming a viable bash replacement. I don't see the amazing aspect to be frank.

Drdrdrq 9y ago
Security? I would love to see whole classes of bug eradicated (stack overflow, use after free, memory leaks,...)

vram22 9y ago

>Imagine a future where we're piping structured objects between programs, scripting in a modern language May not be quite the same as what you mean, but check out osh (a Python tool, a sort of shell, for doing distributed command-line and scripting operations on clusters of nodes). I had blogged about osh briefly here: Some ways of doing UNIX-style pipes in Python: https://jugad2.blogspot.in/2011/09/some-ways-of-doing-unix-style-pipes-in.html https://jugad2.blogspot.in/2011/09/some-ways-of-doing-unix-s... Interestingly, later on, when I blogged this: Akiban, new database, supported by SQLAlchemy: https://jugad2.blogspot.in/2012/10/akiban-new-database-supported-by.html https://jugad2.blogspot.in/2012/10/akiban-new-database-suppo... the creator of osh, Jack Orenstein, who was then at Akiban (the company behind the Akiban product), commented on that post, giving more details about both osh and what it was used for at Archivas, later acquired by Hitachi Data Systems, and also said something about Akiban. And now I just saw by googling that it (Akiban) was acquired by FoundationDB.

throwaway91111 9y ago

Personally the only structured data i have an urge to work with pipes are json via jq. However, you should check out fish; it improves on bash/zsh/whatever syntax in sane ways.

gcr 9y ago

In terms of interactive session,s I’ve always wanted an asynchronous shell thta gave me a prompt back as soon as it started executing the command. It would have real job control, letting me see what was running and page through the output separately. Idk.

skywhopper 9y ago

I certainly did like the idea behind Powershell's structured-object pipelines, but it relies on that "COM baggage" and "elaborate syntax" that you say are its downsides. I don't think you can achieve the goal behind Powershell of a universal standard binary object format without an elaborate infrastructure like COM and without making a bunch of assumptions and requirements on how those objects must look and behave, which will vastly limit the usefulness of the pipeline. All of that said... nothing is stopping you from piping binary objects between tools in Unix shells. In fact, it works great, and I'm curious what functionality you are looking for that doesn't exist. The standard tools like cut and grep don't work great with binary data, but they aren't meant to. There are format-specific tools aplenty, and stuff like bbe exist for generic work. And for structured text, tools like `jq` are phenomenal.

elcritch 9y ago

One thing lacking in traditional shell utilities is difficulty having a universally understood way of passing metadata about the content of the pipes/streams. Other commentators have (rightly!) pointed out the difficulties of requiring a single "structured object" in the core of the ecosystem... but if there were a way to gracefully support multiple structured object formats and let tooling build on top of that. Got me wondering what would happen if we embedded modified form of HTTP response headers into the beginning of every UNIX command line stream. Let's tag this version `HTTP/UNIX.1`. This is to differentiate that this HTTP isn't due to a request, but as part of a standard 'UNIX' command response. Otherwise programs could follow RFC2616 section for HTTP response formats [1]. Speaking to the existing ecosystem of tooling, it'd be straightforward (trivial?) to add native support into existing shells to support this. For example now when you run `ls` your shell settings would be `SHELL_DISPLAY_HTTP_UNIX_PRETTY="terminal/text:terminal/json"`. Programs without native support built in could be handled by adding a "http_strip" program. Otherwise adding support into command line programs would be simple as libraries to handle HTTP/1.1 exist in most any language/platform and only a the "Response" header section would be needed of those. Alternately a generic wrapper could created to handle the http response info. Examples: HTTP/UNIX.1 200 OK Date: Mon, 27 Jul 2009 12:28:53 GMT Server: ifconfig lo0 Content-Length: 248 Content-Type: terminal/text Connection: Closed lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384 options=1203<RXCSUM,TXCSUM,TXSTATUS,SW_TIMESTAMP> inet 127.0.0.1 netmask 0xff000000 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 nd6 options=201<PERFORMNUD,DAD> Next: HTTP/UNIX.1 200 OK Date: Mon, 27 Jul 2009 12:28:53 GMT Server: sysinfo Content-Length: 107 Content-Type: terminal/json Connection: Closed { "aggr0": { "LACP mode": "active", "Interfaces": [ "e1000g1", "e1000g0" ] } } Given this system could be in place, many tools could start adopting "json" or "messagepack" or other structured formats and generic tools could be built to translate in between formats. This is exacerbated now in that there's no way for programs to say "I'm outputting JSON" and "My json is using this json-schema://unix/some/random/filedescription/system". Or tools like `xargs` and `find` could attach header metadata of "Null-Terminator: \0" or "Null-Terminator: \n". Currently shell tools rely on command flags to set that, but if that feature flag could also be set by "Null-Terminator" header flags in the input/output streams it'd make the shell a lot more intuitive without reduce visibility. For example a common `find -print0 /etc | header_inspect` could be used to override shell settings and print out the header. 1: https://www.w3.org/Protocols/rfc2616/rfc2616-sec6.html https://www.w3.org/Protocols/rfc2616/rfc2616-sec6.html edit: formatting whitespace?

johnny22 9y ago

hotwire tried it too. it's been dead for awhile though. https://code.google.com/archive/p/hotwire-shell/wikis https://code.google.com/archive/p/hotwire-shell/wikis

slowmovintarget 9y ago

You've remindeded me of this talk: https://www.destroyallsoftware.com/talks/a-whole-new-world https://www.destroyallsoftware.com/talks/a-whole-new-world Near the end, he starts talking about terminals and making a new terminal standard (~ 17:30 mins in).