10 ms·

Ambitions for a Unix Shell

116 points by luu 7y ago

boona 7y ago
>What languages and tools does Oil aim to "replace"? >bash, awk, make, Python 2 What will in fact happen: It says here on your resume that you know bash, awk, make, and python 2, but do you know Oil as well?
- chubot 7y ago
  "replace" is in quotes for a reason :) The rest of the post gives some more color on it.
jerf 7y ago
"So the biggest change is that Oil will be based on strings (and arrays of strings) like shell" I would advise the author have a look at the internals of Tcl, not because you should copy all the sophisticated features immediately, but because it demonstrates a way to have both "everything is a string" and "advanced data structures" in a relatively principled manner. You don't have to start out with all the optimizations from day one but I always find it helpful to have a look down the road in cases like this, and make sure that if this sort of thing interests you that you don't accidentally make it impossible later.
- chubot 7y ago
  (author here) Yeah I appreciate a lot of things about Tcl, which is basically Lisp + shell (I think Ousterhout even said that?) I read most of the thick book about it even though I don't regularly program in Tcl. A bunch of people have brough up Tcl, and as noted here [1], Oil looks superficially like it because it has proc and set keywords. The biggest difference is that Oil isn't homoiconic in the sense that Tcl is. I believe that implies a lot of dynamic parsing which I'm staying away from [2]. And of course Oil is a compatible Unix shell, which imposes a bunch of constraints. Namely the 'proc' is "isomorphic" to a process -- it takes argv and returns an exit code, and it can be transparently put in pipelines. I wrote 3 years ago about this powerful form of composition [3] and I wish I had finished that :-/ [1] https://www.reddit.com/r/oilshell/comments/esqyaw/ambitions_for_a_shell/ https://www.reddit.com/r/oilshell/comments/esqyaw/ambitions_... [2] http://www.oilshell.org/blog/2019/02/07.html http://www.oilshell.org/blog/2019/02/07.html [3] http://www.oilshell.org/blog/tags.html?tag=shell-the-good-parts#shell-the-good-parts http://www.oilshell.org/blog/tags.html?tag=shell-the-good-pa...
  - sansnomme 7y ago
    What are your thoughts on MSFT PowerShell?
    - 7thaccount 7y ago
      Not the author, but I've used a lot of Unix and Powershell. Powershell is all objects which has some significant consequences. The first is that many operations will only be reasonably performant on tiny files. If you just want to grep a string from a medium sized file (Ex: 80 MB) and then sort it, the Powershell command is trivial, but can take minutes to run as every row becomes an object. To get around this, you have to generally write a fair amount of straight .NET code like C# which defeats the purpose of using Powershell in the first place. The second consequence is convenience. Because everything is an object, when you get files, you have so much metadata at your fingertips. This is definitely harder in shell and even a pain for me in Python. Edit: on one of my older accounts I complained about the impossibility of parsing files with reasonable speed in Powershell despite me testing 5 different methods. Someone replied back with another few methods I've never seen before that were much faster, but still slow. If anyone can find that comment I'd be much obliged (hopefully the former poster remembers and can look back). I really really want Powershell to be more performant with text work as I do a lot of that and everything else in Powershell is so darned convenient. As it stands, it is more of an IT Admin language and possibly DevOps more than anything.
      - jodrellblank 7y ago
        (hopefully the former poster remembers and can look back). I do, and can: https://news.ycombinator.com/item?id=20724899 https://news.ycombinator.com/item?id=20724899 [edit: I didn't mention `get-content -ReadCount 1000` but it might be good for streaming-reading a large file if you don't want it all at once. It outputs blocks of N lines at a time as a single string, which you have to split on newlines by yourself.] There is a related issue in the PowerShell GitHub asking to standardise on a "just give me the data, no metadata" option which would be get-content returning plain strings: https://github.com/PowerShell/PowerShell/issues/7855 https://github.com/PowerShell/PowerShell/issues/7855
skrebbel 7y ago
Mostly off topic, but I've never understood why shells are designed to have this weird dual use case of both being easy to type into a command prompt, and being easy to write 40+ line scripts in. Why does that need to have the exact same syntax? This is the root cause of why bash (and batch and etc etc etc) are so terrible to script in (and why powershell is so terrible to type into a terminal). This weird shit like [ being a command, and having to think long and hard about exactly when variables are replaced and when globs are executed and how errors happen.. It's just pretty pointless. Like, if you write a script using Python or Ruby or Perl or whatever, there's easy ways to call into the shell. Often it's even built-in, like using backticks `echo like this`. These languages aren't designed for shell scripting but even they are often easier and cleaner than a bash script. It could be much better thoughk I bet. I'd love to have a language that is easy and familiar to type into a terminal (and thus, sufficiently bash-like probably), while being significantly easier to write scripts in. You could require the script to have a bit more syntax, a bit more structure, but essentially have the same semantics as what you'd type on the command line, and you'd have the best of both worlds. If well designed, it'd still "feel" like the same thing (much like how all JS programmers immediately grokked Coffeescript when it came out, because it really was just JS); everything you'd learn on the command line, you'd be able to apply in a script and vice versa. The script would just require more structure and thus provide more comfort (good error messages, editor support, etc). For example, globbing. The idea that the shell expands ".foo" into a list of files, before evaluating the command, is handy, but it's also an enormous footgun. What if on the terminal it would do that, but when scripting, you'd be required to write `glob(".foo")` or else it'd error out? THat would force you to clearly control when the expansion happens, it would let you assign it to a variable in a readable way etc. Imagine similar stuff for argument lists, escaping, variable substitution, etc - all the magic available on terminal, but contained in the script. Does this exist? I'd love to have it.
- chubot 7y ago
  (author here) Your first and third paragraph seem to be in conflict? I have heard the feedback that an interactive shell and a programming language should be different, which I disagree with. (For one, shell is already used as a programming language, so the ship has sailed. Also I use shell in the Lisp tradition: build your program up with a REPL.) But yes Oil is designed to "give the best of both worlds" as you say in the third paragraph. This recent post will probably give you the best idea of it: http://www.oilshell.org/blog/2020/01/simplest-explanation.html http://www.oilshell.org/blog/2020/01/simplest-explanation.ht... Rather than echo $mystr # silently splits echo "mystr" # no split In Oil, it's: echo $mystr # no split by default echo @split(mystr) # opt in explicitly This is enabled with options so existing shell scripts work. Also @split isn't really idiomatic since you would likely use arrays, but it could be useful for porting. I've also thought of adding a more explicit alternative for glob, but I actually like the shorter syntax here: echo *.py echo @glob('*.py') # not implemented but is consistent But I'm probably going to implement this safety feature https://github.com/oilshell/oil/issues/552 https://github.com/oilshell/oil/issues/552 to solve the well known issue where files that look like flags can confuse code and data. e.g. 'rm *' with a file named '-rf'.
keeganpoppen 7y ago
not only is this an extremely exciting project that I've been following for what feels like years, but there is such a tremendous richness of stuff only tangentially related to the project that the author has collected as well (one example: https://github.com/oilshell/oil/wiki/ExternalResources https://github.com/oilshell/oil/wiki/ExternalResources). this is clearly such a well-organized, thoughtful endeavor that i have absolute faith that great things will come of it. and he's an excellent writer to boot!
- chubot 7y ago
  Thank you! I'm trying to reign in the scope to make it useful sooner, and I'm always looking for help: http://www.oilshell.org/blog/2019/12/09.html#help-wanted http://www.oilshell.org/blog/2019/12/09.html#help-wanted If you don't have time, maybe you know a programmer who does, and let them know :)
enriquto 7y ago
This is some of the most exciting stuff happening in the unix world these days. My mind was blown after reading two posts on Oil's author blog, a few years ago: * Shell has a Forth-like Quality: https://www.oilshell.org/blog/2017/01/13.html https://www.oilshell.org/blog/2017/01/13.html and * Pipelines Support Vectorized, Point-Free, and Imperative Style: https://www.oilshell.org/blog/2017/01/15.html https://www.oilshell.org/blog/2017/01/15.html Warning: these writings can change your view on the shell language. Now you probably think, as many of us did, that shell is a cute little hack, useful for one-off throwaway stuff, but not really fit for serious work. Afterwards, you will start to see the shell as a glorious, essential element of our civilization, worth of respect and deserving our careful attention. Pipes are an essential programming tool that must be supported by the core language and not by a library construct (as in python).
- jstimpfle 7y ago
  I've written many, many shell scripts, back as a sysadmin, and because I wanted it to not be a cute little hack. I know all the tricks described on the blogpost and many more. I have an intimate understanding how the quoting works and how we can work around its limitations. Let me tell you: shell is sort of a local optimum for one-off throwaway stuff, because it makes commonly needed thing accessible with so few keystrokes. But also, shell is a cute little hack. It's literally impossible to write robust programs in it.
  - cabalamat 7y ago
    > It's literally impossible to write robust programs in it. This is why I never write shell scripts more than about 4 lines long or with any control flow statements in it -- I use Python for that. The Python might be a bit longer, but at least I'll understand it when I come back to it 6 months later.
    - mjevans 7y ago
      I would _like_ to use Python (2) for that, but it's EOL/effectively EOL, and Python 3 made the _wrong_ call with respect to Unicode support. Source code is Unicode? Great. Forcing _everything_ to have an Encoding when the surrounding world isn't an academically clean environment? Nope, this means Unicode Enforcement and Error Handling in what should be simple programs. I don't mind have __optional__ libraries for handling Encoding conversions, corrections, and errors... but if I'm working with a legacy filesystem, human input, or anything else, the only basic language support I want or need is transparency of bits/bytes. Let a programmer decide if or where things are validated and what to do about handling invalid input at that moment; do not force the issue upon them.
deleted 7y ago
[deleted]
techdragon 7y ago
I’m looking forward to the post mentioned towards the end, detailing the cutting of features. I’ve been following the work on this for ages and was looking forward to types data as a way to guard against accidental mistakes, so I’m keen to read the author’s reasoning for switching to strings and arrays of strings. I’m sure there’s a non-trivial reason, his work has always felt careful, measured and well though out to me.
- chubot 7y ago
  Thanks for the support! Since I may not get to those blog posts in a timely fashion, here's a little outline (and maybe this comment will form the seed of them). ----- I should be more specific: I am ruling out reusing Python data structures, but I'm not ruling out typed data forever. I'm leaving space for types. Though given all the work I see in the next year, I don't see a ton of work on them happening. If they happen, they won't not literally be Python types, because they don't seem like a great fit. Background: A major motivation for using Python was that I wanted JSON in shell. JSON is a "versionless" textual interchange format that's ubiquitous on the web, and I see the web as an extension of Unix. And there's an obvious correspondence between Python data structures and JSON: dicts, lists, string, float/int, bool. But those structures are still a "model", and I've had many programming experiences that led me to believe that "bytes on disk/network -> data structure in memory -> bytes" is something of a fallacy. Or at least it's a coarse, limited model, and not very shell-like. (For the purposes of a blog post I should find a name for that fallacy.) It very much relates to these comments: - https://lobste.rs/s/uizrgy/making_light_website_02_design#c_xnagup https://lobste.rs/s/uizrgy/making_light_website_02_design#c_... -- a different (old) way of HTML processing I encountered and used that's not (text -> DOM -> text) - https://news.ycombinator.com/item?id=22156950 https://news.ycombinator.com/item?id=22156950 -- Making the analogy to the flawed Python 3 string type, which is (bytes -> code points -> bytes) - https://news.ycombinator.com/item?id=22111403 https://news.ycombinator.com/item?id=22111403 -- Kragen's "memory models" post got me thinking about these things. Another way to say it would be: instead of "dict, list, string" as the model, I think a better model for a shell is "JSON, CSV/TSV, and HTML". In other words: (1) tree shaped data, (2) table shaped data, and (3) documents. [1] In a more literal and concrete fashion -- NOT really high level abstractions for them. Maybe another slogan could be "Bijections that aren't really bijections". However I'm not promising to get to this in the next year :) ---- But practically speaking, a lot of programming is spent shuffling between representations. And a lot of the times that's not only inefficient, but also incorrect. It relates strongly to this post: How to Quickly and Correctly Generate a Git Log in HTML http://www.oilshell.org/blog/2017/09/19.html http://www.oilshell.org/blog/2017/09/19.html If you want a buzzword I would say shell is a language where you deal with "concretions" -- that is, you deal with reality rather than abstractions. But a crucial point is that a language can still help you avoid mistakes when programming concretely -- in a large part with principled parsing and quoting/dequoting. You're not going to "groveling through backslashes and braces one a time" as I call bash's style. The language can provide sharper tools than (JSON -> dict/list -> JSON) or (HTML -> DOM -> HTML). ---- A related thought is that I noticed that procs and funcs don't really compose (procs being shell-like functions, funcs are Python/JS-like functions). As an analogy: If you've ever written a Python/C extension or a Node/C++ extension, you will also see that those models also compose pretty poorly. Despite the similar syntax, there's a surprising amount of friction semantically (types and GC being two main things). So adding typed data and Python-like functions was almost tantamount to adding a "separate" language within shell itself. The language loses its coherence when the primitives don't compose. There can be too much shuffling. ----- So that's a bunch of thoughts that may go into a blog post. Hopefully at least a few parts made sense :) [1] This reminds me that there's a paper Unifying tables, objects, and documents that I think influenced C# ? I should go re-read it. An important point is that objects are a often a poor fit for representing tables and documents (e.g. ORM problem, awkward DOMs), and tables and documents are more prevalent in the wild! You can deal with tables and documents "directly", rather than awkwardly manipulating object/tree representations of them. CSS selectors are a way of doing this, i.e. you're not "shuffling objects". A comparison I would draw is 2004-style DOM API vs. jQuery / HTML5 DOM. https://scholar.google.com/scholar?cluster=12582942667964919771&hl=en&as_sdt=0,5&sciodt=0,5 https://scholar.google.com/scholar?cluster=12582942667964919...
celeritascelery 7y ago
> Replace the world's usage of bash. I feel like this is hardest part. Bash is installed everywhere, and that ubiquity is one of its greatest assets. Oil can’t replicate that (at least not for a very long time).
- fergie 7y ago
  And bash is also perfectly OK. It doesnt have all the bells and whistles of, say, zsh- but the things that it does do, it does better (especially autocomplete). If you want to write scripts, bash is often a pretty good choice for anything involving simple manipulation of text files. If you want something more advanced then bash will let you edit, compile and run programs in the language of your choice. That said- there may well be a killer usecase for Oil. If so- what is it?
chadrs 7y ago
This kind of reminds me of Ammonite: https://ammonite.io/#Ammonite-Shell https://ammonite.io/#Ammonite-Shell which definitely for a different audience but kind of a blend of being a Scala repl and somewhat-usage shell with several common commands being implemented as JVM functions with sensible types.
Riverheart 7y ago
Good luck Chubot! If you release in 2020 I look forward to using it.
- chubot 7y ago
  I release all the time, you can try it now :) I'm looking for feedback -- a 5 minute first impression is valuable. http://www.oilshell.org/releases.html http://www.oilshell.org/releases.html The home page has links on "where to send feedback" and what to try / "help wanted": http://www.oilshell.org/ http://www.oilshell.org/

116 points by luu 7y ago

>What languages and tools does Oil aim to "replace"? >bash, awk, make, Python 2 What will in fact happen: It says here on your resume that you know bash, awk, make, and python 2, but do you know Oil as well?

chubot 7y ago
"replace" is in quotes for a reason :) The rest of the post gives some more color on it.

jerf 7y ago

"So the biggest change is that Oil will be based on strings (and arrays of strings) like shell" I would advise the author have a look at the internals of Tcl, not because you should copy all the sophisticated features immediately, but because it demonstrates a way to have both "everything is a string" and "advanced data structures" in a relatively principled manner. You don't have to start out with all the optimizations from day one but I always find it helpful to have a look down the road in cases like this, and make sure that if this sort of thing interests you that you don't accidentally make it impossible later.

chubot 7y ago
(author here) Yeah I appreciate a lot of things about Tcl, which is basically Lisp + shell (I think Ousterhout even said that?) I read most of the thick book about it even though I don't regularly program in Tcl. A bunch of people have brough up Tcl, and as noted here [1], Oil looks superficially like it because it has proc and set keywords. The biggest difference is that Oil isn't homoiconic in the sense that Tcl is. I believe that implies a lot of dynamic parsing which I'm staying away from [2]. And of course Oil is a compatible Unix shell, which imposes a bunch of constraints. Namely the 'proc' is "isomorphic" to a process -- it takes argv and returns an exit code, and it can be transparently put in pipelines. I wrote 3 years ago about this powerful form of composition [3] and I wish I had finished that :-/ [1] https://www.reddit.com/r/oilshell/comments/esqyaw/ambitions_for_a_shell/ https://www.reddit.com/r/oilshell/comments/esqyaw/ambitions_... [2] http://www.oilshell.org/blog/2019/02/07.html http://www.oilshell.org/blog/2019/02/07.html [3] http://www.oilshell.org/blog/tags.html?tag=shell-the-good-parts#shell-the-good-parts http://www.oilshell.org/blog/tags.html?tag=shell-the-good-pa...
- sansnomme 7y ago
  What are your thoughts on MSFT PowerShell?
  - 7thaccount 7y ago
    Not the author, but I've used a lot of Unix and Powershell. Powershell is all objects which has some significant consequences. The first is that many operations will only be reasonably performant on tiny files. If you just want to grep a string from a medium sized file (Ex: 80 MB) and then sort it, the Powershell command is trivial, but can take minutes to run as every row becomes an object. To get around this, you have to generally write a fair amount of straight .NET code like C# which defeats the purpose of using Powershell in the first place. The second consequence is convenience. Because everything is an object, when you get files, you have so much metadata at your fingertips. This is definitely harder in shell and even a pain for me in Python. Edit: on one of my older accounts I complained about the impossibility of parsing files with reasonable speed in Powershell despite me testing 5 different methods. Someone replied back with another few methods I've never seen before that were much faster, but still slow. If anyone can find that comment I'd be much obliged (hopefully the former poster remembers and can look back). I really really want Powershell to be more performant with text work as I do a lot of that and everything else in Powershell is so darned convenient. As it stands, it is more of an IT Admin language and possibly DevOps more than anything.
    - jodrellblank 7y ago
      (hopefully the former poster remembers and can look back). I do, and can: https://news.ycombinator.com/item?id=20724899 https://news.ycombinator.com/item?id=20724899 [edit: I didn't mention `get-content -ReadCount 1000` but it might be good for streaming-reading a large file if you don't want it all at once. It outputs blocks of N lines at a time as a single string, which you have to split on newlines by yourself.] There is a related issue in the PowerShell GitHub asking to standardise on a "just give me the data, no metadata" option which would be get-content returning plain strings: https://github.com/PowerShell/PowerShell/issues/7855 https://github.com/PowerShell/PowerShell/issues/7855

skrebbel 7y ago

Mostly off topic, but I've never understood why shells are designed to have this weird dual use case of both being easy to type into a command prompt, and being easy to write 40+ line scripts in. Why does that need to have the exact same syntax? This is the root cause of why bash (and batch and etc etc etc) are so terrible to script in (and why powershell is so terrible to type into a terminal). This weird shit like [ being a command, and having to think long and hard about exactly when variables are replaced and when globs are executed and how errors happen.. It's just pretty pointless. Like, if you write a script using Python or Ruby or Perl or whatever, there's easy ways to call into the shell. Often it's even built-in, like using backticks `echo like this`. These languages aren't designed for shell scripting but even they are often easier and cleaner than a bash script. It could be much better thoughk I bet. I'd love to have a language that is easy and familiar to type into a terminal (and thus, sufficiently bash-like probably), while being significantly easier to write scripts in. You could require the script to have a bit more syntax, a bit more structure, but essentially have the same semantics as what you'd type on the command line, and you'd have the best of both worlds. If well designed, it'd still "feel" like the same thing (much like how all JS programmers immediately grokked Coffeescript when it came out, because it really was just JS); everything you'd learn on the command line, you'd be able to apply in a script and vice versa. The script would just require more structure and thus provide more comfort (good error messages, editor support, etc). For example, globbing. The idea that the shell expands ".foo" into a list of files, before evaluating the command, is handy, but it's also an enormous footgun. What if on the terminal it would do that, but when scripting, you'd be required to write `glob(".foo")` or else it'd error out? THat would force you to clearly control when the expansion happens, it would let you assign it to a variable in a readable way etc. Imagine similar stuff for argument lists, escaping, variable substitution, etc - all the magic available on terminal, but contained in the script. Does this exist? I'd love to have it.

chubot 7y ago
(author here) Your first and third paragraph seem to be in conflict? I have heard the feedback that an interactive shell and a programming language should be different, which I disagree with. (For one, shell is already used as a programming language, so the ship has sailed. Also I use shell in the Lisp tradition: build your program up with a REPL.) But yes Oil is designed to "give the best of both worlds" as you say in the third paragraph. This recent post will probably give you the best idea of it: http://www.oilshell.org/blog/2020/01/simplest-explanation.html http://www.oilshell.org/blog/2020/01/simplest-explanation.ht... Rather than echo $mystr # silently splits echo "mystr" # no split In Oil, it's: echo $mystr # no split by default echo @split(mystr) # opt in explicitly This is enabled with options so existing shell scripts work. Also @split isn't really idiomatic since you would likely use arrays, but it could be useful for porting. I've also thought of adding a more explicit alternative for glob, but I actually like the shorter syntax here: echo *.py echo @glob('*.py') # not implemented but is consistent But I'm probably going to implement this safety feature https://github.com/oilshell/oil/issues/552 https://github.com/oilshell/oil/issues/552 to solve the well known issue where files that look like flags can confuse code and data. e.g. 'rm *' with a file named '-rf'.

keeganpoppen 7y ago

not only is this an extremely exciting project that I've been following for what feels like years, but there is such a tremendous richness of stuff only tangentially related to the project that the author has collected as well (one example: https://github.com/oilshell/oil/wiki/ExternalResources https://github.com/oilshell/oil/wiki/ExternalResources). this is clearly such a well-organized, thoughtful endeavor that i have absolute faith that great things will come of it. and he's an excellent writer to boot!

chubot 7y ago
Thank you! I'm trying to reign in the scope to make it useful sooner, and I'm always looking for help: http://www.oilshell.org/blog/2019/12/09.html#help-wanted http://www.oilshell.org/blog/2019/12/09.html#help-wanted If you don't have time, maybe you know a programmer who does, and let them know :)

enriquto 7y ago

This is some of the most exciting stuff happening in the unix world these days. My mind was blown after reading two posts on Oil's author blog, a few years ago: * Shell has a Forth-like Quality: https://www.oilshell.org/blog/2017/01/13.html https://www.oilshell.org/blog/2017/01/13.html and * Pipelines Support Vectorized, Point-Free, and Imperative Style: https://www.oilshell.org/blog/2017/01/15.html https://www.oilshell.org/blog/2017/01/15.html Warning: these writings can change your view on the shell language. Now you probably think, as many of us did, that shell is a cute little hack, useful for one-off throwaway stuff, but not really fit for serious work. Afterwards, you will start to see the shell as a glorious, essential element of our civilization, worth of respect and deserving our careful attention. Pipes are an essential programming tool that must be supported by the core language and not by a library construct (as in python).

jstimpfle 7y ago
I've written many, many shell scripts, back as a sysadmin, and because I wanted it to not be a cute little hack. I know all the tricks described on the blogpost and many more. I have an intimate understanding how the quoting works and how we can work around its limitations. Let me tell you: shell is sort of a local optimum for one-off throwaway stuff, because it makes commonly needed thing accessible with so few keystrokes. But also, shell is a cute little hack. It's literally impossible to write robust programs in it.
- cabalamat 7y ago
  > It's literally impossible to write robust programs in it. This is why I never write shell scripts more than about 4 lines long or with any control flow statements in it -- I use Python for that. The Python might be a bit longer, but at least I'll understand it when I come back to it 6 months later.
  - mjevans 7y ago
    I would _like_ to use Python (2) for that, but it's EOL/effectively EOL, and Python 3 made the _wrong_ call with respect to Unicode support. Source code is Unicode? Great. Forcing _everything_ to have an Encoding when the surrounding world isn't an academically clean environment? Nope, this means Unicode Enforcement and Error Handling in what should be simple programs. I don't mind have __optional__ libraries for handling Encoding conversions, corrections, and errors... but if I'm working with a legacy filesystem, human input, or anything else, the only basic language support I want or need is transparency of bits/bytes. Let a programmer decide if or where things are validated and what to do about handling invalid input at that moment; do not force the issue upon them.

deleted 7y ago

[deleted]

techdragon 7y ago

I’m looking forward to the post mentioned towards the end, detailing the cutting of features. I’ve been following the work on this for ages and was looking forward to types data as a way to guard against accidental mistakes, so I’m keen to read the author’s reasoning for switching to strings and arrays of strings. I’m sure there’s a non-trivial reason, his work has always felt careful, measured and well though out to me.

chubot 7y ago
Thanks for the support! Since I may not get to those blog posts in a timely fashion, here's a little outline (and maybe this comment will form the seed of them). ----- I should be more specific: I am ruling out reusing Python data structures, but I'm not ruling out typed data forever. I'm leaving space for types. Though given all the work I see in the next year, I don't see a ton of work on them happening. If they happen, they won't not literally be Python types, because they don't seem like a great fit. Background: A major motivation for using Python was that I wanted JSON in shell. JSON is a "versionless" textual interchange format that's ubiquitous on the web, and I see the web as an extension of Unix. And there's an obvious correspondence between Python data structures and JSON: dicts, lists, string, float/int, bool. But those structures are still a "model", and I've had many programming experiences that led me to believe that "bytes on disk/network -> data structure in memory -> bytes" is something of a fallacy. Or at least it's a coarse, limited model, and not very shell-like. (For the purposes of a blog post I should find a name for that fallacy.) It very much relates to these comments: - https://lobste.rs/s/uizrgy/making_light_website_02_design#c_xnagup https://lobste.rs/s/uizrgy/making_light_website_02_design#c_... -- a different (old) way of HTML processing I encountered and used that's not (text -> DOM -> text) - https://news.ycombinator.com/item?id=22156950 https://news.ycombinator.com/item?id=22156950 -- Making the analogy to the flawed Python 3 string type, which is (bytes -> code points -> bytes) - https://news.ycombinator.com/item?id=22111403 https://news.ycombinator.com/item?id=22111403 -- Kragen's "memory models" post got me thinking about these things. Another way to say it would be: instead of "dict, list, string" as the model, I think a better model for a shell is "JSON, CSV/TSV, and HTML". In other words: (1) tree shaped data, (2) table shaped data, and (3) documents. [1] In a more literal and concrete fashion -- NOT really high level abstractions for them. Maybe another slogan could be "Bijections that aren't really bijections". However I'm not promising to get to this in the next year :) ---- But practically speaking, a lot of programming is spent shuffling between representations. And a lot of the times that's not only inefficient, but also incorrect. It relates strongly to this post: How to Quickly and Correctly Generate a Git Log in HTML http://www.oilshell.org/blog/2017/09/19.html http://www.oilshell.org/blog/2017/09/19.html If you want a buzzword I would say shell is a language where you deal with "concretions" -- that is, you deal with reality rather than abstractions. But a crucial point is that a language can still help you avoid mistakes when programming concretely -- in a large part with principled parsing and quoting/dequoting. You're not going to "groveling through backslashes and braces one a time" as I call bash's style. The language can provide sharper tools than (JSON -> dict/list -> JSON) or (HTML -> DOM -> HTML). ---- A related thought is that I noticed that procs and funcs don't really compose (procs being shell-like functions, funcs are Python/JS-like functions). As an analogy: If you've ever written a Python/C extension or a Node/C++ extension, you will also see that those models also compose pretty poorly. Despite the similar syntax, there's a surprising amount of friction semantically (types and GC being two main things). So adding typed data and Python-like functions was almost tantamount to adding a "separate" language within shell itself. The language loses its coherence when the primitives don't compose. There can be too much shuffling. ----- So that's a bunch of thoughts that may go into a blog post. Hopefully at least a few parts made sense :) [1] This reminds me that there's a paper Unifying tables, objects, and documents that I think influenced C# ? I should go re-read it. An important point is that objects are a often a poor fit for representing tables and documents (e.g. ORM problem, awkward DOMs), and tables and documents are more prevalent in the wild! You can deal with tables and documents "directly", rather than awkwardly manipulating object/tree representations of them. CSS selectors are a way of doing this, i.e. you're not "shuffling objects". A comparison I would draw is 2004-style DOM API vs. jQuery / HTML5 DOM. https://scholar.google.com/scholar?cluster=12582942667964919771&hl=en&as_sdt=0,5&sciodt=0,5 https://scholar.google.com/scholar?cluster=12582942667964919...

celeritascelery 7y ago

> Replace the world's usage of bash. I feel like this is hardest part. Bash is installed everywhere, and that ubiquity is one of its greatest assets. Oil can’t replicate that (at least not for a very long time).

fergie 7y ago
And bash is also perfectly OK. It doesnt have all the bells and whistles of, say, zsh- but the things that it does do, it does better (especially autocomplete). If you want to write scripts, bash is often a pretty good choice for anything involving simple manipulation of text files. If you want something more advanced then bash will let you edit, compile and run programs in the language of your choice. That said- there may well be a killer usecase for Oil. If so- what is it?

chadrs 7y ago

This kind of reminds me of Ammonite: https://ammonite.io/#Ammonite-Shell https://ammonite.io/#Ammonite-Shell which definitely for a different audience but kind of a blend of being a Scala repl and somewhat-usage shell with several common commands being implemented as JVM functions with sensible types.

Riverheart 7y ago

Good luck Chubot! If you release in 2020 I look forward to using it.

chubot 7y ago
I release all the time, you can try it now :) I'm looking for feedback -- a 5 minute first impression is valuable. http://www.oilshell.org/releases.html http://www.oilshell.org/releases.html The home page has links on "where to send feedback" and what to try / "help wanted": http://www.oilshell.org/ http://www.oilshell.org/