8 ms·

Oils 0.17.0 – YSH Is Becoming Real

66 points by chubot 3y ago

subjectsigma 3y ago
Oil looks pretty incredible and I think both the work done and the dedication to backwards compatibility are admirable. That being said it seems like Oil is polishing the shiniest turd. I never liked Unix shells or shells much in general and strongly prefer to type out commands in a script (usually Python or Perl) and then run that script. It takes longer but I actually trust the result to work and I’ll understand it two months from now. Also when you do that you get all the benefits of using your riced-out, LSP-enabled editor. Ofc I still use a terminal for apt and other such programs, but not munging data. It seems like if you wanted to make a modern shell you would work on improving the interface and not so much the internals. Like in the SerenityOS terminal emulator, how everything is clickable. I’ve always wanted an easy way to “export” a shell session to a script, including env variables and whatnot. Maybe types in YSH will enable better terminal emulators, I don’t know.
- aidenn0 3y ago
  While OSH aims to be strictly better than Bash, YSH aims to be strictly better than "shell written in Python[1]." If there is something shell-like that you think Python is superior to YSH for, then please give some examples before the language hits 1.0 so they can be taken into account. 1: Obviously a large general purpose program that only occasionally needs to run an external program is probably going to be better in Python due to the library ecosystem alone. I'm talking about those Python programs that use primarily some combination of pathlib, os, sys, and subprocess and exist only because of the deficiencies (perceived and real) of shell.
- chubot 3y ago
  (author here) Funny thing is I also type out most of my shell into a script file. I generally don't create huge commands interactively. I elaborated on that here: https://lobste.rs/s/pcuppu/what_helps_people_get_comfortable_on#c_dqkbrs https://lobste.rs/s/pcuppu/what_helps_people_get_comfortable... Basically I want to be able to use all the power of Unix without remembering it all -- shell is a great tool for that. Someone made a LSP plugin for Oils awhile back, and I expect if enough people use it, those tools will continue to improve, and new ones will appear. They will be MUCH easier to produce for Oils than shell, because we have a principled code representation. (Some tools will be inside the project, and some outside, as usual) --- On the UI side, there are also people exploring shell GUIs via the "headless shell" -- something no other shell has -- but we can use more people to help. screenshots: https://www.oilshell.org/blog/2023/06/release-0.16.0.html#headless-shell-screenshots https://www.oilshell.org/blog/2023/06/release-0.16.0.html#he... i.e. Join #shell-gui on Zulip; don't assume it will get done without your input or effort ... as mentioned in this blog post, the biggest risk is that we don't have enough people helping!
  - subjectsigma 3y ago
    I can't contribute directly to open source because of my employer. But the pages you linked are very interesting. I may be interested in doing beta testing or something, so I'll join Zulip and say hi. Thanks!
- predictabl3 3y ago
  Couldn't agree more and at this point, virtually every new mainstay tool I adopt is written in Zig or Rust and I can't see myself migrating to a shell written in C/C++. Especially when something like nushell so elegantly avoids the situation you're describing. (I use nushell as my daily driver shell, and bang out scripts in it on an hourly basis. It's a blessing to use, though still missing an essential "shell" feature or two)
  - chubot 3y ago
    There's only 7K lines of C++ in Oils, mentioned in this post, and we intentionally keep it down. (It's for the garbage collector and OS bindings -- basically the unsafe parts.) The rest is typed Python (with algebraic data types), which more common and simpler than Zig or Rust. It's not as fast as Zig or Rust, but 1. The shell is the same speed when it's running external processes (and we do measure) 2. I believe it can be as fast as bash's hand-written C, even for code that doesn't do I/O (which isn't common but exists). We're perhaps ~50% the speed now, being "hilariously unoptimized". This release mentions a big optimization, but the bigger point is that we have leverage via metaprogramming. We can do transformations on the entire interpreter (e.g. "boxless" transformation) 3. It means we only have 40K lines of Python, compared to 142K lines of C in bash. If you add YSH, we will only have around 50K lines, compared to probably 200K lines if you wrote it by hand So the whole program is closer to an executable spec -- there is less room for bugs to hide. It's easier to optimize 50K lines that is known to be correct (with thousands of tests), than 200K lines of code --- Our binary is about 1.5 MB of executable code with no dependencies other than libc and (optional) GNU readline. On the other hand, shells in Rust seem to have a big tree of dependencies and end up at 20 - 50 MB. It's not the size I'm concerned about, so much as the stability As mentioned before, our approach is unusual, but I think the "usual" approaches haven't worked well. That said, if someone wrote a compatible shell from scratch in Zig or Rust (with limited dependencies), I think that would be also be valuable contribution --- Oils is also aimed at the system level, and is an "exterior" shell that uses processes and file descriptors. Comparison with PowerShell, Elvish, nushell here: https://www.oilshell.org/blog/2023/06/ysh-design.html#survey-of-alternative-shells https://www.oilshell.org/blog/2023/06/ysh-design.html#survey... i.e. "shell" is a bit overloaded, and in some sense those tools are "one level higher".
- b5n 3y ago
  > Also when you do that you get all the benefits of using your riced-out, LSP-enabled editor https://github.com/bash-lsp/bash-language-server https://github.com/bash-lsp/bash-language-server https://github.com/koalaman/shellcheck https://github.com/koalaman/shellcheck https://github.com/idank/explainshell https://github.com/idank/explainshell
- em-bee 3y ago
  i agree with that. i almost never create scripts though, because i work on multiple computers at once, and copying and adapting commandline snippets is easier than scripts that don't necesarily work the same on every machine. so i like improvements in the interactive experience like fish mostly focuses on. but if i were to write a script i would not use fish either. that said, a language that focuses on shell scripts could be a good idea because there is not much work done yet in that space
- e28eta 3y ago
  > your riced-out, LSP-enabled editor It’s my belief that “riced out” has negative racial overtones, and isn’t an appropriate adjective to be using. Wikipedia has some history on this related term: https://en.wikipedia.org/wiki/Rice_burner https://en.wikipedia.org/wiki/Rice_burner
  - subjectsigma 3y ago
    Been browsing tech forums for years and years and I’ve only ever heard “rice” used as a compliment to refer to software that’s really aesthetically pleasing. Didn’t mean to offend, had no idea it was even a term borrowed from car culture.
- paddim8 3y ago
  Elk shell is quite different from traditional shells. More like Python. https://elk.strct.net https://elk.strct.net
JNRowe 3y ago
I really want this to succeed¹, but I struggle to use it. How are people that are testing/using it coping without the normal niceties of their regular setup? I'm wondering if there are just some tricks I'm missing. Off the top of my head, things I'm thinking of include: * Just a minute ago realising that without zsh globs I have to remember find syntax, and then have to figure out how to sort the results(probably decorate →sort(1) → undecorate). My laziness led me to just C-d and execute my task with zsh. * No context sensitive completion; I don't realise how much I rely on this until it is gone. Using bash-completion gets some completion, but it is a big step backwards. * No programmable input, you can fake some with ~/.inputrc hacks but only basic text support(no macros). (Edit to add: The guard name is "oil" not "osh", if you're looking to do this yourself.) There is a document in the wiki², but it doesn't really cover much. I'd be interested in seeing the configs of people who are using it. -- Should say I really like the idea of hijacking '#!/bin/sh' to inject osh, it feels like a really good way to round out testing of scripts you'd otherwise forget about. I wonder if a LD_PRELOAD'able solution that would work outside osh would be a good idea. ¹ Strong déjà vu on this comment, but if it is real it is because I truly like the goals set out for oil. ² https://github.com/oilshell/oil/wiki/How-To-Test-OSH https://github.com/oilshell/oil/wiki/How-To-Test-OSH
- chubot 3y ago
  (author here) This is good feedback 1) I think you mean stuff like **/*.py from zsh, which bash now emulates with shopt -s globstar. This has come up several times, and we should have it. But I'm actually not entirely sure how ** behaves -- if someone can write an exhaustive shell script that shows what it does (e.g. a bunch of mkdir and touch commands and output), we can copy it in our spec tests: https://github.com/oilshell/oil/tree/master/spec https://github.com/oilshell/oil/tree/master/spec https://www.oilshell.org/release/0.17.0/test/spec.wwz/osh-py/index.html https://www.oilshell.org/release/0.17.0/test/spec.wwz/osh-py... and then implement it. Generally once the behavior is spec'd out, it's not too hard to implement 2) Completion has a long history in the project -- we emulate bash completion very faithfully, but yeah I've realized that's not enough This comment elaborates on a different plan -- I think something like "fishcomp" to emulate fish would be both useful and pretty feasible: https://lobste.rs/s/fdqbhk/oils_0_17_0_ysh_is_becoming_real#c_t8slnw https://lobste.rs/s/fdqbhk/oils_0_17_0_ysh_is_becoming_real#... We can definitely use help with that, and it's relatively self-contained. 3) What exactly do you mean by "programmable input" ? We support GNU readline so everything there should work (maybe the ./configure detection didn't find it for you?) --- But there is a bigger reason for the "struggle" -- the main focus of the project now is YSH, the language with data tYpes, influenced by pYthon, JavaScript, Lisp, etc. That is, a shell is both a language and an interactive UI, and I've long emphasized the language part, e.g. in the FAQ - http://www.oilshell.org/blog/2021/01/why-a-new-shell.html http://www.oilshell.org/blog/2021/01/why-a-new-shell.html The theory was/is that the language is not only important for the system level tasks, but also a good foundation for the UI, e.g. like Emacs is written in Emacs Lisp So I think structured data (in the form of languages/protocols) in YSH is really the killer feature of Oils, that will hopefully bring many other people into the project, who can help with filling out the interactive features you mention, e.g. polishing completion. I wrote FIVE posts about YSH in June: https://www.oilshell.org/blog/2023/06/release-0.16.0.html#some-writing-on-ysh-for-everyone https://www.oilshell.org/blog/2023/06/release-0.16.0.html#so... As mentioned in this post, the biggest risk is that we don't have enough help. All the technical pieces are there now, and the risks are about project management, not the code. I appreciate everyone who contributed while the codebase was undergoing this big transformation to native code -- this was kind of risky and the most unusual part of the project, but I think it paid off! (We also have big ideas for the interactive shell in the form of a headless protocol for GUIs, but again need help -- https://www.oilshell.org/blog/2023/06/release-0.16.0.html#headless-shell-screenshots https://www.oilshell.org/blog/2023/06/release-0.16.0.html#he... )

66 points by chubot 3y ago

Oil looks pretty incredible and I think both the work done and the dedication to backwards compatibility are admirable. That being said it seems like Oil is polishing the shiniest turd. I never liked Unix shells or shells much in general and strongly prefer to type out commands in a script (usually Python or Perl) and then run that script. It takes longer but I actually trust the result to work and I’ll understand it two months from now. Also when you do that you get all the benefits of using your riced-out, LSP-enabled editor. Ofc I still use a terminal for apt and other such programs, but not munging data. It seems like if you wanted to make a modern shell you would work on improving the interface and not so much the internals. Like in the SerenityOS terminal emulator, how everything is clickable. I’ve always wanted an easy way to “export” a shell session to a script, including env variables and whatnot. Maybe types in YSH will enable better terminal emulators, I don’t know.

aidenn0 3y ago
While OSH aims to be strictly better than Bash, YSH aims to be strictly better than "shell written in Python[1]." If there is something shell-like that you think Python is superior to YSH for, then please give some examples before the language hits 1.0 so they can be taken into account. 1: Obviously a large general purpose program that only occasionally needs to run an external program is probably going to be better in Python due to the library ecosystem alone. I'm talking about those Python programs that use primarily some combination of pathlib, os, sys, and subprocess and exist only because of the deficiencies (perceived and real) of shell.
chubot 3y ago
(author here) Funny thing is I also type out most of my shell into a script file. I generally don't create huge commands interactively. I elaborated on that here: https://lobste.rs/s/pcuppu/what_helps_people_get_comfortable_on#c_dqkbrs https://lobste.rs/s/pcuppu/what_helps_people_get_comfortable... Basically I want to be able to use all the power of Unix without remembering it all -- shell is a great tool for that. Someone made a LSP plugin for Oils awhile back, and I expect if enough people use it, those tools will continue to improve, and new ones will appear. They will be MUCH easier to produce for Oils than shell, because we have a principled code representation. (Some tools will be inside the project, and some outside, as usual) --- On the UI side, there are also people exploring shell GUIs via the "headless shell" -- something no other shell has -- but we can use more people to help. screenshots: https://www.oilshell.org/blog/2023/06/release-0.16.0.html#headless-shell-screenshots https://www.oilshell.org/blog/2023/06/release-0.16.0.html#he... i.e. Join #shell-gui on Zulip; don't assume it will get done without your input or effort ... as mentioned in this blog post, the biggest risk is that we don't have enough people helping!
- subjectsigma 3y ago
  I can't contribute directly to open source because of my employer. But the pages you linked are very interesting. I may be interested in doing beta testing or something, so I'll join Zulip and say hi. Thanks!
predictabl3 3y ago
Couldn't agree more and at this point, virtually every new mainstay tool I adopt is written in Zig or Rust and I can't see myself migrating to a shell written in C/C++. Especially when something like nushell so elegantly avoids the situation you're describing. (I use nushell as my daily driver shell, and bang out scripts in it on an hourly basis. It's a blessing to use, though still missing an essential "shell" feature or two)
- chubot 3y ago
  There's only 7K lines of C++ in Oils, mentioned in this post, and we intentionally keep it down. (It's for the garbage collector and OS bindings -- basically the unsafe parts.) The rest is typed Python (with algebraic data types), which more common and simpler than Zig or Rust. It's not as fast as Zig or Rust, but 1. The shell is the same speed when it's running external processes (and we do measure) 2. I believe it can be as fast as bash's hand-written C, even for code that doesn't do I/O (which isn't common but exists). We're perhaps ~50% the speed now, being "hilariously unoptimized". This release mentions a big optimization, but the bigger point is that we have leverage via metaprogramming. We can do transformations on the entire interpreter (e.g. "boxless" transformation) 3. It means we only have 40K lines of Python, compared to 142K lines of C in bash. If you add YSH, we will only have around 50K lines, compared to probably 200K lines if you wrote it by hand So the whole program is closer to an executable spec -- there is less room for bugs to hide. It's easier to optimize 50K lines that is known to be correct (with thousands of tests), than 200K lines of code --- Our binary is about 1.5 MB of executable code with no dependencies other than libc and (optional) GNU readline. On the other hand, shells in Rust seem to have a big tree of dependencies and end up at 20 - 50 MB. It's not the size I'm concerned about, so much as the stability As mentioned before, our approach is unusual, but I think the "usual" approaches haven't worked well. That said, if someone wrote a compatible shell from scratch in Zig or Rust (with limited dependencies), I think that would be also be valuable contribution --- Oils is also aimed at the system level, and is an "exterior" shell that uses processes and file descriptors. Comparison with PowerShell, Elvish, nushell here: https://www.oilshell.org/blog/2023/06/ysh-design.html#survey-of-alternative-shells https://www.oilshell.org/blog/2023/06/ysh-design.html#survey... i.e. "shell" is a bit overloaded, and in some sense those tools are "one level higher".
b5n 3y ago
> Also when you do that you get all the benefits of using your riced-out, LSP-enabled editor https://github.com/bash-lsp/bash-language-server https://github.com/bash-lsp/bash-language-server https://github.com/koalaman/shellcheck https://github.com/koalaman/shellcheck https://github.com/idank/explainshell https://github.com/idank/explainshell
em-bee 3y ago
i agree with that. i almost never create scripts though, because i work on multiple computers at once, and copying and adapting commandline snippets is easier than scripts that don't necesarily work the same on every machine. so i like improvements in the interactive experience like fish mostly focuses on. but if i were to write a script i would not use fish either. that said, a language that focuses on shell scripts could be a good idea because there is not much work done yet in that space
e28eta 3y ago
> your riced-out, LSP-enabled editor It’s my belief that “riced out” has negative racial overtones, and isn’t an appropriate adjective to be using. Wikipedia has some history on this related term: https://en.wikipedia.org/wiki/Rice_burner https://en.wikipedia.org/wiki/Rice_burner
- subjectsigma 3y ago
  Been browsing tech forums for years and years and I’ve only ever heard “rice” used as a compliment to refer to software that’s really aesthetically pleasing. Didn’t mean to offend, had no idea it was even a term borrowed from car culture.
paddim8 3y ago
Elk shell is quite different from traditional shells. More like Python. https://elk.strct.net https://elk.strct.net

JNRowe 3y ago

I really want this to succeed¹, but I struggle to use it. How are people that are testing/using it coping without the normal niceties of their regular setup? I'm wondering if there are just some tricks I'm missing. Off the top of my head, things I'm thinking of include: * Just a minute ago realising that without zsh globs I have to remember find syntax, and then have to figure out how to sort the results(probably decorate →sort(1) → undecorate). My laziness led me to just C-d and execute my task with zsh. * No context sensitive completion; I don't realise how much I rely on this until it is gone. Using bash-completion gets some completion, but it is a big step backwards. * No programmable input, you can fake some with ~/.inputrc hacks but only basic text support(no macros). (Edit to add: The guard name is "oil" not "osh", if you're looking to do this yourself.) There is a document in the wiki², but it doesn't really cover much. I'd be interested in seeing the configs of people who are using it. -- Should say I really like the idea of hijacking '#!/bin/sh' to inject osh, it feels like a really good way to round out testing of scripts you'd otherwise forget about. I wonder if a LD_PRELOAD'able solution that would work outside osh would be a good idea. ¹ Strong déjà vu on this comment, but if it is real it is because I truly like the goals set out for oil. ² https://github.com/oilshell/oil/wiki/How-To-Test-OSH https://github.com/oilshell/oil/wiki/How-To-Test-OSH

chubot 3y ago
(author here) This is good feedback 1) I think you mean stuff like **/*.py from zsh, which bash now emulates with shopt -s globstar. This has come up several times, and we should have it. But I'm actually not entirely sure how ** behaves -- if someone can write an exhaustive shell script that shows what it does (e.g. a bunch of mkdir and touch commands and output), we can copy it in our spec tests: https://github.com/oilshell/oil/tree/master/spec https://github.com/oilshell/oil/tree/master/spec https://www.oilshell.org/release/0.17.0/test/spec.wwz/osh-py/index.html https://www.oilshell.org/release/0.17.0/test/spec.wwz/osh-py... and then implement it. Generally once the behavior is spec'd out, it's not too hard to implement 2) Completion has a long history in the project -- we emulate bash completion very faithfully, but yeah I've realized that's not enough This comment elaborates on a different plan -- I think something like "fishcomp" to emulate fish would be both useful and pretty feasible: https://lobste.rs/s/fdqbhk/oils_0_17_0_ysh_is_becoming_real#c_t8slnw https://lobste.rs/s/fdqbhk/oils_0_17_0_ysh_is_becoming_real#... We can definitely use help with that, and it's relatively self-contained. 3) What exactly do you mean by "programmable input" ? We support GNU readline so everything there should work (maybe the ./configure detection didn't find it for you?) --- But there is a bigger reason for the "struggle" -- the main focus of the project now is YSH, the language with data tYpes, influenced by pYthon, JavaScript, Lisp, etc. That is, a shell is both a language and an interactive UI, and I've long emphasized the language part, e.g. in the FAQ - http://www.oilshell.org/blog/2021/01/why-a-new-shell.html http://www.oilshell.org/blog/2021/01/why-a-new-shell.html The theory was/is that the language is not only important for the system level tasks, but also a good foundation for the UI, e.g. like Emacs is written in Emacs Lisp So I think structured data (in the form of languages/protocols) in YSH is really the killer feature of Oils, that will hopefully bring many other people into the project, who can help with filling out the interactive features you mention, e.g. polishing completion. I wrote FIVE posts about YSH in June: https://www.oilshell.org/blog/2023/06/release-0.16.0.html#some-writing-on-ysh-for-everyone https://www.oilshell.org/blog/2023/06/release-0.16.0.html#so... As mentioned in this post, the biggest risk is that we don't have enough help. All the technical pieces are there now, and the risks are about project management, not the code. I appreciate everyone who contributed while the codebase was undergoing this big transformation to native code -- this was kind of risky and the most unusual part of the project, but I think it paid off! (We also have big ideas for the interactive shell in the form of a headless protocol for GUIs, but again need help -- https://www.oilshell.org/blog/2023/06/release-0.16.0.html#headless-shell-screenshots https://www.oilshell.org/blog/2023/06/release-0.16.0.html#he... )