4 ms·

FWIW I wrote a pretty complete shell in ~11K lines of Python: https://github.com/oilshell/oil https://github.com/oilshell/oil (osh/ and core/ directories): Ri

by chubot 9y ago

FWIW I wrote a pretty complete shell in ~11K lines of Python: https://github.com/oilshell/oil https://github.com/oilshell/oil (osh/ and core/ directories): Right now the goal is to clone bash but have better parse time and runtime errors. I hope to make an initial release this summer. But the project is much larger (see the blog if interested).

fiatjaf 9y ago
11k? Whoa.
- chubot 9y ago
  FWIW dash is 20K lines of C code, although it has many fewer features than bash. I use metaprogramming in several places to make the code shorter, so 11K sounds about right for Python vs. C. I think it will be 15K with all the bash features filled out, excluding interactive parts. I use GNU readline and not my own code, so that substantial part isn't counted. The osh/osh.asdl file gives a concise overview of what features are represented and implemented.
enriquto 9y ago
This project is one of the coolest things in the unix world today. I do not care too much about the python implementation (writing it in C alone would be even cooler), but the work is just awesome. Your blog post about "vectorized, point-free and imperative style" was so beautiful that it brought tears to my eyes. Thank you!
- eternalban 9y ago
  http://www.oilshell.org/blog/2017/01/15.html http://www.oilshell.org/blog/2017/01/15.html
- chubot 9y ago
  Wow thanks! I appreciate the encouragement. The architecture has settled, and I'm slogging through all the features now, so that helps. Yeah Python wasn't the original plan, but it helped me focus on correctness. I was able to iteratively reverse engineer bash and other shells, while keeping the code relatively clean. I had hoped to rewrite it in native code, but it's a huge amount of effort, so the first release will be Python. Existence and correctness are both higher priorities than performance :) I disguised Python by bundling a slice of the Python interpreter, which will allow me to rewrite it over time without user-visible packaging/deployment changes. I'm also able to fork the Python language because I have the "OPy" compiler. One thing that should make it faster is using integer offsets everywhere instead of dictionary lookups, and that should be possible without much code change. It is written in a fairly "static" style, although with some important metaprogramming. EDIT: I also hope to resume blogging more regularly once the code is released. I have a big pile of ideas/notes. But getting it out there in any primitive form is the most important thing right now.
windLO 9y ago
I have to ask, from a student perspective how does one acquire sufficient knowledge of writing a shell? Does one need to take OS? Or is knowing data structure enough?
- chubot 9y ago
  The shell should be significantly easier than say writing a compiler, although I think it's less well documented. Much of the work in writing a shell is parsing -- I would estimate that parsing is 60% of the work, whereas it might only be 10-20% of the work of a compiler. This is both because interpreters are smaller than compilers (fewer transformations), and because shell is harder to parse than most programming languages. But as you can see from my blog, it requires a few different parsing techniques, and there are some non-obvious choices I made that I think made it easier. (e.g. Lexer modes aka lexical state are a huge win for reducing complexity.) The shell uses surprisingly few system calls -- fork/exec/wait, open/close/read/write, pipe/dup/fcntl, and that's almost it. These are well worth studying. File descriptors are non-obvious and essential. I never took the OS class that most people take, which shows you how C and Unix work. This thread lists a few, and also check out xv6 in the parent thread: https://news.ycombinator.com/item?id=13112960 https://news.ycombinator.com/item?id=13112960 I learned a lot about system calls by using strace on many programs over the years. I think Python helps a lot because it is interpreted and has relatively direct bindings to Unix system calls. FWIW, although I learned interesting things about parsing, I don't think there is that much "hard" in the computer science sense about shell. It's mainly an exercise in software engineering -- how do you keep your program from degrading into an enormous mass of poorly-debugged if statements? (I would classify bash in that category, unfortunately) As far as data structures, you should be able to write a shell without anything fancy at all. In fact some shells are basically just tons of linked lists. Linked lists make memory management easier in C, although I'm using a more modern, high-level style. I also spent a significant amount of time reading bash, dash, and mksh code for this project. (and to a lesser degree zsh). Happy to answer any other questions.
  - laumars 9y ago
    To be honest the biggest "gotchyas" I stumbled across when writing my shell was handling PTYs. In the end I used someone else's module instead of writing my own because they supported several different flavours of UNIX as well so it seemed a pointless duplication of effort for me to roll my own. However it was real humbling when I discovered just how little I actually new about PTYs compared to what I thought I knew.

by chubot 9y ago

fiatjaf 9y ago

11k? Whoa.

chubot 9y ago
FWIW dash is 20K lines of C code, although it has many fewer features than bash. I use metaprogramming in several places to make the code shorter, so 11K sounds about right for Python vs. C. I think it will be 15K with all the bash features filled out, excluding interactive parts. I use GNU readline and not my own code, so that substantial part isn't counted. The osh/osh.asdl file gives a concise overview of what features are represented and implemented.

enriquto 9y ago

This project is one of the coolest things in the unix world today. I do not care too much about the python implementation (writing it in C alone would be even cooler), but the work is just awesome. Your blog post about "vectorized, point-free and imperative style" was so beautiful that it brought tears to my eyes. Thank you!

eternalban 9y ago
http://www.oilshell.org/blog/2017/01/15.html http://www.oilshell.org/blog/2017/01/15.html
chubot 9y ago
Wow thanks! I appreciate the encouragement. The architecture has settled, and I'm slogging through all the features now, so that helps. Yeah Python wasn't the original plan, but it helped me focus on correctness. I was able to iteratively reverse engineer bash and other shells, while keeping the code relatively clean. I had hoped to rewrite it in native code, but it's a huge amount of effort, so the first release will be Python. Existence and correctness are both higher priorities than performance :) I disguised Python by bundling a slice of the Python interpreter, which will allow me to rewrite it over time without user-visible packaging/deployment changes. I'm also able to fork the Python language because I have the "OPy" compiler. One thing that should make it faster is using integer offsets everywhere instead of dictionary lookups, and that should be possible without much code change. It is written in a fairly "static" style, although with some important metaprogramming. EDIT: I also hope to resume blogging more regularly once the code is released. I have a big pile of ideas/notes. But getting it out there in any primitive form is the most important thing right now.

windLO 9y ago

I have to ask, from a student perspective how does one acquire sufficient knowledge of writing a shell? Does one need to take OS? Or is knowing data structure enough?

chubot 9y ago
The shell should be significantly easier than say writing a compiler, although I think it's less well documented. Much of the work in writing a shell is parsing -- I would estimate that parsing is 60% of the work, whereas it might only be 10-20% of the work of a compiler. This is both because interpreters are smaller than compilers (fewer transformations), and because shell is harder to parse than most programming languages. But as you can see from my blog, it requires a few different parsing techniques, and there are some non-obvious choices I made that I think made it easier. (e.g. Lexer modes aka lexical state are a huge win for reducing complexity.) The shell uses surprisingly few system calls -- fork/exec/wait, open/close/read/write, pipe/dup/fcntl, and that's almost it. These are well worth studying. File descriptors are non-obvious and essential. I never took the OS class that most people take, which shows you how C and Unix work. This thread lists a few, and also check out xv6 in the parent thread: https://news.ycombinator.com/item?id=13112960 https://news.ycombinator.com/item?id=13112960 I learned a lot about system calls by using strace on many programs over the years. I think Python helps a lot because it is interpreted and has relatively direct bindings to Unix system calls. FWIW, although I learned interesting things about parsing, I don't think there is that much "hard" in the computer science sense about shell. It's mainly an exercise in software engineering -- how do you keep your program from degrading into an enormous mass of poorly-debugged if statements? (I would classify bash in that category, unfortunately) As far as data structures, you should be able to write a shell without anything fancy at all. In fact some shells are basically just tons of linked lists. Linked lists make memory management easier in C, although I'm using a more modern, high-level style. I also spent a significant amount of time reading bash, dash, and mksh code for this project. (and to a lesser degree zsh). Happy to answer any other questions.
- laumars 9y ago
  To be honest the biggest "gotchyas" I stumbled across when writing my shell was handling PTYs. In the end I used someone else's module instead of writing my own because they supported several different flavours of UNIX as well so it seemed a pointless duplication of effort for me to roll my own. However it was real humbling when I discovered just how little I actually new about PTYs compared to what I thought I knew.