6 ms·
Writing a Unix Shell
- teddyh 9y agoThere is more to writing a shell than simply reading commands and executing them after fork()ing. TTY handling must be done right¹ and interrupt and signal handling is rather tricky to get right as well². ① http://www.linusakesson.net/programming/tty/ http://www.linusakesson.net/programming/tty/ ② https://www.cons.org/cracauer/sigint.html https://www.cons.org/cracauer/sigint.html
- tyingq 9y agoThe title does say "Part 1"
- luckydude 9y agoMarc Rochkind's Advanced Unix Programming is a very approachable book that walks you through, among other things, writing a shell. It's a good book, give it a read.
- vram22 9y agoYes, good recommendation. I have mentioned it here on HN before. I had got to read it when working in an HP joint venture, years ago. They (HP) used to ship a complementary copy of his book with every new HP-UX business server or workstation (CAD etc.) sold. IIRC the book even shows how to write a simple DBMS (not RDBMS), with client and server parts. And shows how to write all that code somewhat portably (in C) across some Unix versions that were common then. Quite an achievement. The book is still available (in an updated version - after 19 years!): http://basepath.com/aup/ http://basepath.com/aup/ It also shows how to set up pipes - using the pipe(), dup() etc. system calls. There are some subtleties involved in that. And that stuff is used for the shell he creates. Good stuff.
- vram22 9y agotypo: s/complementary/complimentary (though the other meaning also holds, in this case :)
- wruza 9y agoThis one is an incorrect fork() tutorial, not the writing of unix shell in any way.
- b6 9y agoPlease just explain what's wrong. It's more constructive.
- dewyatt 9y agoI can point out one thing: http://rachelbythebay.com/w/2014/08/19/fork/ http://rachelbythebay.com/w/2014/08/19/fork/
- pacaro 9y agoThere is an assertion that the child pid will be greater than the parent pid. While this is usually/often true, it isn't always true. This isn't a big deal per se but is indicative that the author has a limited understanding of the subject. Which itself isn't a big deal either, but the article would be improved by the author stating up front that they don't know the material very well and they too are learning as they go along.
- dhanush 9y agoHello, author here. Thank you for pointing these out. I've corrected my assumption and put up a disclaimer on the post.
- 9y ago
- axman6 9y agoMark Hibberd gave a great talk at Yow! LambdaJam this yeah on writing mundane code in Haskell, with a tutorial on writing a shell. The code can be found on https://github.com/markhibberd/xsh https://github.com/markhibberd/xsh and the talk should be on YouTube in the near future (I hope).
- v3gas 9y agoYeah! :)
- chubot 9y agoFWIW I wrote a pretty complete shell in ~11K lines of Python: https://github.com/oilshell/oil https://github.com/oilshell/oil (osh/ and core/ directories): Right now the goal is to clone bash but have better parse time and runtime errors. I hope to make an initial release this summer. But the project is much larger (see the blog if interested).
- fiatjaf 9y ago11k? Whoa.
- chubot 9y agoFWIW dash is 20K lines of C code, although it has many fewer features than bash. I use metaprogramming in several places to make the code shorter, so 11K sounds about right for Python vs. C. I think it will be 15K with all the bash features filled out, excluding interactive parts. I use GNU readline and not my own code, so that substantial part isn't counted. The osh/osh.asdl file gives a concise overview of what features are represented and implemented.
- enriquto 9y agoThis project is one of the coolest things in the unix world today. I do not care too much about the python implementation (writing it in C alone would be even cooler), but the work is just awesome. Your blog post about "vectorized, point-free and imperative style" was so beautiful that it brought tears to my eyes. Thank you!
- eternalban 9y agohttp://www.oilshell.org/blog/2017/01/15.html http://www.oilshell.org/blog/2017/01/15.html
- chubot 9y agoWow thanks! I appreciate the encouragement. The architecture has settled, and I'm slogging through all the features now, so that helps. Yeah Python wasn't the original plan, but it helped me focus on correctness. I was able to iteratively reverse engineer bash and other shells, while keeping the code relatively clean. I had hoped to rewrite it in native code, but it's a huge amount of effort, so the first release will be Python. Existence and correctness are both higher priorities than performance :) I disguised Python by bundling a slice of the Python interpreter, which will allow me to rewrite it over time without user-visible packaging/deployment changes. I'm also able to fork the Python language because I have the "OPy" compiler. One thing that should make it faster is using integer offsets everywhere instead of dictionary lookups, and that should be possible without much code change. It is written in a fairly "static" style, although with some important metaprogramming. EDIT: I also hope to resume blogging more regularly once the code is released. I have a big pile of ideas/notes. But getting it out there in any primitive form is the most important thing right now.
- reacweb 9y agoWriting a shell, like writing a compiler are very good exercises. Writing a compiler gives a feeling of power, but I think writing a shell teaches humility.
- DroidX86 9y agoNot a very good tutorial on fork() let alone writing your own shell.
- laumars 9y agoPlease excuse the self promotion but I'm also writing a Unix shell in Go. It's not designed to be a drop in replacement for Bash (ie it follows a few Bash-isms but is essentially my own bespoke scripting language) however it does spawn processes in their own PTY and support interactive commands such as vi and top so it does work as a proper $SHELL. It's still very much alpha but at a stage where I'm ready for feedback. https://github.com/lmorg/murex https://github.com/lmorg/murex
- xiaq 9y agoFWIW, here is a pretty complete shell mostly written by me in ~18K lines of Go: https://github.com/elves/elvish https://github.com/elves/elvish
- bluejekyll 9y agofork() also clones all the filedescriptors that might be open to the child. It is the resposibility of the forking code to close filedescriptors it doesn't need. Easy way to cause too many open files errors. (I read this article quickly, so apologies if I missed it.) And then there's the fun of dealing with stdout, and stderr at the same time; in a single thread. One more fun thing to do with fork is to learn to pass ownership of the child process to pid 1, what the command nohup does. A parent process when it exits, requires that all child processes exit as well.
- geocar 9y ago> fork() also clones all the filedescriptors that might be open to the child. ... Easy way to cause too many open files errors. fork() cannot fail with EMFILE or ENFILE[1] [1]: http://pubs.opengroup.org/onlinepubs/9699919799/functions/fork.html#tag_16_156_05 http://pubs.opengroup.org/onlinepubs/9699919799/functions/fo... > It is the resposibility of the forking code to close filedescriptors it doesn't need. FD_CLOEXEC can cause exec() to close file descriptors automatically. > A parent process when it exits, requires that all child processes exit as well. This is not true
- bluejekyll 9y ago> fork() cannot fail with EMFILE or ENFILE[1] This is true. But if you believe you're only closing files or sockets in the child process, then running into this issue is very simple. > > A parent process when it exits, requires that all child processes exit as well. > This is not true Sorry, you're right I was thinking of the SIGHUP to the child, it doesn't actually require an exit.
- temporary_art 9y agoWe did this as a class assignment in undergrad, definitely one of the most useful projects in college.
- Spakman 9y agoSeems like a good time to point to Urchin, the Bash-ish like shell I wrote that also allows writing Ruby directly on the command line. I've been using it happily for years but haven't packaged up a release for a recent version: https://github.com/Spakman/urchin https://github.com/Spakman/urchin
- jrimbault 9y agoHow did this get 86 points (as of writing this) ? It's basically the third chapter of every C course every CS student will get. And not very well done.
- marssaxman 9y agoPerhaps this "third chapter" is not as universal as you expect; nor, I believe, is a formal CS education as universal among the HN population as you seem to be assuming. Diving in and trying it out is a good way to learn. Sharing your results can be helpful for others, even if you haven't learned very much yet; it can be a good starting point, and others can correct your misunderstandings and comment on areas you might investigate further. I'm always happy to see people who have enough intellectual humility to show their incomplete work as they go, so that others can benefit from the learning process.
- LyndsySimon 9y agoAgreed. I've been a professional dev for a decade now, but failed out of college due to mental health issues (depression). I know a ton about stuff higher in the stack, but I've never spent much time writing C and mucking around with the OS itself. This project is interesting enough to me to be worth the time to read it, and it's given me the idea of writing my own shell as a means of learning a Nim.
- jrimbault 9y agoI concede on all your 3 points. I did not think.
- eriknstr 9y agoI have a pretty good idea for a shell but I don't know when I'll ever find time to implement it. The idea is that the shell should be able to optimize pipelines. Pipeline here meaning a chain of commands piped into one-another. So if you have a pipeline like grep ^2017-05-29 /var/log/somefile | grep -v 'INFO|WARN' | tail -n5 | cut -f1 -f3 Then the idea is that instead of piping the whole result from command to command, your shell would compile a set of commands into a single program and run that. Now you might say that that sounds like it goes against the Unix philosophy, but actually it doesn't need to. If all of the core utilities were implemented in such a way that their logic could be extracted without duplication of code then the shell can still be doing "one thing and one thing well". --- Another idea I have is to make these core utilities pipe objects instead of text, like on Windows. I am very fond of bash but I think one thing that Microsoft seems to have done right is to have the idea of being able to work on objects in PowerShell. But I don't want PowerShell. I want a mostly Unix shell, except, as I said, with objects. I just wish I had more time :(
- geocar 9y agoYou might find running across four cores is actually faster than "compiling into a single program"
- glangdale 9y agoHmmn. A 'smart enough' compiler would combine the two grep patterns into a single pattern and run them in reverse, stopping at the first 5 lines. :-) Even if we don't assume we have such astounding transformations, I think you're vastly overestimating the power of multithreading in this example. There's very little for each core to do, especially the final step (which is getting a whopping 5 lines to process). All that being said, I'm not sure that, if I wanted highly parallel text processing OR deeply clever compiler-style optimizations, I'd start from "existing Unix utilities and their associated cruft".
- eriknstr 9y agoNot everyone understood what I meant to say but you did, thanks :) Starting from the Unix utilites is simply because I'm so used to them. I can assure you, however, that anyone who was under the impression that my system was actually Unix would be in for a nasty surprise :-> My system would be, shall we say, not POSIX compliant. My system would take the things I like about Unix, such as being command-line centric and having utilities focused on being good at small individual tasks and using pipes to glue them together seemingly (but actually as mentioned each pipeline is compiled to an optimized program when run), and the filesystem would remain essential and hierarchical. I would separate Operating System binary data, Operating System config files, Application binary data, Application config files, User config files, User downloaded data and User created data STRICTLY! Also like I mentioned, to have structured binary data instead of text. This would lead to commands which are named like in POSIX but which operate a bit differently with regards to the arguments they take. Probably a lot of other things as well that I've been thinking about but which escapes me at the moment. But yeah, all of that remains a dream because it would involve more work than I have time for :(