4 ms·

Building a Shell

180 points by ingve 7mo ago

hristian 7mo ago
[flagged]
- Retr0id 7mo ago
  Fascinating that you resurrected an account from 2014 just for LLM spam, were the credentials compromised or something?
  - IncreasePosts 7mo ago
    Maybe the author had it logged into something that their claw had access to
- gf000 7mo ago
  Not sure it tells all that much about 'how the OS works'. This is a historical abstraction that happened to look how it looks today with all its numerous warts and shortcomings. We can easily imagine it done a better way - for all the criticism of Windows, PowerShell gives a glimpse into this hypothetical future.
- jmmv 7mo ago
  Somebody blamed this comment on LLMs, and maybe/probably it is, but I think the first sentence is spot-on so I thought it was worth replying to. Dealing with the corner cases ends up teaching you a lot about a language and for an ancient language like the shell, dealing with the corner cases also takes you through the thinking process of the original authors and the constraints they were subject to. I found myself in this situation while writing EndBASIC and wrote an article with the surprises I encountered, because I found the journey fascinating: https://www.endbasic.dev/2023/01/endbasic-parsing-difficulties.html https://www.endbasic.dev/2023/01/endbasic-parsing-difficulti...
zokier 7mo ago
Bit of pedantry but I don't think traditional unix shell (like this) follows repl model; the shell is not usually doing printing of the result of evaluation. Instead the printing happens more as a side effect of the commands.
- deleted 7mo ago
  [deleted]
- jermaustin1 7mo ago
  I remember my first shell programming I ever did was batch in windows back in the 3.11/95 days. The first line was always to turn off echo, and I've always wondered why that was a decision for batch script. Or I'm misremembering. 30 years of separation makes it hard to remember the details.
  - enoint 7mo ago
    Echo in that case prints command lines before executing them. Its analog is `set -x` rather than `echo`.
- skydhash 7mo ago
  It’s a shell, not the whole thing. The whole thing is the shell+kernel+programs.
  - zokier 7mo ago
    Even if you view the system as a whole the printing is deeply intertwined with the evaluation, which is very different from repl where eval returns a value and print prints it
- themafia 7mo ago
  It prints a prompt.
  - zokier 7mo ago
    That's not what print in repl means.
lvales 7mo ago
Building a shell is a great exercise, but honestly having to deal with string parsing is such a bother that it robs like 2/3 of the joy along the way. I once built a very simple one in Go [0] as a learning exercise and I stopped once I started getting frustrated with all the corner cases. [0] https://github.com/lourencovales/codecrafters/blob/master/shell-go/shell-go.go https://github.com/lourencovales/codecrafters/blob/master/sh...
- healeycodes 7mo ago
  Author here, and yeah, I agree. I skipped writing a parser altogether and just split on whitespace and `|` so that I could get to the interesting bits. For side-projects, I have to ask myself if I'm writing a parser, or if I'm building something else; e.g. for a toy programming language, it's way more fun to start with an AST and play around, and come back to the parser if you really fall in love with it.
- ferguess_k 7mo ago
  Can say the same for control characters in terminals. I even think maybe it's just easier to ditch them all and use QT to build a "terminal" with clickable urls, something similar to what TempleOS does.
- chubot 7mo ago
  A common problem I noticed is that if you took certain courses in computer science, you may have a pre-conceived notion of how to parse programming languages, and the shell language doesn't quite fit that model I have seen this misconception many times In Oils, we have some pretty minor elaborations of the standard model, and it makes things a lot easier How to Parse Shell Like a Programming Language - https://www.oilshell.org/blog/2019/02/07.html https://www.oilshell.org/blog/2019/02/07.html Everything I wrote there still holds, although that post could use some minor updates (and OSH is the most bash-compatible shell, and more POSIX-compatible than /bin/sh on Debian - e.g. https://pages.oils.pub/spec-compat/2025-11-02/renamed-tmp/spec/compat/TOP.html https://pages.oils.pub/spec-compat/2025-11-02/renamed-tmp/sp... ) --- To summarize that, I'd say that doing as much work as possible in the lexer, with regular languages and "lexer modes", drastically reduces the complexity of writing a shell parser And it's not just one parser -- shell actually has 5 to 15 different parsers, depending on how you count I often show this file to make that point: https://oils.pub/release/0.37.0/pub/src-tree.wwz/_gen/_tmp/match.re2c-input.h.html https://oils.pub/release/0.37.0/pub/src-tree.wwz/_gen/_tmp/m... (linked from https://oils.pub/release/0.37.0/quality.html https://oils.pub/release/0.37.0/quality.html) Fine-grained heterogenous algebraic data types also help. Shells in C tend to use a homogeneous command* and word* kind of representation https://oils.pub/release/0.37.0/pub/src-tree.wwz/frontend/syntax.asdl.html https://oils.pub/release/0.37.0/pub/src-tree.wwz/frontend/sy... (~700 lines of type definitions)
mzs 7mo ago
Had an assignment to build a shell in a week, how hard could it be? controlling terminal session leader job control The parser was easy in comparison.
emersion 7mo ago
Some time ago I've written an article about a particular aspect of shells, job control: https://emersion.fr/blog/2019/job-control/ https://emersion.fr/blog/2019/job-control/
leontloveless 7mo ago
[dead]
rigorclaw 7mo ago
[flagged]
austy69 7mo ago
Fun read. Wonder if you are able to edit text in the shell, or if you need to implement a gap buffer to allow it?
- healeycodes 7mo ago
  Editing the current line works because I brought in https://man7.org/linux/man-pages/man3/readline.3.html https://man7.org/linux/man-pages/man3/readline.3.html towards the end so I could support editing, tab completion, and history. IIRC readline uses a `char *` internally since the length of a user-edited line is fairly bounded.
  - austy69 7mo ago
    Very cool. Currently working on the beginning of a small text editor so this part seemed interesting and was curious of any overlap. Thanks for the interesting post!
  - zokier 7mo ago
    worth noting that you get basic line editing for "free" from kernels tty subsystem even if you don't use readline.
    - 1718627440 7mo ago
      Yes, but it is really basic. Is it more than backspace? Most cursor key presses are just forwarded to the program as escape sequences.
wei03288 7mo ago
[dead]
- chubot 7mo ago
  Yup, job control is a huge mess. I think Bill Joy was able to modify the shell, the syscall interface, and the terminal driver at the same time to implement the hacky mechanism of job control. But a few years later that kind of crosscutting change would have been harder One thing we learned from implementing job control in https://oils.pub https://oils.pub is that the differing pipeline semantics of bash and zsh makes a difference In bash, the last part of the pipeline is forked (unless shopt -s lastpipe) In zsh, it isn't $ bash -c 'echo hi | read x; echo $x' # no output $ zsh -c 'echo hi | read x; echo $x' hi And then that affects this case: bash$ sleep 5 | read ^Z [1]+ Stopped sleep 5 | read zsh$ sleep 5 | read # job control doesn't apply to this case in zsh ^Zzsh: job can't be suspended So yeah the semantics of shell are not very well specified (which is one reason for OSH and YSH). I recall a bug running an Alpine Linux shell script where this difference matters -- if the last part is NOT forked, then the script doesn't run I think there was almost a "double bug" -- the script relied on the `read` output being "lost", even though that was likely not the intended behavior
  - chubot 7mo ago
    FWIW here is another piece of trivia about job control: the API means you can't spawn a child process "safely" in POSIX -- you have to trust that that the executable you're spawning is well-behaved (or use more advanced Linux process isolation) In this case it was the Zed editor spawning the zsh shell: How to Lose Control of your Shell - https://registerspill.thorstenball.com/p/how-to-lose-control-of-your-shell https://registerspill.thorstenball.com/p/how-to-lose-control... zsh has a bug where it doesn't do the job control cleanup properly in some cases -- when fork-exec() optimizations happen. This can mess up the calling process. For example, here you could no longer kill Zed by hitting Ctrl-C, even after zsh is done. My comment: https://lobste.rs/s/hru0ib/how_lose_control_your_shell#c_dfluhm https://lobste.rs/s/hru0ib/how_lose_control_your_shell#c_dfl...
- sloum 7mo ago
hexer303 7mo ago
Unix shells are conceptually simple but hide a surprising amount of complexity under the hood that we take for granted. I recently had build my own PTY controller. There were so many edge-cases to deal with. It took weeks of stress testing and writing many tests to get it right.
dirk94018 7mo ago
Interesting. I wanted to do toast | bash to let the AI drive the computer but the bash shell really got in the way. Too much complexity. The things that annoy humans, $ expansion, special characters, etc don't work for AI either. Ended up writing a custom shell for AI (and humans). When a tool gets in the way, sometimes it just time to change the tool.
lioeters 7mo ago
Link was previously posted by author: https://news.ycombinator.com/item?id=47398749 https://news.ycombinator.com/item?id=47398749 There are other good quality articles on their site, and maybe deserves the imaginary points.
ratzkewatzke 7mo ago
There's a very good exercise on Codecrafters (https://app.codecrafters.io/courses/shell/overview https://app.codecrafters.io/courses/shell/overview) to walk you through writing your own shell. I found it enlightening, as well as a good way to learn a new language.
rrampage 7mo ago
Fun read! I built a minimal Linux shell [0] in c and Zig last year which does not depend on libc. It was a great way to learn about execve, the new-ish clone3 syscall and how Linux starts a process. Parsing strings is the least fun part of the building the shell. [0] https://gist.github.com/rrampage/5046b60ca2d040bcffb49ee38e86041f https://gist.github.com/rrampage/5046b60ca2d040bcffb49ee38e8...
doe88 7mo ago
Is there a (real) shell whose code is relatively short and self contained and would be valuable to read? This was always something I wanted to do but never quite spent time to look for a good one to explore.
- giancarlostoro 7mo ago
  Although not the same... Destroy All Software has videos on building your own shell using Ruby. I watched it to learn and it was a lot of fun to watch him basically building a shell, I'm not really a Ruby guy, but it was easy to grasp. It's not free, you would need a subscription, but its worth the watch otherwise. https://www.destroyallsoftware.com/screencasts/catalog/shell-from-scratch https://www.destroyallsoftware.com/screencasts/catalog/shell...
- epr 7mo ago
  I think there's a good one if you search around for "xv6 sh.c". Hard to tell immediately from a google search just now since there are many implementations (people do it in school) and github's currently blocking requests from my phone. Also helpful may be running strace on your shell, then reviewing the output line by line to make sure you understand each. This is a VERY instructive exercise to do in general.
- willx86 7mo ago
  https://st.suckless.org/ https://st.suckless.org/
  - vidarh 7mo ago
    St is a terminal, not a shell.
- tame3902 7mo ago
  It depends on what you are looking for. My recommendation for learning "how is X done in a shell" is the OpenBSD ksh: https://github.com/ibara/oksh https://github.com/ibara/oksh It's what they use for /bin/sh, it has everything that a complete shell needs (including a mechanism for providing command completions) and has code that is much easier to read than bash or zsh. Something that I also would recommend is the design document for the plan9 rc shell; it is a worthwhile read for anybody interested in shells: https://doc.cat-v.org/plan_9/4th_edition/papers/rc https://doc.cat-v.org/plan_9/4th_edition/papers/rc An implementation is also available if one wants to look at how it could be done: https://github.com/rakitzis/rc https://github.com/rakitzis/rc
stainlu 7mo ago
[dead]
lasgawe 7mo ago
Great article. There are many things every developer should do when starting to learn programming or when trying to improve their skills. This is one of them. I once built a shell-like programming language (not an interpreter). If anyone reading this wants to improve their skills, I strongly suggest building your own shell from scratch.
Heer_J 7mo ago
[dead]

Building a Shell

180 points by ingve 7mo ago

hristian 7mo ago

[flagged]

Retr0id 7mo ago
Fascinating that you resurrected an account from 2014 just for LLM spam, were the credentials compromised or something?
- IncreasePosts 7mo ago
  Maybe the author had it logged into something that their claw had access to
gf000 7mo ago
Not sure it tells all that much about 'how the OS works'. This is a historical abstraction that happened to look how it looks today with all its numerous warts and shortcomings. We can easily imagine it done a better way - for all the criticism of Windows, PowerShell gives a glimpse into this hypothetical future.
jmmv 7mo ago
Somebody blamed this comment on LLMs, and maybe/probably it is, but I think the first sentence is spot-on so I thought it was worth replying to. Dealing with the corner cases ends up teaching you a lot about a language and for an ancient language like the shell, dealing with the corner cases also takes you through the thinking process of the original authors and the constraints they were subject to. I found myself in this situation while writing EndBASIC and wrote an article with the surprises I encountered, because I found the journey fascinating: https://www.endbasic.dev/2023/01/endbasic-parsing-difficulties.html https://www.endbasic.dev/2023/01/endbasic-parsing-difficulti...

zokier 7mo ago

Bit of pedantry but I don't think traditional unix shell (like this) follows repl model; the shell is not usually doing printing of the result of evaluation. Instead the printing happens more as a side effect of the commands.

deleted 7mo ago
[deleted]
jermaustin1 7mo ago
I remember my first shell programming I ever did was batch in windows back in the 3.11/95 days. The first line was always to turn off echo, and I've always wondered why that was a decision for batch script. Or I'm misremembering. 30 years of separation makes it hard to remember the details.
- enoint 7mo ago
  Echo in that case prints command lines before executing them. Its analog is `set -x` rather than `echo`.
skydhash 7mo ago
It’s a shell, not the whole thing. The whole thing is the shell+kernel+programs.
- zokier 7mo ago
  Even if you view the system as a whole the printing is deeply intertwined with the evaluation, which is very different from repl where eval returns a value and print prints it
themafia 7mo ago
It prints a prompt.
- zokier 7mo ago
  That's not what print in repl means.

lvales 7mo ago

Building a shell is a great exercise, but honestly having to deal with string parsing is such a bother that it robs like 2/3 of the joy along the way. I once built a very simple one in Go [0] as a learning exercise and I stopped once I started getting frustrated with all the corner cases. [0] https://github.com/lourencovales/codecrafters/blob/master/shell-go/shell-go.go https://github.com/lourencovales/codecrafters/blob/master/sh...

healeycodes 7mo ago
Author here, and yeah, I agree. I skipped writing a parser altogether and just split on whitespace and `|` so that I could get to the interesting bits. For side-projects, I have to ask myself if I'm writing a parser, or if I'm building something else; e.g. for a toy programming language, it's way more fun to start with an AST and play around, and come back to the parser if you really fall in love with it.
ferguess_k 7mo ago
Can say the same for control characters in terminals. I even think maybe it's just easier to ditch them all and use QT to build a "terminal" with clickable urls, something similar to what TempleOS does.
chubot 7mo ago
A common problem I noticed is that if you took certain courses in computer science, you may have a pre-conceived notion of how to parse programming languages, and the shell language doesn't quite fit that model I have seen this misconception many times In Oils, we have some pretty minor elaborations of the standard model, and it makes things a lot easier How to Parse Shell Like a Programming Language - https://www.oilshell.org/blog/2019/02/07.html https://www.oilshell.org/blog/2019/02/07.html Everything I wrote there still holds, although that post could use some minor updates (and OSH is the most bash-compatible shell, and more POSIX-compatible than /bin/sh on Debian - e.g. https://pages.oils.pub/spec-compat/2025-11-02/renamed-tmp/spec/compat/TOP.html https://pages.oils.pub/spec-compat/2025-11-02/renamed-tmp/sp... ) --- To summarize that, I'd say that doing as much work as possible in the lexer, with regular languages and "lexer modes", drastically reduces the complexity of writing a shell parser And it's not just one parser -- shell actually has 5 to 15 different parsers, depending on how you count I often show this file to make that point: https://oils.pub/release/0.37.0/pub/src-tree.wwz/_gen/_tmp/match.re2c-input.h.html https://oils.pub/release/0.37.0/pub/src-tree.wwz/_gen/_tmp/m... (linked from https://oils.pub/release/0.37.0/quality.html https://oils.pub/release/0.37.0/quality.html) Fine-grained heterogenous algebraic data types also help. Shells in C tend to use a homogeneous command* and word* kind of representation https://oils.pub/release/0.37.0/pub/src-tree.wwz/frontend/syntax.asdl.html https://oils.pub/release/0.37.0/pub/src-tree.wwz/frontend/sy... (~700 lines of type definitions)

mzs 7mo ago

Had an assignment to build a shell in a week, how hard could it be? controlling terminal session leader job control The parser was easy in comparison.

emersion 7mo ago

Some time ago I've written an article about a particular aspect of shells, job control: https://emersion.fr/blog/2019/job-control/ https://emersion.fr/blog/2019/job-control/

leontloveless 7mo ago

[dead]

rigorclaw 7mo ago

[flagged]

austy69 7mo ago

Fun read. Wonder if you are able to edit text in the shell, or if you need to implement a gap buffer to allow it?

healeycodes 7mo ago
Editing the current line works because I brought in https://man7.org/linux/man-pages/man3/readline.3.html https://man7.org/linux/man-pages/man3/readline.3.html towards the end so I could support editing, tab completion, and history. IIRC readline uses a `char *` internally since the length of a user-edited line is fairly bounded.
- austy69 7mo ago
  Very cool. Currently working on the beginning of a small text editor so this part seemed interesting and was curious of any overlap. Thanks for the interesting post!
- zokier 7mo ago
  worth noting that you get basic line editing for "free" from kernels tty subsystem even if you don't use readline.
  - 1718627440 7mo ago
    Yes, but it is really basic. Is it more than backspace? Most cursor key presses are just forwarded to the program as escape sequences.

wei03288 7mo ago

[dead]

chubot 7mo ago
Yup, job control is a huge mess. I think Bill Joy was able to modify the shell, the syscall interface, and the terminal driver at the same time to implement the hacky mechanism of job control. But a few years later that kind of crosscutting change would have been harder One thing we learned from implementing job control in https://oils.pub https://oils.pub is that the differing pipeline semantics of bash and zsh makes a difference In bash, the last part of the pipeline is forked (unless shopt -s lastpipe) In zsh, it isn't $ bash -c 'echo hi | read x; echo $x' # no output $ zsh -c 'echo hi | read x; echo $x' hi And then that affects this case: bash$ sleep 5 | read ^Z [1]+ Stopped sleep 5 | read zsh$ sleep 5 | read # job control doesn't apply to this case in zsh ^Zzsh: job can't be suspended So yeah the semantics of shell are not very well specified (which is one reason for OSH and YSH). I recall a bug running an Alpine Linux shell script where this difference matters -- if the last part is NOT forked, then the script doesn't run I think there was almost a "double bug" -- the script relied on the `read` output being "lost", even though that was likely not the intended behavior
- chubot 7mo ago
  FWIW here is another piece of trivia about job control: the API means you can't spawn a child process "safely" in POSIX -- you have to trust that that the executable you're spawning is well-behaved (or use more advanced Linux process isolation) In this case it was the Zed editor spawning the zsh shell: How to Lose Control of your Shell - https://registerspill.thorstenball.com/p/how-to-lose-control-of-your-shell https://registerspill.thorstenball.com/p/how-to-lose-control... zsh has a bug where it doesn't do the job control cleanup properly in some cases -- when fork-exec() optimizations happen. This can mess up the calling process. For example, here you could no longer kill Zed by hitting Ctrl-C, even after zsh is done. My comment: https://lobste.rs/s/hru0ib/how_lose_control_your_shell#c_dfluhm https://lobste.rs/s/hru0ib/how_lose_control_your_shell#c_dfl...
sloum 7mo ago

hexer303 7mo ago

Unix shells are conceptually simple but hide a surprising amount of complexity under the hood that we take for granted. I recently had build my own PTY controller. There were so many edge-cases to deal with. It took weeks of stress testing and writing many tests to get it right.

dirk94018 7mo ago

Interesting. I wanted to do toast | bash to let the AI drive the computer but the bash shell really got in the way. Too much complexity. The things that annoy humans, $ expansion, special characters, etc don't work for AI either. Ended up writing a custom shell for AI (and humans). When a tool gets in the way, sometimes it just time to change the tool.

lioeters 7mo ago

Link was previously posted by author: https://news.ycombinator.com/item?id=47398749 https://news.ycombinator.com/item?id=47398749 There are other good quality articles on their site, and maybe deserves the imaginary points.

ratzkewatzke 7mo ago

There's a very good exercise on Codecrafters (https://app.codecrafters.io/courses/shell/overview https://app.codecrafters.io/courses/shell/overview) to walk you through writing your own shell. I found it enlightening, as well as a good way to learn a new language.

rrampage 7mo ago

Fun read! I built a minimal Linux shell [0] in c and Zig last year which does not depend on libc. It was a great way to learn about execve, the new-ish clone3 syscall and how Linux starts a process. Parsing strings is the least fun part of the building the shell. [0] https://gist.github.com/rrampage/5046b60ca2d040bcffb49ee38e86041f https://gist.github.com/rrampage/5046b60ca2d040bcffb49ee38e8...

doe88 7mo ago

Is there a (real) shell whose code is relatively short and self contained and would be valuable to read? This was always something I wanted to do but never quite spent time to look for a good one to explore.

giancarlostoro 7mo ago
Although not the same... Destroy All Software has videos on building your own shell using Ruby. I watched it to learn and it was a lot of fun to watch him basically building a shell, I'm not really a Ruby guy, but it was easy to grasp. It's not free, you would need a subscription, but its worth the watch otherwise. https://www.destroyallsoftware.com/screencasts/catalog/shell-from-scratch https://www.destroyallsoftware.com/screencasts/catalog/shell...
epr 7mo ago
I think there's a good one if you search around for "xv6 sh.c". Hard to tell immediately from a google search just now since there are many implementations (people do it in school) and github's currently blocking requests from my phone. Also helpful may be running strace on your shell, then reviewing the output line by line to make sure you understand each. This is a VERY instructive exercise to do in general.
willx86 7mo ago
https://st.suckless.org/ https://st.suckless.org/
- vidarh 7mo ago
  St is a terminal, not a shell.
tame3902 7mo ago
It depends on what you are looking for. My recommendation for learning "how is X done in a shell" is the OpenBSD ksh: https://github.com/ibara/oksh https://github.com/ibara/oksh It's what they use for /bin/sh, it has everything that a complete shell needs (including a mechanism for providing command completions) and has code that is much easier to read than bash or zsh. Something that I also would recommend is the design document for the plan9 rc shell; it is a worthwhile read for anybody interested in shells: https://doc.cat-v.org/plan_9/4th_edition/papers/rc https://doc.cat-v.org/plan_9/4th_edition/papers/rc An implementation is also available if one wants to look at how it could be done: https://github.com/rakitzis/rc https://github.com/rakitzis/rc

stainlu 7mo ago

[dead]

lasgawe 7mo ago

Great article. There are many things every developer should do when starting to learn programming or when trying to improve their skills. This is one of them. I once built a shell-like programming language (not an interpreter). If anyone reading this wants to improve their skills, I strongly suggest building your own shell from scratch.

Heer_J 7mo ago

[dead]