4 ms·

So he writes Oil Shell in Python because he wants to get it done quicker than he would writing in C. And yet now he's found that the performance in Python is in

So he writes Oil Shell in Python because he wants to get it done quicker than he would writing in C. And yet now he's found that the performance in Python is inadequate for a shell to compete with Bash (pretty predictable) and so he's basically writing a new Python interpreter (both bytecode compiler and VM), which (as he'll find out) is a huge task, regardless of how simplistic he makes the interpreter. That said, I think Oil Shell the shell language looks pretty cool, so I wish he had done it in a language like C, Rust, or even OCaml (which he apparently considered).

chubot 9y ago
(author here) Well, a key point is that it doesn't have to be an entire Python interpreter. It just has to be the subset that Oil uses. (This applies to both both Python syntax and semantics.) I also forked the 8K LOC bytecode compiler -- I didn't write it from scratch. That's the point of "Cobbling together a Python interpreter". So for a cartoonish view, compare these implementation strategies. 1. Clone bash in C, which (charitably) would involve writing around 80K lines of C code from scratch, given that bash is 160K lines. You can probably do better by not writing it over 30 years :-) 2. Write 16K lines of Python, fork a bytecode compiler in Python (8K lines), and then either fork CPython's VM or write your own. The post implies that #2 is probably easier than #1. I haven't done the last part of #2, so I could be wrong. I think it's an interesting experiment to find out whether you can write a bespoke compiler/VM for a specific program in a high level language. ----- It's a little bit like TeX and Pascal. I discovered awhile ago that that TeX is written in an abstract subset of Pascal. Apparently the version you use on most Linux distros is compiled to machine code via a Pascal-to-C translator. I'm not sure if that translator was written specifically for TeX or not, but if anyone knows, please chime in! But I have seen enough of Knuth's code to know that he does not rely on implementation details. That is roughly the case with Oil and Python. As I mention in the post, Oil is written in Python+ASDL, not Python. And I would even call it Python + ASDL + some abstract regex subset. That subset could be called the intersection of Python regexes + re2c regexes, e.g. at the end of this comment: https://news.ycombinator.com/item?id=16525378 https://news.ycombinator.com/item?id=16525378 Basically when writing Oil, I try to be clear about the algorithms, and not think in terms of implementation details of a particular platform. This is possible for a shell because it's mostly string handling, and the only libraries it uses are a handful of libc calls (which are 40 years old, standardized, and well understood.)
- svat 9y ago
  Since you asked: Yes, the Pascal-to-C translator was written specifically for TeX. To be more precise (and add more detail than anyone asked for!): - TeX is not written in Pascal itself, but in a literate-programming system called WEB [1,2] (a language basically created for writing the TeX program in!), which lets you mix Pascal source code with TeX documentation, and lets you write your program as a “web” of “sections”, each independently understandable, and referring to other sections (with macros etc). The “tangle” program converts the WEB source code to Pascal source code. - The Pascal used in the TeX program is a limited subset [3] of Pascal (and specifically of what Knuth called “Pascal-H”, the Pascal implementation that was available to him on the DEC PDP-10 system used at Stanford SAIL). - It is not exactly true that he does not rely on implementation details, but he's conscious about where he does so, and he's included a list of all such places in the index under “system dependencies” [4] — in most places he gives suggestions on what could be done if the compiler didn't work the same way. - In the early days, when Pascal was the most common language available at the places TeX was usually run (universities, research labs, etc.), the Pascal code (generated by Tangle) was directly compiled into a Pascal program and run. - With the rise of C (into the wider world outside Bell Labs), there arose both hand-translations of the WEB (or Pascal) source code, and programs to do this automatically. - Today, major TeX distributions have their own Pascal(WEB)-to-C converters, written specifically for the TeX (and METAFONT) program. For example, TeX Live uses web2c[5], MiKTeX uses its own “C4P”[6], and even the more obscure distributions like KerTeX[7] have their own WEB/Pascal-to-C translators. One interesting project is web2w[8,9], which translates the TeX program from WEB (the Pascal-based literate programming system) to CWEB (the C-based literate programming system). - The only exception I'm aware of (that does not translate WEB or Pascal to C) is the TeX-GPC distribution [10,11,12], which makes only the changes needed to get the TeX program running with a modern Pascal compiler (GPC, GNU Pascal). [1]: http://literateprogramming.com/knuthweb.pdf http://literateprogramming.com/knuthweb.pdf [2]: http://mirrors.ctan.org/info/knuth/webman.pdf http://mirrors.ctan.org/info/knuth/webman.pdf [3]: http://texdoc.net/texmf-dist/doc/generic/knuth/tex/tex.pdf#page=3 http://texdoc.net/texmf-dist/doc/generic/knuth/tex/tex.pdf#p... (section 3) [4]: http://texdoc.net/texmf-dist/doc/generic/knuth/tex/tex.pdf#page=514 http://texdoc.net/texmf-dist/doc/generic/knuth/tex/tex.pdf#p... (entry “system dependencies”) [5]: https://tug.org/web2c/ https://tug.org/web2c/ [6]: http://tug.org/interviews/schenk.html http://tug.org/interviews/schenk.html [7]: https://tex.stackexchange.com/questions/111332/how-to-compile-the-source-code-of-tex/111371#111371 https://tex.stackexchange.com/questions/111332/how-to-compil... [8]: https://w3-o.cs.hm.edu/~ruckert/web2w/index.html https://w3-o.cs.hm.edu/~ruckert/web2w/index.html [9]: https://ctan.org/pkg/web2w https://ctan.org/pkg/web2w [10]: http://www.tex.ac.uk/FAQ-sysunix.html http://www.tex.ac.uk/FAQ-sysunix.html [11]: http://mirrors.ctan.org/systems/unix/tex-gpc/README http://mirrors.ctan.org/systems/unix/tex-gpc/README [12]: http://mirrors.ctan.org/systems/unix/tex-gpc/tex.pdf http://mirrors.ctan.org/systems/unix/tex-gpc/tex.pdf
kristianp 9y ago
Indeed, I feel like the author loves Python so much and also loves creating parsers and compilers so much, that they've gone down a rabbit hole. Using a practical, "fast enough" language like Go or a language that allows for DSLs, such as scheme or common lisp would have been better.
- chubot 9y ago
  I definitely love Python, but I started with C++ as explained in the first link in the post. I also did an experiment with femtolisp, because it was successfully used to bootstrap Julia (and has some interesting connections to its macro system). I admit the way I'm doing things is quirky (as I do in the blog post), but if you look at how femtolisp and Julia work, it's also quirky. The authors clearly love Lisp. Enough that they wrote their OWN Lisp -- I did not write my own Python front end, as explained in this post. But hey, it works. We'll see if my strategy works. I don't think Go is a great choice for a "true shell", first because it doesn't use libc (which can be worked around), but also because it only exposes ForkExec() portably and syscall.Exec non-portably, due to the migration of goroutines between threads. You can probably make it work, but you're going to be fighting it. Go makes the shell less portable, not more. Python is a lot more straightforward since it's written in C and integrates with C. The runtime doesn't start threads behind your back, etc. I should probably write a FAQ about this since it keeps coming up. https://www.reddit.com/r/ProgrammingLanguages/comments/81wkgv/building_oil_with_the_opy_bytecode_compiler/ https://www.reddit.com/r/ProgrammingLanguages/comments/81wkg...

chubot 9y ago

(author here) Well, a key point is that it doesn't have to be an entire Python interpreter. It just has to be the subset that Oil uses. (This applies to both both Python syntax and semantics.) I also forked the 8K LOC bytecode compiler -- I didn't write it from scratch. That's the point of "Cobbling together a Python interpreter". So for a cartoonish view, compare these implementation strategies. 1. Clone bash in C, which (charitably) would involve writing around 80K lines of C code from scratch, given that bash is 160K lines. You can probably do better by not writing it over 30 years :-) 2. Write 16K lines of Python, fork a bytecode compiler in Python (8K lines), and then either fork CPython's VM or write your own. The post implies that #2 is probably easier than #1. I haven't done the last part of #2, so I could be wrong. I think it's an interesting experiment to find out whether you can write a bespoke compiler/VM for a specific program in a high level language. ----- It's a little bit like TeX and Pascal. I discovered awhile ago that that TeX is written in an abstract subset of Pascal. Apparently the version you use on most Linux distros is compiled to machine code via a Pascal-to-C translator. I'm not sure if that translator was written specifically for TeX or not, but if anyone knows, please chime in! But I have seen enough of Knuth's code to know that he does not rely on implementation details. That is roughly the case with Oil and Python. As I mention in the post, Oil is written in Python+ASDL, not Python. And I would even call it Python + ASDL + some abstract regex subset. That subset could be called the intersection of Python regexes + re2c regexes, e.g. at the end of this comment: https://news.ycombinator.com/item?id=16525378 https://news.ycombinator.com/item?id=16525378 Basically when writing Oil, I try to be clear about the algorithms, and not think in terms of implementation details of a particular platform. This is possible for a shell because it's mostly string handling, and the only libraries it uses are a handful of libc calls (which are 40 years old, standardized, and well understood.)

svat 9y ago
Since you asked: Yes, the Pascal-to-C translator was written specifically for TeX. To be more precise (and add more detail than anyone asked for!): - TeX is not written in Pascal itself, but in a literate-programming system called WEB [1,2] (a language basically created for writing the TeX program in!), which lets you mix Pascal source code with TeX documentation, and lets you write your program as a “web” of “sections”, each independently understandable, and referring to other sections (with macros etc). The “tangle” program converts the WEB source code to Pascal source code. - The Pascal used in the TeX program is a limited subset [3] of Pascal (and specifically of what Knuth called “Pascal-H”, the Pascal implementation that was available to him on the DEC PDP-10 system used at Stanford SAIL). - It is not exactly true that he does not rely on implementation details, but he's conscious about where he does so, and he's included a list of all such places in the index under “system dependencies” [4] — in most places he gives suggestions on what could be done if the compiler didn't work the same way. - In the early days, when Pascal was the most common language available at the places TeX was usually run (universities, research labs, etc.), the Pascal code (generated by Tangle) was directly compiled into a Pascal program and run. - With the rise of C (into the wider world outside Bell Labs), there arose both hand-translations of the WEB (or Pascal) source code, and programs to do this automatically. - Today, major TeX distributions have their own Pascal(WEB)-to-C converters, written specifically for the TeX (and METAFONT) program. For example, TeX Live uses web2c[5], MiKTeX uses its own “C4P”[6], and even the more obscure distributions like KerTeX[7] have their own WEB/Pascal-to-C translators. One interesting project is web2w[8,9], which translates the TeX program from WEB (the Pascal-based literate programming system) to CWEB (the C-based literate programming system). - The only exception I'm aware of (that does not translate WEB or Pascal to C) is the TeX-GPC distribution [10,11,12], which makes only the changes needed to get the TeX program running with a modern Pascal compiler (GPC, GNU Pascal). [1]: http://literateprogramming.com/knuthweb.pdf http://literateprogramming.com/knuthweb.pdf [2]: http://mirrors.ctan.org/info/knuth/webman.pdf http://mirrors.ctan.org/info/knuth/webman.pdf [3]: http://texdoc.net/texmf-dist/doc/generic/knuth/tex/tex.pdf#page=3 http://texdoc.net/texmf-dist/doc/generic/knuth/tex/tex.pdf#p... (section 3) [4]: http://texdoc.net/texmf-dist/doc/generic/knuth/tex/tex.pdf#page=514 http://texdoc.net/texmf-dist/doc/generic/knuth/tex/tex.pdf#p... (entry “system dependencies”) [5]: https://tug.org/web2c/ https://tug.org/web2c/ [6]: http://tug.org/interviews/schenk.html http://tug.org/interviews/schenk.html [7]: https://tex.stackexchange.com/questions/111332/how-to-compile-the-source-code-of-tex/111371#111371 https://tex.stackexchange.com/questions/111332/how-to-compil... [8]: https://w3-o.cs.hm.edu/~ruckert/web2w/index.html https://w3-o.cs.hm.edu/~ruckert/web2w/index.html [9]: https://ctan.org/pkg/web2w https://ctan.org/pkg/web2w [10]: http://www.tex.ac.uk/FAQ-sysunix.html http://www.tex.ac.uk/FAQ-sysunix.html [11]: http://mirrors.ctan.org/systems/unix/tex-gpc/README http://mirrors.ctan.org/systems/unix/tex-gpc/README [12]: http://mirrors.ctan.org/systems/unix/tex-gpc/tex.pdf http://mirrors.ctan.org/systems/unix/tex-gpc/tex.pdf

kristianp 9y ago

Indeed, I feel like the author loves Python so much and also loves creating parsers and compilers so much, that they've gone down a rabbit hole. Using a practical, "fast enough" language like Go or a language that allows for DSLs, such as scheme or common lisp would have been better.

chubot 9y ago
I definitely love Python, but I started with C++ as explained in the first link in the post. I also did an experiment with femtolisp, because it was successfully used to bootstrap Julia (and has some interesting connections to its macro system). I admit the way I'm doing things is quirky (as I do in the blog post), but if you look at how femtolisp and Julia work, it's also quirky. The authors clearly love Lisp. Enough that they wrote their OWN Lisp -- I did not write my own Python front end, as explained in this post. But hey, it works. We'll see if my strategy works. I don't think Go is a great choice for a "true shell", first because it doesn't use libc (which can be worked around), but also because it only exposes ForkExec() portably and syscall.Exec non-portably, due to the migration of goroutines between threads. You can probably make it work, but you're going to be fighting it. Go makes the shell less portable, not more. Python is a lot more straightforward since it's written in C and integrates with C. The runtime doesn't start threads behind your back, etc. I should probably write a FAQ about this since it keeps coming up. https://www.reddit.com/r/ProgrammingLanguages/comments/81wkgv/building_oil_with_the_opy_bytecode_compiler/ https://www.reddit.com/r/ProgrammingLanguages/comments/81wkg...