6 ms·
Writing a basic x86-64 JIT compiler from scratch in stock Python
- pjmlp 9y agoLove the idea of only using stock Python. Congratulations on the article.
- kovrik 9y agoThat is awesome! Thanks! Does anybody know the easiest way to compile to JVM bytecode? I have a Scheme interpreter written in Kotlin, what is the best way to compile it, instead of interpreting? Where do I start?
- jcdavis 9y agoThe most widely used library for dynamic bytecode generation is asm: http://asm.ow2.org/ http://asm.ow2.org/
- tomp 9y agoAnother possibility is using Truffle & Graal.
- deleted 9y ago[deleted]
- pjmlp 9y agoAnother alternative to jcdavis answer, is to just output bytecode in text format and use jasmin to compile it. http://jasmin.sourceforge.net/ http://jasmin.sourceforge.net/ Or have a lookt at Kawa. https://www.gnu.org/software/kawa/ https://www.gnu.org/software/kawa/
- doall 9y agoExamples of Lisp-2 compiler using Java and ASM: Shen to JVM bytecode compiler in Java 6 https://github.com/otabat/shen-jvm https://github.com/otabat/shen-jvm Shen to JVM bytecode compiler in Java 8 with indy https://github.com/hraberg/Shen.java https://github.com/hraberg/Shen.java
- bjpbakker 9y agoFor one thing Clojure comes to mind: https://github.com/clojure/clojure https://github.com/clojure/clojure
- UncleEntity 9y agoI was poking around* with this yesterday, has two different simple implementation of a lispkit compiler: http://www.cs.ncl.ac.uk/publications/trs/papers/129.pdf http://www.cs.ncl.ac.uk/publications/trs/papers/129.pdf *by "poking around" I mean I typed the code into a text editor for further study: code: https://pastebin.com/6pAkqE2R https://pastebin.com/6pAkqE2R
- tamer_c 9y agooff topic question: When do you find time do such interesting things as these? I'd really like to dive deeper in this topic. As a CS student, who also got to work as a dev at the same time for a living, I'm even happy to get to sleep 8 hours.
- csl 9y agoAlmost always in evenings, whenever I have time and feel like it. If you're a student, you're probably hosed with deadlines, so pick very small things to try out — things like really simple interpreters. This particular project luckily turned out to be quick to get working. Also, I kept the scope very small, calling it a day right before getting big ideas.
- OxO4 9y agoWhy not take a class about compilers and implement something like that as a class project?
- dboreham 9y agoWhere's the fun in that.
- dsco 9y agoJust what I love, a scientific and pedagogic CS article, in well formulated English. I would appreciate some introductory notes or links to prerequisites though. That would perhaps entice an audience who has no previous experience in writing compilers.
- giancarlostoro 9y agoI really want to be able to wrap my head around compilers but I'm not sure what a good "user friendly" resource is that wont stuff lots of fancy compiler buzzwords in my face and become white noise to me.
- shakna 9y agoLisp In Small Pieces would be my reccomendation. A relatively small book that takes you through writing 11 interpreters and 2 compilers.
- csl 9y agoIs it really that friendly when starting out, though? I found it brilliant, but quite the challenging read. I'd rather just read chapters four and five of SICP [1] and move on to Queinnec as a follow-up on more advanced topics. [1]: The book is free as well: https://mitpress.mit.edu/sicp/full-text/book/book-Z-H-4.html#%_toc_start https://mitpress.mit.edu/sicp/full-text/book/book-Z-H-4.html...
- shakna 9y agoTo be honest, I found LISP to be a lot easier than SICP, but that's probably just some learning styles stuff at play. SICP being freely available in both book form, and the lectures [0] makes it one of the best resources out there. (I just never found it's section on compilers that helpful. Again, probably learning styles.) [0] https://www.youtube.com/playlist?list=PLB63C06FAF154F047 https://www.youtube.com/playlist?list=PLB63C06FAF154F047
- 9y ago
- one_thawt 9y agoIn my opinion, the most tedious part of writing this sort of thing from scratch is the x86/x86-64 instruction encoding. Grab the Intel/AMD manuals or an opcode database and have fun. Other projects to study: LLVM, dynasm, asmjit, peachpy, luajit jit, Intel XED, etc.
- blt 9y agoyeah, it sucks. That is what turned me off of initial messing around with JITs. Wish I had known about GNU Lightning then.
- eliben 9y agoFWIW, the series concluding at https://eli.thegreenplace.net/2017/adventures-in-jit-compilation-part-4-in-python/ https://eli.thegreenplace.net/2017/adventures-in-jit-compila... build a Brainf*ck JIT in pure C++, then using LLVM, then in Python using PeachPy
- fasquoika 9y ago>dynasm, ... luajit jit Aren't these the same thing?
- Jasper_ 9y agoTo be clear about the `del block` thing, the only thing that the `del` keyword does in Python is unbind a name from the local scope. It does not cause an object to be destructed, call `__del__`, or anything else. It's morally equivalent to "block = None", except future references to block will raise a NameError instead of giving you None. When used at the end of a function, when all locals go out of scope, it is actually doing nothing.
- ben-schaaf 9y agoUnbinding a name does do one thing which might call `__del__`: It decreases the reference count by 1.
- Bojjagani 9y agogood performance
- peteforde 9y agoI am reminded of a most excellent piece of science fiction, "Coding Machines". https://www.teamten.com/lawrence/writings/coding-machines/ https://www.teamten.com/lawrence/writings/coding-machines/
- defined 9y agoThis is a very interesting and well-written article, but I am puzzled by the technique being described as "JIT compilation". Isn't JIT compilation the process of compiling some sort of intermediate code into machine code at runtime? For example, compiling JVM bytecode into machine code during execution, and using the cached machine code transparently on subsequent calls to that segment of code. Not to detract from the article, or the interesting techniques and explanation, but I didn't see any compilation other than by gcc, which IMHO makes this AOT rather than JIT compilation. What am I missing?
- rdc12 9y agoFor me the key concept that comes to mind for a JIT compiler, is profiling at runtime to decide what to compile and how aggressively which this article doesn't deal with. The other big thing is how to make the profiling and compilation not be too expensive to negate itself. The article author seems to be using GCC not at runtime, but more as a generator for templates (at the machine code level) for the rest of the software to use. You don't really want to be assembling a template at runtime in this context.
- MaxBarraclough 9y agoMixed-mode execution (interpreter+JIT when justified) isn't the only way to go. .Net doesn't work that way, for instance. Unlike Java's HotSpot, .Net never interprets CIL instructions. They're always compiled down to native code for execution. At least, that was true in 2009, according to Jon Skeet - https://stackoverflow.com/a/1255832/2307853 https://stackoverflow.com/a/1255832/2307853
- defined 9y ago> .Net never interprets CIL instructions. I wasn't aware of this. To be clear, it's not 100% accurate to say "never", because it depends on the .NET environment, but point taken. I now understand that .NET (on desktop, anyway) compiles all intermediate code (CIL) to native code on every run. This is a way different kind of JIT than Java HotSpot, which executes intermediate code (bytecode) initially, profiles intermediate code execution, and only compiles the hot spots to native code after a while. It seems like what "JIT" means has evolved since the time it meant "JVM HotSpot". There's a very interesting snippet of an interview with Anders Hejlsberg at https://stackoverflow.com/a/1255828/4158187 https://stackoverflow.com/a/1255828/4158187, which exposes his reasoning.
- mnembrini 9y agoA nice trick for debugging code generated at runtime is temporarily adding a CC instruction at the beginning of your assembly function, so you don't have to care where in memory your snippet is going to end up