4 ms·

I see. Sounds like an interesting project. Sure, MRI is boring, but it's the best implementation right now (though some may argue that JRuby is better), and it

I see. Sounds like an interesting project. Sure, MRI is boring, but it's the best implementation right now (though some may argue that JRuby is better), and it's in desperate need of VM innovations. Any new compiler/VM starting from scratch will be years away from being available for use in production environments. By the time it's finished, we'll all be using Go. (Sigh.)

vidarh 14y ago
I think we need both. MRI can keep getting better, as can JRuby (which is an amazing feat, but to me, running on top of the JVM makes it a non-starter) or Rubinius, but they're fundamentally side-stepping the really hard problems. E.g. nothing will stop MRI from having to interpret thousands of lines of code each time because it can't draw a line between runtime and compile time, while for an ahead of time compiler for Ruby, finding a pragmatic line between what needs to be executed at runtime vs. compile time is essential (consider for example the tendency to do stuff like getting the list of files in a directory and require all of them in turn).
- lobster_johnson 14y ago
  We do need both. But some of the things you mention (like constructing vtables) could be applied to MRI's VM model without writing a compiler from scratch. Frankly, I would much rather have large performance improvements now than in 2-3 years. (I agree about JRuby. I also wonder why Rubinius, which showed so much promise at the beginning, has stagnated. Is it simply the lack of developers?)
  - vidarh 14y ago
    I agree the performance increase would be great, but I think it needs to come gradually to MRI. E.g. trying to do anything fancy with the old AST-based interpreter would've been pretty pointless. After YARV, it is probably starting to get more attractive, but at the same time they've added method caching which gives a decent amount of the benefits. A vtable will still get faster, but it might not be the most immediately expedient way of speeding things up vs. e.g. Sasadas latest project of adding a generational gc. Regarding Rubinius, writing compilers for dynamic languages is hard. Most textbooks you'll find cover techniques most suitable for statically typed languages (the best resource I know for starting to catch up on compiling dynamic languages is actually the Self papers). So you need more than an unusual level of interest in writing compilers to be likely to try to tackle a language like Ruby which is tricky even for dynamic languages (e.g. my favorite pet problem to meditate on: What constitutes 'compile time' vs. 'runtime' for ahead of time compiled Ruby?), and even more to actually persevere until you start getting proper results where you can get decent results in days with a simpler language. It's made worse because of Ruby's horrendous grammar. And I mean that from a compiler writers perspective - as a developer I love to use Ruby to a large extent because the complexities of the grammar means it reads and writes better 95% of the time. But MRI's bison based parser was 6k-7k lines with ugly parser/lexer interplay last time I checked... There are full compilers substantially smaller than that for other languages... To me, these complexities are part of what makes it fascinating. I firmly believe you can parse Ruby fully with a much, much simpler parser for example. A lot of the ugliness can be abstracted away, and C parser code is rarely good examples of succint code. I did start playing with MRI years ago, specifically the parser, actually, and started chopping out redundant pieces, but got frustrated and bored with it. That's part of the problem - it's one thing to play around with a toy compiler like I've done, and another entirely to put in the effort to push a major change to MRI through to production quality given the number of years of accumulated history encapsulated in it. Doing the latter as a hobby is a daunting task.
    - lobster_johnson 14y ago
      Just a note on Rubinius: The PyPy guys seem to have done pretty well at this. I don't know how similar they are to Rubinius; PyPy reduces to a RPython as an initial step, whereas I believe Rubinius compiles to LLVM's LI.

vidarh 14y ago

I think we need both. MRI can keep getting better, as can JRuby (which is an amazing feat, but to me, running on top of the JVM makes it a non-starter) or Rubinius, but they're fundamentally side-stepping the really hard problems. E.g. nothing will stop MRI from having to interpret thousands of lines of code each time because it can't draw a line between runtime and compile time, while for an ahead of time compiler for Ruby, finding a pragmatic line between what needs to be executed at runtime vs. compile time is essential (consider for example the tendency to do stuff like getting the list of files in a directory and require all of them in turn).

lobster_johnson 14y ago
We do need both. But some of the things you mention (like constructing vtables) could be applied to MRI's VM model without writing a compiler from scratch. Frankly, I would much rather have large performance improvements now than in 2-3 years. (I agree about JRuby. I also wonder why Rubinius, which showed so much promise at the beginning, has stagnated. Is it simply the lack of developers?)
- vidarh 14y ago
  I agree the performance increase would be great, but I think it needs to come gradually to MRI. E.g. trying to do anything fancy with the old AST-based interpreter would've been pretty pointless. After YARV, it is probably starting to get more attractive, but at the same time they've added method caching which gives a decent amount of the benefits. A vtable will still get faster, but it might not be the most immediately expedient way of speeding things up vs. e.g. Sasadas latest project of adding a generational gc. Regarding Rubinius, writing compilers for dynamic languages is hard. Most textbooks you'll find cover techniques most suitable for statically typed languages (the best resource I know for starting to catch up on compiling dynamic languages is actually the Self papers). So you need more than an unusual level of interest in writing compilers to be likely to try to tackle a language like Ruby which is tricky even for dynamic languages (e.g. my favorite pet problem to meditate on: What constitutes 'compile time' vs. 'runtime' for ahead of time compiled Ruby?), and even more to actually persevere until you start getting proper results where you can get decent results in days with a simpler language. It's made worse because of Ruby's horrendous grammar. And I mean that from a compiler writers perspective - as a developer I love to use Ruby to a large extent because the complexities of the grammar means it reads and writes better 95% of the time. But MRI's bison based parser was 6k-7k lines with ugly parser/lexer interplay last time I checked... There are full compilers substantially smaller than that for other languages... To me, these complexities are part of what makes it fascinating. I firmly believe you can parse Ruby fully with a much, much simpler parser for example. A lot of the ugliness can be abstracted away, and C parser code is rarely good examples of succint code. I did start playing with MRI years ago, specifically the parser, actually, and started chopping out redundant pieces, but got frustrated and bored with it. That's part of the problem - it's one thing to play around with a toy compiler like I've done, and another entirely to put in the effort to push a major change to MRI through to production quality given the number of years of accumulated history encapsulated in it. Doing the latter as a hobby is a daunting task.
  - lobster_johnson 14y ago
    Just a note on Rubinius: The PyPy guys seem to have done pretty well at this. I don't know how similar they are to Rubinius; PyPy reduces to a RPython as an initial step, whereas I believe Rubinius compiles to LLVM's LI.