3 ms·

This is a serious question. Can someone explain to me why higher-level languages like Python, Ruby, and JavaScript can't share a fast VM? It doesn't have to be

This is a serious question. Can someone explain to me why higher-level languages like Python, Ruby, and JavaScript can't share a fast VM? It doesn't have to be the JVM or .NET, but why is everybody duplicating so much work? I know the Perl guys were moving in this direction, but nobody else seems to be onboard. Even within Google, they have v8 for JavaScript already.

llimllib 17y ago
why ask this question on an article about python possibly moving to LLVM? Rubinius and MacRuby target LLVM as well, for example.
- ori_b 17y ago
  LLVM isn't so much a VM for those languages to use directly as a portable assembly toolkit. It's too low level to do compile to directly. It'd occupy a place in the stack somewhere near assembly, but with an optimizer in place. It's more of a VM toolkit than a VM itself.
briansmith 17y ago
Why not just write a program that can convert any program written in one language to any of the others? It is basically the same problem and about the same level of difficulty.
- johnnybgoode 17y ago
  Edit: Were you being sarcastic? I don't think it's exactly the same, because being able to compile many languages down to a single language is easier than also having to go in the other direction. Original comment: Yeah, that would be one solution, but I think the shared VM helps so you can easily access libraries written in different languages. I remember why (of Ruby fame) trying to run Ruby code on Google App Engine back when it was Python-only. He discovered that Ruby bytecode is nearly identical to Python bytecode. Like me, he didn't understand why Python and Ruby still need to be on separate runtimes.
  - karanbhangui 17y ago
    That wasn't sarcasm; it was a rhetorical question to illustrate a point.
    - johnnybgoode 17y ago
      Thank you for adding so much to the discussion. (That was sarcasm.)
  - briansmith 17y ago
    No, I wasn't being sarcastic. If you want to use libraries written in Python in Ruby (just as an example), you need to be able to convert all the Pythonisms to Rubyisms and/or Rubyisms to Pythonisms. That is the case whether you are doing source-to-source rewriting or using a shared VM. Note that the JVM and .NET runtimes do pretty much exactly what was asked--they run programs written in multiple languages on the same VM. Because the "native" VMs for Python and Ruby are so bad, the JVM and .NET VMs can even out-perform the native ones for most tasks.
- ori_b 17y ago
  How much of the source level semantics do you want to preserve? If you don't care about many of them then it's not too hard to do (although quite pointless). Just compile to an array of bytes, and a turing machine emulator or equivalent, in your target language of choice, and you've converted. The hard part comes when you want to preserve source-level semantics and provide a natural-feeling translation so that a human can understand, read, and write the source. However, with a VM you don't have that problem. With a VM, a lossy, one-way transform is sufficient. You can easily throw away information as you compile, since you're not targeting humans consumption for the output. You can replace, say, overloaded functions with mangled names. You can turn multiple dispatch into decision trees of conditional statements. And so on. With a VM, you would want to keep enough high-level detail to efficiently JIT stuff, and it's a bit of a challenge to decide what detail is sufficiently useful to include, but it's not insurmountable.
ori_b 17y ago
Look up the "Parrot" VM. That seems to be their goal.
- johnnybgoode 17y ago
  I know the Perl guys were moving in this direction, but nobody else seems to be onboard. That's what I meant. As far as I know, nobody else is taking it seriously as a primary VM for their language.
  - granular 17y ago
    There are a number of Parrot languages at various stages of development. As soon as they mature a bit, you might find people taking them pretty seriously. :)
    - johnnybgoode 17y ago
      That would be great. But I'm wondering why the authors of other major dynamic languages haven't chosen to collaborate and use Parrot as their primary VM.
      - ori_b 17y ago
        Mostly, because they already have a VM that works.
        johnnybgoode 17y ago
        I guess, but many of them seem to want a more advanced VM. (http://news.ycombinator.com/item?id=656594 http://news.ycombinator.com/item?id=656594)
      - granular 17y ago
        Parrot 1.0.0 was released in March of this year. Give it time.
kingsley_20 17y ago
Isnt that the JVM? don't know about fast though, but fast enough for me.
etal 17y ago
It's usually one or more of: 1. Using a simple prototype VM seemed easier during initial development 2. When the language was developed, no suitable common VM existed 3. The language developers thought they could do it better for their specific case, i.e. the point of the implementation is to showcase new VM strategies Remember that a language and an interpreter are separate things, and usually an author who wants to try an idea for one has to find some suitable stand-in for the other in order to get started.
- johnnybgoode 17y ago
  Thanks. Your #1 and #2 seem to be very common. It looks like we're at a point now where many dynamic languages are attempting to transition to a more advanced VM. This includes Perl, Python, Ruby, and some implementations of JavaScript. I assume they all want to take advantage of the latest VM techniques, and they're all making major changes anyway, so again I just wonder why they don't seriously consider collaborating now.
  - etal 17y ago
    I think it's interesting how the success of projects like this sometimes seems to depend completely on a single person having the right combination of interest and skill. The Jython project was just the coolest darn thing until, from what I can tell, the lead developer got tired of maintaining a mature project with a hairy codebase and left to join (start?) PyPy, an even more ambitious project. So Jython was stuck in a coma with a pre-alpha 2.2 release for like half a decade, while CPython marched on, and no new maintainer was able to bring it back to consciousness -- despite huge demand for it, since Jython at the time was the perfect solution to problems in Java and Python. Finally Frank Wierzbicki came along a couple years ago, squeezed out the stubborn 2.2 release and is on the verge of version 2.5, at which point we can consider Jython back open for business. There's a kind of similar story with Psyco, and I think that one also concludes with PyPy being the "right answer" to the original problem. But back on topic -- in some ways, the dynamic-language communities are collaborating on common VMs. The JVM team is trying to make things easier for other languages with "invokedynamic", the .NET world is keeping up at least (DLR? dunno), and LLVM has good traction as a compilation target. However, I doubt we'll see anyone devote significant effort to porting Python or Ruby to Parrot until Perl 6 lands and provides a compelling demonstration. Meanwhile, PyPy is getting close enough to its goal that we might just want to wait and see if their JIT-compiler-generator solves all of these problems in one shot.
pronoiac 17y ago
I've heard there was an attempt to pursue this a few years ago, but there were very serious fundamental incompatibilities, with examples given that went way over my head.
- johnnybgoode 17y ago
  Interesting, do you remember anything else, like which languages were involved?
  - pronoiac 17y ago
    I associate it with the Perl Apocalypses. I think it predated Parrot, whose popularity makes this hard to search for, & the languages I'm surest of are Python & Perl, unfortunately. I think it took place at a conference, possibly OSCON.
tlrobinson 17y ago
V8 goes down to machine code, not bytecode, btw.

llimllib 17y ago

why ask this question on an article about python possibly moving to LLVM? Rubinius and MacRuby target LLVM as well, for example.

ori_b 17y ago
LLVM isn't so much a VM for those languages to use directly as a portable assembly toolkit. It's too low level to do compile to directly. It'd occupy a place in the stack somewhere near assembly, but with an optimizer in place. It's more of a VM toolkit than a VM itself.

briansmith 17y ago

Why not just write a program that can convert any program written in one language to any of the others? It is basically the same problem and about the same level of difficulty.

johnnybgoode 17y ago
Edit: Were you being sarcastic? I don't think it's exactly the same, because being able to compile many languages down to a single language is easier than also having to go in the other direction. Original comment: Yeah, that would be one solution, but I think the shared VM helps so you can easily access libraries written in different languages. I remember why (of Ruby fame) trying to run Ruby code on Google App Engine back when it was Python-only. He discovered that Ruby bytecode is nearly identical to Python bytecode. Like me, he didn't understand why Python and Ruby still need to be on separate runtimes.
- karanbhangui 17y ago
  That wasn't sarcasm; it was a rhetorical question to illustrate a point.
  - johnnybgoode 17y ago
    Thank you for adding so much to the discussion. (That was sarcasm.)
- briansmith 17y ago
  No, I wasn't being sarcastic. If you want to use libraries written in Python in Ruby (just as an example), you need to be able to convert all the Pythonisms to Rubyisms and/or Rubyisms to Pythonisms. That is the case whether you are doing source-to-source rewriting or using a shared VM. Note that the JVM and .NET runtimes do pretty much exactly what was asked--they run programs written in multiple languages on the same VM. Because the "native" VMs for Python and Ruby are so bad, the JVM and .NET VMs can even out-perform the native ones for most tasks.
ori_b 17y ago
How much of the source level semantics do you want to preserve? If you don't care about many of them then it's not too hard to do (although quite pointless). Just compile to an array of bytes, and a turing machine emulator or equivalent, in your target language of choice, and you've converted. The hard part comes when you want to preserve source-level semantics and provide a natural-feeling translation so that a human can understand, read, and write the source. However, with a VM you don't have that problem. With a VM, a lossy, one-way transform is sufficient. You can easily throw away information as you compile, since you're not targeting humans consumption for the output. You can replace, say, overloaded functions with mangled names. You can turn multiple dispatch into decision trees of conditional statements. And so on. With a VM, you would want to keep enough high-level detail to efficiently JIT stuff, and it's a bit of a challenge to decide what detail is sufficiently useful to include, but it's not insurmountable.

ori_b 17y ago

Look up the "Parrot" VM. That seems to be their goal.

johnnybgoode 17y ago
I know the Perl guys were moving in this direction, but nobody else seems to be onboard. That's what I meant. As far as I know, nobody else is taking it seriously as a primary VM for their language.
- granular 17y ago
  There are a number of Parrot languages at various stages of development. As soon as they mature a bit, you might find people taking them pretty seriously. :)
  - johnnybgoode 17y ago
    That would be great. But I'm wondering why the authors of other major dynamic languages haven't chosen to collaborate and use Parrot as their primary VM.
    - ori_b 17y ago
      Mostly, because they already have a VM that works.
      - johnnybgoode 17y ago
        I guess, but many of them seem to want a more advanced VM. (http://news.ycombinator.com/item?id=656594 http://news.ycombinator.com/item?id=656594)
    - granular 17y ago
      Parrot 1.0.0 was released in March of this year. Give it time.

kingsley_20 17y ago

Isnt that the JVM? don't know about fast though, but fast enough for me.

etal 17y ago

It's usually one or more of: 1. Using a simple prototype VM seemed easier during initial development 2. When the language was developed, no suitable common VM existed 3. The language developers thought they could do it better for their specific case, i.e. the point of the implementation is to showcase new VM strategies Remember that a language and an interpreter are separate things, and usually an author who wants to try an idea for one has to find some suitable stand-in for the other in order to get started.

johnnybgoode 17y ago
Thanks. Your #1 and #2 seem to be very common. It looks like we're at a point now where many dynamic languages are attempting to transition to a more advanced VM. This includes Perl, Python, Ruby, and some implementations of JavaScript. I assume they all want to take advantage of the latest VM techniques, and they're all making major changes anyway, so again I just wonder why they don't seriously consider collaborating now.
- etal 17y ago
  I think it's interesting how the success of projects like this sometimes seems to depend completely on a single person having the right combination of interest and skill. The Jython project was just the coolest darn thing until, from what I can tell, the lead developer got tired of maintaining a mature project with a hairy codebase and left to join (start?) PyPy, an even more ambitious project. So Jython was stuck in a coma with a pre-alpha 2.2 release for like half a decade, while CPython marched on, and no new maintainer was able to bring it back to consciousness -- despite huge demand for it, since Jython at the time was the perfect solution to problems in Java and Python. Finally Frank Wierzbicki came along a couple years ago, squeezed out the stubborn 2.2 release and is on the verge of version 2.5, at which point we can consider Jython back open for business. There's a kind of similar story with Psyco, and I think that one also concludes with PyPy being the "right answer" to the original problem. But back on topic -- in some ways, the dynamic-language communities are collaborating on common VMs. The JVM team is trying to make things easier for other languages with "invokedynamic", the .NET world is keeping up at least (DLR? dunno), and LLVM has good traction as a compilation target. However, I doubt we'll see anyone devote significant effort to porting Python or Ruby to Parrot until Perl 6 lands and provides a compelling demonstration. Meanwhile, PyPy is getting close enough to its goal that we might just want to wait and see if their JIT-compiler-generator solves all of these problems in one shot.

pronoiac 17y ago

I've heard there was an attempt to pursue this a few years ago, but there were very serious fundamental incompatibilities, with examples given that went way over my head.

johnnybgoode 17y ago
Interesting, do you remember anything else, like which languages were involved?
- pronoiac 17y ago
  I associate it with the Perl Apocalypses. I think it predated Parrot, whose popularity makes this hard to search for, & the languages I'm surest of are Python & Perl, unfortunately. I think it took place at a conference, possibly OSCON.

tlrobinson 17y ago

V8 goes down to machine code, not bytecode, btw.