6 ms·

I would be very curious to side by side performance benchmarks between this, GraalVM, and vanilla JDK. My gut tells me (with no data to back this us) that the v

I would be very curious to side by side performance benchmarks between this, GraalVM, and vanilla JDK. My gut tells me (with no data to back this us) that the vanilla JVM will inch ahead once it's paid the cost of starting up but I would be interested to see how wrong I am.

twic 3y ago
This JVM has no JIT, so OpenJDK and Graal will rather more than inch past it. It's not even a given that it will start up faster, since i don't believe it has been optimised nearly as hard.
- pg_1234 3y ago
  You'd probably get better results writing a Go compiler in Java ...
  - cfiggers 3y ago
    Wrote a Go compiler in Java, used it to compile this JVM implementation written in Go
    - seanw444 3y ago
      Ouroboros
      - KRAKRISMOTT 3y ago
        Lambda the ultimate
    - rizky05 3y ago
      [dead]
    - paulddraper 3y ago
      Great, now just use this JVM to run your compiler :)
  - adra 3y ago
    Well, you'd probably get pretty good perf with graalvm truffle, but certainly the building blocks of golang shouldn't be that problematic to implement in java given that the go language model is more straight-forward IMHO. I wouldn't assume it'd beat pure golang compilation but who knows.
jsiepkes 3y ago
It looks like a cool research project. But realistically I would be surprised if it could beat OpenJDK in any benchmark (though I don't think that's the purpose of the project). Writing something which is compatible with a specification is one thing, making it performant is another thing. For example it's not that hard to create a webserver from scratch but making it performant takes quite some effort.
layer8 3y ago
The startup cost is dominated by class loading, which Jacobin will have to do more or less the same as OpenJDK [*]. GraalVM of course profits from AOT compilation. [*] until the latter will implement condensers: https://openjdk.org/projects/leyden/notes/03-toward-condensers https://openjdk.org/projects/leyden/notes/03-toward-condense...
- ptx 3y ago
  Didn't they introduce some new format with Java 9 and jlink to store modules more efficiently? Why hasn't that fixed class loading performance?
  - papercrane 3y ago
    The modules system is for better encapsulation and the ability to produce a JVM without features you don't need. Faster start up time is in the works with projects like coordinated restore at checkpoint which will let you start a JVM up in an already "warmed" state. https://github.com/CRaC/docs https://github.com/CRaC/docs
    - ptx 3y ago
      If I understand the README correctly, CRaC in an abstract API that delegates to an implementation (CRIU) which is Linux-specific, described in its README as "the never-ending story, because we have to always keep up with the Linux kernel". So this wouldn't help if I'm deploying a GUI app for Windows, for example.
      - papercrane 3y ago
        Correct, it might, someday, become part of Java SE. If that ever happens I would expect it working on every platform Java supports would be a prerequisite for that. Right now it's primarily useful for reducing start up times in AWS Lambdas and auto-scaling workloads.
  - layer8 3y ago
    The issue isn’t the format. Class loading includes bytecode verification (verifying the type soundness of whatever code the class files contain), and then executing the initialization code defined by each class (static code blocks etc.). The optimizations in JDK 9 are for reducing the file size of the JDK (in terms of number of classes) when distributed with a standalone application, but that by itself doesn’t significantly affect startup time I believe, because classes are lazy-loaded only as required in any case.
tgv 3y ago
I see it more as an niche opportunity to integrate bits of Java code in Go applications. The start-up costs can be "paid" when the app starts. Performance will be bad, but it might save some rewrites (provided they can get things like db connections going).
- tombert 3y ago
  Yeah that makes some sense; there's probably millions of lines of Java code out there where performance doesn't really matter, and so aren't worth rewriting. Being able to embed a JVM just to run those from Go projects isn't a bad idea. Tangential, but a long time ago, I wanted to reuse a node.js slug library in .NET. I thought about trying to port it at first, but then I realized that this actual job didn't need to be terribly fast, so I instead embedded the Jurassic JS library into my code, and was able to load in the slug library that way directly). It wasn't especially fast (but actually faster than I thought it would be!), but it was certainly fast enough, and I didn't have to worry about not having feature parity.
  - kaba0 3y ago
    Why not just do traditional IPC, or even a batch process starting up a JVM from time to time when needed?
    - tgv 3y ago
      There are always other solutions. This one might just work in some cases. As I wrote: it's (probably) niche.
  - int_19h 3y ago
    Another good example was IKVM, which basically transpiled Java bytecode into .NET IL. Saxon XSLT used to build on top of that to provide .NET libraries from the same codebase as the original Java version. (Saxon had since moved to transpiling the source code instead - the languages are close enough that this is fairly straightforward.)
- paulddraper 3y ago
suresk 3y ago
I'm not the author, but I have contributed a few things to the project over the past year. The performance isn't anywhere close to the vanilla JVM - several times slower at best. It is entirely interpreted and there aren't any of the optimizations that have made their way into the JVM over the decades it has been around. It has been a fun project to play around with for someone like me who thinks this kind of stuff is fun and interesting but will probably never get a chance to work on it full time. Cool to see it noticed, though!
jillesvangurp 3y ago
Clearly this is not going to be a high performance thing. I'd be surprised if it is even close. It seems the goals for this are purely academic and about figuring out how Java works. They'll probably just do a simple interpreter and not a JIT compiler. That would be good enough for a POC. Additionally, they already indicated that they'll use Go's garbage collector, which won't be setting speed records with this either. And Java's typical usage of memory might actually stress it out a bit. Then there is the standard library which is going to need plenty of support for things like Threads, IO, various synchronization primitives and locks, etc. Doing that in Go is going to be a bit interesting but probably doable. Alternatively, they might just interface with native code directly and bypass the Go ecosystem. They might even reuse some things from openjdk for that. Speaking of which, native code and JNI would need to be implemented anyway.
andrewbinstock 3y ago
Lead dev here. We've run a couple of benchmarks internally just for kicks. To create a fair comparison, you have to run the Hotspot JVM with the -Xint flag, which says interpret only. Right now our performance is anywhere from 15-25% of the speed of Hotspot with -Xint on small benchmarks. We figure that the use of go alone creates some important portion of that overhead when compared with the C++ of the Hotspot JVM. We're guessing that a well-optimized Jacobin interpreter will eventually get to 50-60% of the Hotspot's -Xint speed. But we first want to get feature parity, before pivoting to performance. When we have feature parity, we'll run the Computer Language Benchmarks and post the results. That'll be fun to see!
- tombert 3y ago
  That's fair; I thought maybe the use of the Go GC might make the results a bit more interesting.
  - kaba0 3y ago
    Go has a much more primitive GC, so I wouldn’t expect a positive result from that itself.
- kaba0 3y ago
  Do you (can you even have in go?) have a direct threaded interpreter? That alone may give a 2x performance difference last time I checked, or is that no longer the case?
  - wahern 3y ago
    I don't think you can even implement indirect threading in Go. Go supports neither computed goto nor mutually recursive tail-call optimization (only self-recursive). Jump tables for switch statements was only implemented last year. If you squint that's close to indirect threading, but still with at least one unnecessary conditional per op. For the curious, here's their giant switch: https://github.com/platypusguy/jacobin/blob/c508ec50f55ef38183f395481ebb0bbdd053e1bc/src/jvm/run.go#L139 https://github.com/platypusguy/jacobin/blob/c508ec50f55ef381... In practice compilers have always been finicky when it comes to coaxing them to emit jump tables from switch statements, and I bet this is especially truly for Go.

twic 3y ago

This JVM has no JIT, so OpenJDK and Graal will rather more than inch past it. It's not even a given that it will start up faster, since i don't believe it has been optimised nearly as hard.

pg_1234 3y ago
You'd probably get better results writing a Go compiler in Java ...
- cfiggers 3y ago
  Wrote a Go compiler in Java, used it to compile this JVM implementation written in Go
  - seanw444 3y ago
    Ouroboros
    - KRAKRISMOTT 3y ago
      Lambda the ultimate
  - rizky05 3y ago
    [dead]
  - paulddraper 3y ago
    Great, now just use this JVM to run your compiler :)
- adra 3y ago
  Well, you'd probably get pretty good perf with graalvm truffle, but certainly the building blocks of golang shouldn't be that problematic to implement in java given that the go language model is more straight-forward IMHO. I wouldn't assume it'd beat pure golang compilation but who knows.

jsiepkes 3y ago

It looks like a cool research project. But realistically I would be surprised if it could beat OpenJDK in any benchmark (though I don't think that's the purpose of the project). Writing something which is compatible with a specification is one thing, making it performant is another thing. For example it's not that hard to create a webserver from scratch but making it performant takes quite some effort.

layer8 3y ago

The startup cost is dominated by class loading, which Jacobin will have to do more or less the same as OpenJDK [*]. GraalVM of course profits from AOT compilation. [*] until the latter will implement condensers: https://openjdk.org/projects/leyden/notes/03-toward-condensers https://openjdk.org/projects/leyden/notes/03-toward-condense...

ptx 3y ago
Didn't they introduce some new format with Java 9 and jlink to store modules more efficiently? Why hasn't that fixed class loading performance?
- papercrane 3y ago
  The modules system is for better encapsulation and the ability to produce a JVM without features you don't need. Faster start up time is in the works with projects like coordinated restore at checkpoint which will let you start a JVM up in an already "warmed" state. https://github.com/CRaC/docs https://github.com/CRaC/docs
  - ptx 3y ago
    If I understand the README correctly, CRaC in an abstract API that delegates to an implementation (CRIU) which is Linux-specific, described in its README as "the never-ending story, because we have to always keep up with the Linux kernel". So this wouldn't help if I'm deploying a GUI app for Windows, for example.
    - papercrane 3y ago
      Correct, it might, someday, become part of Java SE. If that ever happens I would expect it working on every platform Java supports would be a prerequisite for that. Right now it's primarily useful for reducing start up times in AWS Lambdas and auto-scaling workloads.
- layer8 3y ago
  The issue isn’t the format. Class loading includes bytecode verification (verifying the type soundness of whatever code the class files contain), and then executing the initialization code defined by each class (static code blocks etc.). The optimizations in JDK 9 are for reducing the file size of the JDK (in terms of number of classes) when distributed with a standalone application, but that by itself doesn’t significantly affect startup time I believe, because classes are lazy-loaded only as required in any case.

tgv 3y ago

I see it more as an niche opportunity to integrate bits of Java code in Go applications. The start-up costs can be "paid" when the app starts. Performance will be bad, but it might save some rewrites (provided they can get things like db connections going).

tombert 3y ago
Yeah that makes some sense; there's probably millions of lines of Java code out there where performance doesn't really matter, and so aren't worth rewriting. Being able to embed a JVM just to run those from Go projects isn't a bad idea. Tangential, but a long time ago, I wanted to reuse a node.js slug library in .NET. I thought about trying to port it at first, but then I realized that this actual job didn't need to be terribly fast, so I instead embedded the Jurassic JS library into my code, and was able to load in the slug library that way directly). It wasn't especially fast (but actually faster than I thought it would be!), but it was certainly fast enough, and I didn't have to worry about not having feature parity.
- kaba0 3y ago
  Why not just do traditional IPC, or even a batch process starting up a JVM from time to time when needed?
  - tgv 3y ago
    There are always other solutions. This one might just work in some cases. As I wrote: it's (probably) niche.
- int_19h 3y ago
  Another good example was IKVM, which basically transpiled Java bytecode into .NET IL. Saxon XSLT used to build on top of that to provide .NET libraries from the same codebase as the original Java version. (Saxon had since moved to transpiling the source code instead - the languages are close enough that this is fairly straightforward.)
paulddraper 3y ago

suresk 3y ago

I'm not the author, but I have contributed a few things to the project over the past year. The performance isn't anywhere close to the vanilla JVM - several times slower at best. It is entirely interpreted and there aren't any of the optimizations that have made their way into the JVM over the decades it has been around. It has been a fun project to play around with for someone like me who thinks this kind of stuff is fun and interesting but will probably never get a chance to work on it full time. Cool to see it noticed, though!

jillesvangurp 3y ago

Clearly this is not going to be a high performance thing. I'd be surprised if it is even close. It seems the goals for this are purely academic and about figuring out how Java works. They'll probably just do a simple interpreter and not a JIT compiler. That would be good enough for a POC. Additionally, they already indicated that they'll use Go's garbage collector, which won't be setting speed records with this either. And Java's typical usage of memory might actually stress it out a bit. Then there is the standard library which is going to need plenty of support for things like Threads, IO, various synchronization primitives and locks, etc. Doing that in Go is going to be a bit interesting but probably doable. Alternatively, they might just interface with native code directly and bypass the Go ecosystem. They might even reuse some things from openjdk for that. Speaking of which, native code and JNI would need to be implemented anyway.

andrewbinstock 3y ago

Lead dev here. We've run a couple of benchmarks internally just for kicks. To create a fair comparison, you have to run the Hotspot JVM with the -Xint flag, which says interpret only. Right now our performance is anywhere from 15-25% of the speed of Hotspot with -Xint on small benchmarks. We figure that the use of go alone creates some important portion of that overhead when compared with the C++ of the Hotspot JVM. We're guessing that a well-optimized Jacobin interpreter will eventually get to 50-60% of the Hotspot's -Xint speed. But we first want to get feature parity, before pivoting to performance. When we have feature parity, we'll run the Computer Language Benchmarks and post the results. That'll be fun to see!

tombert 3y ago
That's fair; I thought maybe the use of the Go GC might make the results a bit more interesting.
- kaba0 3y ago
  Go has a much more primitive GC, so I wouldn’t expect a positive result from that itself.
kaba0 3y ago
Do you (can you even have in go?) have a direct threaded interpreter? That alone may give a 2x performance difference last time I checked, or is that no longer the case?
- wahern 3y ago
  I don't think you can even implement indirect threading in Go. Go supports neither computed goto nor mutually recursive tail-call optimization (only self-recursive). Jump tables for switch statements was only implemented last year. If you squint that's close to indirect threading, but still with at least one unnecessary conditional per op. For the curious, here's their giant switch: https://github.com/platypusguy/jacobin/blob/c508ec50f55ef38183f395481ebb0bbdd053e1bc/src/jvm/run.go#L139 https://github.com/platypusguy/jacobin/blob/c508ec50f55ef381... In practice compilers have always been finicky when it comes to coaxing them to emit jump tables from switch statements, and I bet this is especially truly for Go.