8 ms·

LLVM based JavaScript JIT that beats V8 on Asm.js benchmarks

122 points by officialjunk 12y ago

officialjunk 12y ago
Performance benchmarks: http://arewefastyet.com/#machine=12&view=breakdown&suite=asmjs-apps http://arewefastyet.com/#machine=12&view=breakdown&suite=asm...
- joosters 12y ago
  All the graphs are missing the keys to identify which coloured line represents which machine/setup. And the y-axes aren't labelled, so you can't tell whether higher is better or not :(
  - sanxiyn 12y ago
    Quoting from "About" link: "Some benchmarks use time (lower is better), and some use points (higher is better). We orient all graphs so lower is better, visually."
  - radiospiel 12y ago
    color coding is in the left column, at least .
  - sp332 12y ago
    If you hover over a point, it tells you which source it came from. Not sure how to do that on mobile though.
sanxiyn 12y ago
Note that while it beats V8, it does not beat SpiderMonkey.
zurn 12y ago
The web page doesn't say anything about asm.js - so this is a generic JS JIT that happens to beat V8 in asm.js? V8 doesn't explicitly support asm.js so it's easier to beat than Firefox.
- sanxiyn 12y ago
  Yes, it is generic.
  - cliffbean 12y ago
    It is generic, but the graphs on arewefastyet.com suggest that it isn't actually making most regular benchmarks go faster, just some of the asm.js-style ones.
    - azakai 12y ago
      It makes one or two of the Octane benchmarks faster, like Mandreel. But Mandreel is quasi-asm.js as well. A powerful JIT like LLVM will tend to do best on large long-running benchmarks, like asm.js.
    - indolering 12y ago
      Well, that would make sense as asm.js is just a formalization of what the optimizing compilers were doing with LLVM transpiled JS code: http://mozakai.blogspot.com/2013/06/what-asmjs-is-and-what-asmjs-isnt.html http://mozakai.blogspot.com/2013/06/what-asmjs-is-and-what-a...
thinxer 12y ago
Does "FTL" stands for "faster than light"?
- masklinn 12y ago
  Er… first phrase of the page: > JavaScriptCore uses LLVM for a top-tier high-throughput optimizing JIT, which we call the FTL (Fourth Tier LLVM).
  - coldtea 12y ago
    Sure, they write that. But why did the chose the acronym. Sometimes you pick them to convey two messages.
  - aidenn0 12y ago
    And I think the double meaning is about as transparent as "Simultaneous Peripheral Operation OnLine"
- sanxiyn 12y ago
  Indeed that's an obvious allusion, and actually is explicitly mentioned here: https://bugs.webkit.org/show_bug.cgi?id=112840 https://bugs.webkit.org/show_bug.cgi?id=112840
notthetup 12y ago
Wait.. I thought V8 doesn't support asm.js? Or are they comparing it against a fork of V8 which does support asm.js.. Because if they're saying the new webkit jsc (using asm.js) is faster than V8 (not using using asm.js), then it's not really a fair comparison.
- sanxiyn 12y ago
  Of course V8 supports asm.js. asm.js is just JavaScript. Okay, I know what you mean, but still why wouldn't it be a fair comparison? V8 is free to "support" asm.js, whatever "support" means.
  - coldtea 12y ago
    How about "it's not a comparison that matters"? Nobody will see any benefits from JSCore being "faster than V8 on asm.js". In fact, if JSC is slower on non asm.js code, the comparison would also be extremely misleading too, because 99.9999% of web code is NOT asm.js. At the moment it's like saying "JSC is faster than v8 on a synthetic benchmark almost nobody really uses in the actual world, and v8 hasn't bothered optimizing for".
    - sanxiyn 12y ago
      Obviously, people using asm.js will benefit from JavaScriptCore being faster than V8 on asm.js. Whether it matters is an interesting question, but V8 developers seem to disagree with you, because they included asm.js benchmark in Octane benchmark suite, which V8 developers use to tune V8.
      - coldtea 12y ago
        >Obviously, people using asm.js will benefit from JavaScriptCore being faster than V8 on asm.js Also obviously I covered that, when I wrote "99.9999% of web code is NOT asm.js". So who are these "people"? All 10 of them? >Whether it matters is an interesting question, but V8 developers seem to disagree with you, because they included asm.js benchmark in Octane benchmark suite, which V8 developers use to tune V8. If anything, it shows that they agree with me. See how I compared asm.js to a "synthetic benchmark" in my comment?
mantrax5 12y ago
To have a meaningful "victory", you want to have one of those: 1. A generic JS JIT that runs generic JS faster than another generic JS JIT. 2. ASM.JS JIT that runs ASM.JS faster than another ASM.JS JIT. What we have here is something else: 3. A generic JS JIT that runs ASM.JS faster than another generic JS JIT. If people are not familiar with ASM.JS, the somewhat misleading (to the uninitiated) headline sounds more interesting than it is actually. It's not a significant milestone, as SpiderMonkey's ASM.JS implementation outperforms this one, and V8 isn't implementing ASM.JS in any way. As in at all. So of course it'll be slower. The purpose of ASM.JS is to allow a separate ASM.JS runtime to run ASM.JS code. The reason it happens to be a JS subset, is so you can have a fallback for backwards compatibility. Not to say this LLVM engine doesn't show potential. But implementing ASM.JS in LLVM is not the hard part of implementing a generic JS JIT. ASM.JS is entirely static, so it's comparatively trivial to JIT, it's literally an ASM to ASM transpiler.
- flohofwoe 12y ago
  IIRC, V8 had implemented optimizations that specifically target asm.js code patterns in the past (the performance gap for asm.js code between FF and Chrome definitely got a lot smaller in 2013), they just don't rely on the "use asm" keyword or do AOT compilation like Odinmonkey.
  - lucian1900 12y ago
    It was more fixing bugs in their JIT that were only really exercised by asm.js-like code.
- sanxiyn 12y ago
  Edit: the particular quote I objected to was edited out by parent. I object to "all the dynamicness of the rest of the JS is what V8 excels at". V8 excels at a subset of dynamic JavaScript (namely those benefit from Crankshaft) and is not very good outside of the subset. Typescript benchmark included in Octane benchmark suite is a good example. In Google's own words: "This test measures the time TypeScript takes to compile itself and is a proxy of how well a VM handles complex and sizable Javascript applications". In particular, Crankshaft is not effective for this benchmark. (Check yourself by running it with Crankshaft disabled: the score actually improves because you save compilation effort.) And as you can check yourself or check on AreWeFastYet, JavaScriptCore beats V8 by 2x on this benchmark. I admit Crankshaft subset of JavaScript is large and interesting (certainly compared to asm.js subset) but it is not true that V8 excels at dynamic JavaScript, for sufficiently large value of dynamic. JavaScriptCore does.
marijn 12y ago
Does anyone have numbers handy on the amount of size linking LLVM would add to a browser? Installing llvm-3.2-dev gives me over 75mb of static libraries, but I don't know how many of those would need to be shipped to support such a JIT.
- joakleaf 12y ago
  Did you try to strip the libraries for debug symbols?
  - sanxiyn 12y ago
    Assuming llvm-3.2-dev refers to Ubuntu package of the name, it is already stripped of debug symbols.
- flohofwoe 12y ago
  It may be that the LLVM core parts are already included in the OSX distribution, at least it was used a while ago for dynamic code generation in OpenGL (http://lists.cs.uiuc.edu/pipermail/llvmdev/2006-August/006492.html http://lists.cs.uiuc.edu/pipermail/llvmdev/2006-August/00649...), pure speculation on my side though.
stuaxo 12y ago
Now feed the output of that into enscripten ...

122 points by officialjunk 12y ago

Performance benchmarks: http://arewefastyet.com/#machine=12&view=breakdown&suite=asmjs-apps http://arewefastyet.com/#machine=12&view=breakdown&suite=asm...

joosters 12y ago
All the graphs are missing the keys to identify which coloured line represents which machine/setup. And the y-axes aren't labelled, so you can't tell whether higher is better or not :(
- sanxiyn 12y ago
  Quoting from "About" link: "Some benchmarks use time (lower is better), and some use points (higher is better). We orient all graphs so lower is better, visually."
- radiospiel 12y ago
  color coding is in the left column, at least .
- sp332 12y ago
  If you hover over a point, it tells you which source it came from. Not sure how to do that on mobile though.

sanxiyn 12y ago

Note that while it beats V8, it does not beat SpiderMonkey.

zurn 12y ago

The web page doesn't say anything about asm.js - so this is a generic JS JIT that happens to beat V8 in asm.js? V8 doesn't explicitly support asm.js so it's easier to beat than Firefox.

sanxiyn 12y ago
Yes, it is generic.
- cliffbean 12y ago
  It is generic, but the graphs on arewefastyet.com suggest that it isn't actually making most regular benchmarks go faster, just some of the asm.js-style ones.
  - azakai 12y ago
    It makes one or two of the Octane benchmarks faster, like Mandreel. But Mandreel is quasi-asm.js as well. A powerful JIT like LLVM will tend to do best on large long-running benchmarks, like asm.js.
  - indolering 12y ago
    Well, that would make sense as asm.js is just a formalization of what the optimizing compilers were doing with LLVM transpiled JS code: http://mozakai.blogspot.com/2013/06/what-asmjs-is-and-what-asmjs-isnt.html http://mozakai.blogspot.com/2013/06/what-asmjs-is-and-what-a...

thinxer 12y ago

Does "FTL" stands for "faster than light"?

masklinn 12y ago
Er… first phrase of the page: > JavaScriptCore uses LLVM for a top-tier high-throughput optimizing JIT, which we call the FTL (Fourth Tier LLVM).
- coldtea 12y ago
  Sure, they write that. But why did the chose the acronym. Sometimes you pick them to convey two messages.
- aidenn0 12y ago
  And I think the double meaning is about as transparent as "Simultaneous Peripheral Operation OnLine"
sanxiyn 12y ago
Indeed that's an obvious allusion, and actually is explicitly mentioned here: https://bugs.webkit.org/show_bug.cgi?id=112840 https://bugs.webkit.org/show_bug.cgi?id=112840

notthetup 12y ago

Wait.. I thought V8 doesn't support asm.js? Or are they comparing it against a fork of V8 which does support asm.js.. Because if they're saying the new webkit jsc (using asm.js) is faster than V8 (not using using asm.js), then it's not really a fair comparison.

sanxiyn 12y ago
Of course V8 supports asm.js. asm.js is just JavaScript. Okay, I know what you mean, but still why wouldn't it be a fair comparison? V8 is free to "support" asm.js, whatever "support" means.
- coldtea 12y ago
  How about "it's not a comparison that matters"? Nobody will see any benefits from JSCore being "faster than V8 on asm.js". In fact, if JSC is slower on non asm.js code, the comparison would also be extremely misleading too, because 99.9999% of web code is NOT asm.js. At the moment it's like saying "JSC is faster than v8 on a synthetic benchmark almost nobody really uses in the actual world, and v8 hasn't bothered optimizing for".
  - sanxiyn 12y ago
    Obviously, people using asm.js will benefit from JavaScriptCore being faster than V8 on asm.js. Whether it matters is an interesting question, but V8 developers seem to disagree with you, because they included asm.js benchmark in Octane benchmark suite, which V8 developers use to tune V8.
    - coldtea 12y ago
      >Obviously, people using asm.js will benefit from JavaScriptCore being faster than V8 on asm.js Also obviously I covered that, when I wrote "99.9999% of web code is NOT asm.js". So who are these "people"? All 10 of them? >Whether it matters is an interesting question, but V8 developers seem to disagree with you, because they included asm.js benchmark in Octane benchmark suite, which V8 developers use to tune V8. If anything, it shows that they agree with me. See how I compared asm.js to a "synthetic benchmark" in my comment?

mantrax5 12y ago

To have a meaningful "victory", you want to have one of those: 1. A generic JS JIT that runs generic JS faster than another generic JS JIT. 2. ASM.JS JIT that runs ASM.JS faster than another ASM.JS JIT. What we have here is something else: 3. A generic JS JIT that runs ASM.JS faster than another generic JS JIT. If people are not familiar with ASM.JS, the somewhat misleading (to the uninitiated) headline sounds more interesting than it is actually. It's not a significant milestone, as SpiderMonkey's ASM.JS implementation outperforms this one, and V8 isn't implementing ASM.JS in any way. As in at all. So of course it'll be slower. The purpose of ASM.JS is to allow a separate ASM.JS runtime to run ASM.JS code. The reason it happens to be a JS subset, is so you can have a fallback for backwards compatibility. Not to say this LLVM engine doesn't show potential. But implementing ASM.JS in LLVM is not the hard part of implementing a generic JS JIT. ASM.JS is entirely static, so it's comparatively trivial to JIT, it's literally an ASM to ASM transpiler.

flohofwoe 12y ago
IIRC, V8 had implemented optimizations that specifically target asm.js code patterns in the past (the performance gap for asm.js code between FF and Chrome definitely got a lot smaller in 2013), they just don't rely on the "use asm" keyword or do AOT compilation like Odinmonkey.
- lucian1900 12y ago
  It was more fixing bugs in their JIT that were only really exercised by asm.js-like code.
sanxiyn 12y ago
Edit: the particular quote I objected to was edited out by parent. I object to "all the dynamicness of the rest of the JS is what V8 excels at". V8 excels at a subset of dynamic JavaScript (namely those benefit from Crankshaft) and is not very good outside of the subset. Typescript benchmark included in Octane benchmark suite is a good example. In Google's own words: "This test measures the time TypeScript takes to compile itself and is a proxy of how well a VM handles complex and sizable Javascript applications". In particular, Crankshaft is not effective for this benchmark. (Check yourself by running it with Crankshaft disabled: the score actually improves because you save compilation effort.) And as you can check yourself or check on AreWeFastYet, JavaScriptCore beats V8 by 2x on this benchmark. I admit Crankshaft subset of JavaScript is large and interesting (certainly compared to asm.js subset) but it is not true that V8 excels at dynamic JavaScript, for sufficiently large value of dynamic. JavaScriptCore does.

marijn 12y ago

Does anyone have numbers handy on the amount of size linking LLVM would add to a browser? Installing llvm-3.2-dev gives me over 75mb of static libraries, but I don't know how many of those would need to be shipped to support such a JIT.

joakleaf 12y ago
Did you try to strip the libraries for debug symbols?
- sanxiyn 12y ago
  Assuming llvm-3.2-dev refers to Ubuntu package of the name, it is already stripped of debug symbols.
flohofwoe 12y ago
It may be that the LLVM core parts are already included in the OSX distribution, at least it was used a while ago for dynamic code generation in OpenGL (http://lists.cs.uiuc.edu/pipermail/llvmdev/2006-August/006492.html http://lists.cs.uiuc.edu/pipermail/llvmdev/2006-August/00649...), pure speculation on my side though.

stuaxo 12y ago

Now feed the output of that into enscripten ...