18 ms·

Massive: The asm.js Benchmark

109 points by rnyman 12y ago

itsbits 12y ago
would love to know how is the performance in V8 chrome js engine??
- acqq 12y ago
  Run it with your Chrome and post the results: http://kripken.github.io/Massive/ http://kripken.github.io/Massive/
  - fridek 12y ago
    I have Intel i7 running on Linux. Got 2,984 running Chrome 38 and 8,778 on Firefox 33, but I think now it's just fair to wait for couple of iterations of Chrome to identify bottlenecks and fix them. It's not uncommon for benchmarks to focus on some particular strength/weakness that is easily fixable. The same happened with Firefox and Octane 2.0.
    - TazeTSchnitzel 12y ago
      I'm not sure if Chrome is ever going to catch up to Firefox here. It's been slower than Firefox at asm.js ever since it was announced, because Chrome doesn't do AOT compilation for it. EDIT: Although I suppose JIT compilers would love asm.js as it has all the type info they want, but there's probably tons of overhead there thus explaining how Chrome has worse results.
      - pjmlp 12y ago
        I would say Google team has other priorities....
        TazeTSchnitzel 12y ago
        Yes, like trying to destroy the open web with NaCl.
        blibble 12y ago
        I don't really understand how asm.js is any different... you've still got applications compiled into binary blob, the difference between NaCl and asm.js being that asm.js just happens to be binary encoded as executable JavaScript.
        TazeTSchnitzel 12y ago
        asm.js isn't a binary blob, it's JavaScript. It's not a threat to a web of open standards.
        cromwellian 12y ago
        It's a blob transpiled from C, not exactly how people imagined the Web working a few years ago. It's a non-W3C/TC39 approved set of compiler intrinsics/pragmas that switches on special behavior in one particular environment. When Dart was introduced years ago, Brendan Eich was all over HN complaining about how Dart2JS doesn't solve the problem, because DartVM would be more performance, and introduce a tendency to optimize JS VMs towards those semantics. Well, here we have a different problem. If asmjs takes off and everyone starts writing Web apps by transpiling C code, Mozilla will have forced a sea change in the Web, forcing all other vendors into ahead-of-time compilation to compete, and forcing JS VMs to optimize for this particular kind of code. Now, maybe that's not such a bad thing. I'm not arguing it isn't, but I think arguing "it's just JS, the syntax is the same, therefore it's open!" is somewhat disingenuous. ASMJS, if super successful, would create an implicit implementation spec for everyone. It will have in essence, created a language with identical syntax to Javascript, but with completely different expected execution semantics. Why? Because developers would come to depend on ahead-of-time compilation, depend on apps that basically use emulated malloc'ed heaps, etc. Imagine I took the Java language, removed "new" and forced all code to allocate DirectBuffers, and then said, all classes must consist of static final methods (no polymorphism), and can't have any fields or constructors. Call this ASMJAVA. I now assert, this is "Just java", hey, it runs in any JVM. Except, I'm shipping a new VM called "SuperExpresso" that does ahead-of-time compilation for my mobile platform based on this, and I'm telling people they don't need to use native SDKs, they can just transpile C code to Java. Did I create a new standard? I think I just did. Arguably, you can say the same thing about Android 4.4/5.0 "ART" runtime, or GWT which I work on. They use the Java "language", but the runtime behavior is completely different, and at least for GWT, we make no bones about this. So, this is not an attack on asmjs, but I do think there is a distinction between "reusing the syntax" of a language, and creating expectations of new semantics. Languages carry with them implicit sets of runtime performance characteristics, even bugs, that developers tend to treat holistically. Not many people care about what the specs say. They care about how it runs in the real world.
      - azakai 12y ago
        Chrome is currently slower, but the v8 team is working on a new optimizing compiler called TurboFan. There are indications in the source code that it detects asm.js code as a compiler hint of some form, hopefully this means it will eventually get quite fast on that type of content.
  - TazeTSchnitzel 12y ago
    I got 2,996 in Firefox and 1,277 in Chrome (2013 MacBook Air 13"). Interestingly, Chrome had much worse performance for float32 than float64... yet Firefox had slightly better performance. I guess the Chrome team implemented Math.fround yet none of the optimisations which make it useful!
    - wldcordeiro 12y ago
      I got a 4,690 on Firefox Nightly so there's so big improvements coming down the pipeline.
      - TazeTSchnitzel 12y ago
        I got 3,115 in Firefox Nightly. That's a slight improvement over the 2,996 in normal Fx, but not that big, really.
- specialdragon 12y ago
  Their 'copy results' isn't terribly useful however: http://pastie.org/9692975 http://pastie.org/9692975 My results, Chrome Version 38.0.2125.111 (64-bit) on Ubuntu 14.04
amelius 12y ago
They forgot to include two things: * Performance on the native architecture * Performance on other browsers (Chrome/IE in particular)
higherpurpose 12y ago
I prefer asm.js over NaCl, especially when NaCl has been available for years, and Google still supports the ARM architecture poorly, and as a second class citizen in NaCl. It's unacceptable that a "browser-os" that should have no problem being architecture agnostic, still gives Intel an edge with ChromeOS, because not all NaCl apps work on ARM Chromebooks. Seriously, how crazy is that? I can understand ARM not having a serious chance on Windows laptops because of all the legacy x86 programs, but at worst it should be equal with x86 running an OS such as Chrome OS.
- TazeTSchnitzel 12y ago
  I agree. asm.js has several strengths over NaCl and even PNaCl: * It is just JavaScript, so any browser can execute asm.js code, but it can be AOT compiled by some browsers for extra performance - This means every browser supports asm.js, and supports it today * It doesn't require a new runtime or API, merely using the existing web APIs * It is simple for other browser vendors to implement, and isn't reliant on a single implementation * It can interface with existing JavaScript code, so it can use JS libraries and, similarly, JS code can use asm.js libraries * It is a completely open standard * It is architecture-independent (so's PNaCl, though)
  - masklinn 12y ago
    > It is just JavaScript, so any browser can execute asm.js code, but it can be AOT compiled by some browsers for extra performance Or JIT-compiled with phases specifically built for the kind of JS it produces e.g. Webkit's FTL tends to work very well on asm.js code, though it's not an asm.js AOT compiler. > It doesn't require a new runtime or API, merely using the existing web APIs That's not entirely true, AFAIK asm.js adds at least two functions to the Math namespace (imul and fround)
    - TazeTSchnitzel 12y ago
      > Or JIT-compiled with phases specifically built for the kind of JS it produces e.g. Webkit's FTL tends to work very well on asm.js code, though it's not an asm.js AOT compiler. This is true, too. While I think you'd get the best results from AOT compilation, it should also compile really well under JITs since it has all the type information you normally have to guess. > That's not entirely true, AFAIK asm.js adds at least two functions to the Math namespace (imul and fround) That's an addition to JavaScript which isn't specific to asm.js, however, and both can be polyfilled. Math.fround, in particular, is useful outside of asm.js. The stuff asm.js needs is all part of standard JS now, while NaCl and PNaCl require their own special API (Pepper) which only has one implementation (Google's).
StevePerkins 12y ago
I'm often dismissive of Mozilla, for being so all-over-the-map and unfocused sometimes. However, I'm grateful for them being the only browser player that really does seem interested in progressing JavaScript. All of the other players seem more focused on transpiling solutions, which are really about replacing JavaScript with something else under their control. Mozilla is pretty much the only player that genuinely works to improve JavaScript without another agenda.
- rictic 12y ago
  I don't think it's fair to characterize Traceur or TypeScript as plays to put Javascript under their author's control. Both have been useful for developing ES6, which Firefox has also been a leading implementer of. From my perspective almost all of the browser makers doing lots of cool stuff to push forward javascript and the web.
  - TazeTSchnitzel 12y ago
    You're assuming StevePerkins was referring to Tracuer or TypeScript, I don't think they were. I assume they meant things like NaCl and Dart.
    - cromwellian 12y ago
      Dart compiled to JS just like Emscripten.
      - TazeTSchnitzel 12y ago
        Technically it compiles to JS, yes, but the reason people have a problem with Dart is the idea of embedding a Dart VM in the browser. (And some features of the core language only work with an embedded Dart VM)
    - rictic 12y ago
      A bit of an assumption yes, here's my reasoning though. Traceur and Typescript (and perhaps Flow and AtScript) are similar efforts made by different players that require transpilation and are potential visions for a direction javascript could go in. This combined with the author's quote "All of the other players..." made me think that's the family of solutions they were referring to. NaCl and Dart are both by a single player and there's nothing else like them that I know about but Mozilla's asm.js, which the author is praising here, so they didn't seem like likely candidates.

109 points by rnyman 12y ago

itsbits 12y ago

would love to know how is the performance in V8 chrome js engine??

acqq 12y ago
Run it with your Chrome and post the results: http://kripken.github.io/Massive/ http://kripken.github.io/Massive/
- fridek 12y ago
  I have Intel i7 running on Linux. Got 2,984 running Chrome 38 and 8,778 on Firefox 33, but I think now it's just fair to wait for couple of iterations of Chrome to identify bottlenecks and fix them. It's not uncommon for benchmarks to focus on some particular strength/weakness that is easily fixable. The same happened with Firefox and Octane 2.0.
  - TazeTSchnitzel 12y ago
    I'm not sure if Chrome is ever going to catch up to Firefox here. It's been slower than Firefox at asm.js ever since it was announced, because Chrome doesn't do AOT compilation for it. EDIT: Although I suppose JIT compilers would love asm.js as it has all the type info they want, but there's probably tons of overhead there thus explaining how Chrome has worse results.
    - pjmlp 12y ago
      I would say Google team has other priorities....
      - TazeTSchnitzel 12y ago
        Yes, like trying to destroy the open web with NaCl.
        blibble 12y ago
        I don't really understand how asm.js is any different... you've still got applications compiled into binary blob, the difference between NaCl and asm.js being that asm.js just happens to be binary encoded as executable JavaScript.
        TazeTSchnitzel 12y ago
        asm.js isn't a binary blob, it's JavaScript. It's not a threat to a web of open standards.
        cromwellian 12y ago
        It's a blob transpiled from C, not exactly how people imagined the Web working a few years ago. It's a non-W3C/TC39 approved set of compiler intrinsics/pragmas that switches on special behavior in one particular environment. When Dart was introduced years ago, Brendan Eich was all over HN complaining about how Dart2JS doesn't solve the problem, because DartVM would be more performance, and introduce a tendency to optimize JS VMs towards those semantics. Well, here we have a different problem. If asmjs takes off and everyone starts writing Web apps by transpiling C code, Mozilla will have forced a sea change in the Web, forcing all other vendors into ahead-of-time compilation to compete, and forcing JS VMs to optimize for this particular kind of code. Now, maybe that's not such a bad thing. I'm not arguing it isn't, but I think arguing "it's just JS, the syntax is the same, therefore it's open!" is somewhat disingenuous. ASMJS, if super successful, would create an implicit implementation spec for everyone. It will have in essence, created a language with identical syntax to Javascript, but with completely different expected execution semantics. Why? Because developers would come to depend on ahead-of-time compilation, depend on apps that basically use emulated malloc'ed heaps, etc. Imagine I took the Java language, removed "new" and forced all code to allocate DirectBuffers, and then said, all classes must consist of static final methods (no polymorphism), and can't have any fields or constructors. Call this ASMJAVA. I now assert, this is "Just java", hey, it runs in any JVM. Except, I'm shipping a new VM called "SuperExpresso" that does ahead-of-time compilation for my mobile platform based on this, and I'm telling people they don't need to use native SDKs, they can just transpile C code to Java. Did I create a new standard? I think I just did. Arguably, you can say the same thing about Android 4.4/5.0 "ART" runtime, or GWT which I work on. They use the Java "language", but the runtime behavior is completely different, and at least for GWT, we make no bones about this. So, this is not an attack on asmjs, but I do think there is a distinction between "reusing the syntax" of a language, and creating expectations of new semantics. Languages carry with them implicit sets of runtime performance characteristics, even bugs, that developers tend to treat holistically. Not many people care about what the specs say. They care about how it runs in the real world.
    - azakai 12y ago
      Chrome is currently slower, but the v8 team is working on a new optimizing compiler called TurboFan. There are indications in the source code that it detects asm.js code as a compiler hint of some form, hopefully this means it will eventually get quite fast on that type of content.
- TazeTSchnitzel 12y ago
  I got 2,996 in Firefox and 1,277 in Chrome (2013 MacBook Air 13"). Interestingly, Chrome had much worse performance for float32 than float64... yet Firefox had slightly better performance. I guess the Chrome team implemented Math.fround yet none of the optimisations which make it useful!
  - wldcordeiro 12y ago
    I got a 4,690 on Firefox Nightly so there's so big improvements coming down the pipeline.
    - TazeTSchnitzel 12y ago
      I got 3,115 in Firefox Nightly. That's a slight improvement over the 2,996 in normal Fx, but not that big, really.
specialdragon 12y ago
Their 'copy results' isn't terribly useful however: http://pastie.org/9692975 http://pastie.org/9692975 My results, Chrome Version 38.0.2125.111 (64-bit) on Ubuntu 14.04

amelius 12y ago

They forgot to include two things: * Performance on the native architecture * Performance on other browsers (Chrome/IE in particular)

higherpurpose 12y ago

I prefer asm.js over NaCl, especially when NaCl has been available for years, and Google still supports the ARM architecture poorly, and as a second class citizen in NaCl. It's unacceptable that a "browser-os" that should have no problem being architecture agnostic, still gives Intel an edge with ChromeOS, because not all NaCl apps work on ARM Chromebooks. Seriously, how crazy is that? I can understand ARM not having a serious chance on Windows laptops because of all the legacy x86 programs, but at worst it should be equal with x86 running an OS such as Chrome OS.

TazeTSchnitzel 12y ago
I agree. asm.js has several strengths over NaCl and even PNaCl: * It is just JavaScript, so any browser can execute asm.js code, but it can be AOT compiled by some browsers for extra performance - This means every browser supports asm.js, and supports it today * It doesn't require a new runtime or API, merely using the existing web APIs * It is simple for other browser vendors to implement, and isn't reliant on a single implementation * It can interface with existing JavaScript code, so it can use JS libraries and, similarly, JS code can use asm.js libraries * It is a completely open standard * It is architecture-independent (so's PNaCl, though)
- masklinn 12y ago
  > It is just JavaScript, so any browser can execute asm.js code, but it can be AOT compiled by some browsers for extra performance Or JIT-compiled with phases specifically built for the kind of JS it produces e.g. Webkit's FTL tends to work very well on asm.js code, though it's not an asm.js AOT compiler. > It doesn't require a new runtime or API, merely using the existing web APIs That's not entirely true, AFAIK asm.js adds at least two functions to the Math namespace (imul and fround)
  - TazeTSchnitzel 12y ago
    > Or JIT-compiled with phases specifically built for the kind of JS it produces e.g. Webkit's FTL tends to work very well on asm.js code, though it's not an asm.js AOT compiler. This is true, too. While I think you'd get the best results from AOT compilation, it should also compile really well under JITs since it has all the type information you normally have to guess. > That's not entirely true, AFAIK asm.js adds at least two functions to the Math namespace (imul and fround) That's an addition to JavaScript which isn't specific to asm.js, however, and both can be polyfilled. Math.fround, in particular, is useful outside of asm.js. The stuff asm.js needs is all part of standard JS now, while NaCl and PNaCl require their own special API (Pepper) which only has one implementation (Google's).

StevePerkins 12y ago

I'm often dismissive of Mozilla, for being so all-over-the-map and unfocused sometimes. However, I'm grateful for them being the only browser player that really does seem interested in progressing JavaScript. All of the other players seem more focused on transpiling solutions, which are really about replacing JavaScript with something else under their control. Mozilla is pretty much the only player that genuinely works to improve JavaScript without another agenda.

rictic 12y ago
I don't think it's fair to characterize Traceur or TypeScript as plays to put Javascript under their author's control. Both have been useful for developing ES6, which Firefox has also been a leading implementer of. From my perspective almost all of the browser makers doing lots of cool stuff to push forward javascript and the web.
- TazeTSchnitzel 12y ago
  You're assuming StevePerkins was referring to Tracuer or TypeScript, I don't think they were. I assume they meant things like NaCl and Dart.
  - cromwellian 12y ago
    Dart compiled to JS just like Emscripten.
    - TazeTSchnitzel 12y ago
      Technically it compiles to JS, yes, but the reason people have a problem with Dart is the idea of embedding a Dart VM in the browser. (And some features of the core language only work with an embedded Dart VM)
  - rictic 12y ago
    A bit of an assumption yes, here's my reasoning though. Traceur and Typescript (and perhaps Flow and AtScript) are similar efforts made by different players that require transpilation and are potential visions for a direction javascript could go in. This combined with the author's quote "All of the other players..." made me think that's the family of solutions they were referring to. NaCl and Dart are both by a single player and there's nothing else like them that I know about but Mozilla's asm.js, which the author is praising here, so they didn't seem like likely candidates.