5 ms·

They probably got tired of FF & Edge beating them in their own benchmark. Can you imagine the conversation with some middle level manager when you were asked to

by skrowl 10y ago

They probably got tired of FF & Edge beating them in their own benchmark. Can you imagine the conversation with some middle level manager when you were asked to explain that?

Lerc 10y ago
That does seem to be a concern here. They are replacing Octane with what amounts to a-collection-of-stuff. Without a specific workset (ie. a benchmark) there is no clear way to measure one thing against another. This seems like a mechanism to hide where you stand. Now they can just say "That's not what we find in our tests". I'm inclined to agree that becnhmarks do become obsolete as workloads change and development targets benchmark pointscoring. The solution is to make a new standardised benchmark, or indeed many of them. If they are clear about what they are measuring then the relative perfomance of different JavaScript engines can be understood by developers to give a nuanced view of what performs well.
- codinghorror 10y ago
  We find that http://browserbench.org/Speedometer/ http://browserbench.org/Speedometer/ -- which is what v8 is optimizing against now -- is extremely close to real world Discourse (EmberJS, Glimmer) performance. We always wondered why Android was so crazy ridiculously slow at our real world JS vs. iOS starting in late 2013 onward.. getting worse and worse over 2014 and 2015.. and "gee, we optimized for the wrong thing aka Octane" was a big part of that. http://benediktmeurer.de/2016/12/16/the-truth-about-traditional-javascript-benchmarks/ http://benediktmeurer.de/2016/12/16/the-truth-about-traditio... You can read more about the history at https://meta.discourse.org/t/the-state-of-javascript-on-android-in-2015-is-poor/33889 https://meta.discourse.org/t/the-state-of-javascript-on-andr...
  - om2 10y ago
    JavaScriptCore (WebKit's JS engine) also optimized for Octane. But we additionally optimized for Speedometer, Sunspider (the not totally broken subset that made it into Jetstream) and JSBench. While Octane isn't the single answer to all JS performance measurement, using it in combination with better things seems better than not using it. To go back to your example, iOS both has way better real-world perf and way better Octane results than Android devices. BTW benchmarks I'd endorse include MotionMark, Speedometer and Jetstream from http://browserbench.org/ http://browserbench.org/ as well as JSBench http://plg.uwaterloo.ca/~dynjs/jsbench/ http://plg.uwaterloo.ca/~dynjs/jsbench/ These are all good at testing different aspects of browser performance including relevant testing of the JS engine.
pizlonator 10y ago
WebKit is currently king on Octane fwiw. We still think it's a great benchmark. V8 made the mistake of focusing only on this benchmark, so they didn't know that some of their Octane optimizations were bad for other scenarios. We didn't have that problem because we have always used a broader spectrum of benchmarks.
- bterlson 10y ago
  Agreed, Octane was good as far as benchmarks go but you have to balance with a much larger set of benchmarks and always ground your work in real-world scenarios (something we focused heavily on from day one e.g. when we realized having an interpreter really helps startup on lots of real-world apps but doesn't do much for Octane score). That said, FWIW I think Edge/Chakra gets the crown on Windows! (I work on Chakra)
  - vanderZwan 10y ago
    Out of curiosity: does only having to target one OS make a difference here? Is it "easier" to get it to run fast? Any worries that you don't have compared to Firefox and Chrome, even though you purely work on the JS side of things?
    - bterlson 10y ago
      I'm sorry I missed your comment earlier, but hopefully replying later is better than never :) ChakraCore is cross-platform now so we no longer get the benefit of a single platform. However back when we were only Windows there were advantages, certainly, if only because all the work spent getting everything working on different OSes can instead be invested in improving performance.
- cromwellian 10y ago
  I don't think they only focused on Octane, it's possible they may have weighted it more highly, I can't speak for the v8 team (I don't work on v8), but that sounds like an oversimplification of what probably happened. One example could be focusing on throughput instead of startup latency which could yield distorted outcomes for real world apps, or mostly app code that is mostly early-bound style OO code/monomorphic functional code vs code that is hostile to class shape analysis/PIC. It seems in the last few years, especially with the idea of Web competing with native with stuff like asm.js for games, there was a focus on maximize throughput, but it's rare your typical interactive web app is running code that benefits from that.

by skrowl 10y ago

They probably got tired of FF & Edge beating them in their own benchmark. Can you imagine the conversation with some middle level manager when you were asked to explain that?

Lerc 10y ago

That does seem to be a concern here. They are replacing Octane with what amounts to a-collection-of-stuff. Without a specific workset (ie. a benchmark) there is no clear way to measure one thing against another. This seems like a mechanism to hide where you stand. Now they can just say "That's not what we find in our tests". I'm inclined to agree that becnhmarks do become obsolete as workloads change and development targets benchmark pointscoring. The solution is to make a new standardised benchmark, or indeed many of them. If they are clear about what they are measuring then the relative perfomance of different JavaScript engines can be understood by developers to give a nuanced view of what performs well.

codinghorror 10y ago
We find that http://browserbench.org/Speedometer/ http://browserbench.org/Speedometer/ -- which is what v8 is optimizing against now -- is extremely close to real world Discourse (EmberJS, Glimmer) performance. We always wondered why Android was so crazy ridiculously slow at our real world JS vs. iOS starting in late 2013 onward.. getting worse and worse over 2014 and 2015.. and "gee, we optimized for the wrong thing aka Octane" was a big part of that. http://benediktmeurer.de/2016/12/16/the-truth-about-traditional-javascript-benchmarks/ http://benediktmeurer.de/2016/12/16/the-truth-about-traditio... You can read more about the history at https://meta.discourse.org/t/the-state-of-javascript-on-android-in-2015-is-poor/33889 https://meta.discourse.org/t/the-state-of-javascript-on-andr...
- om2 10y ago
  JavaScriptCore (WebKit's JS engine) also optimized for Octane. But we additionally optimized for Speedometer, Sunspider (the not totally broken subset that made it into Jetstream) and JSBench. While Octane isn't the single answer to all JS performance measurement, using it in combination with better things seems better than not using it. To go back to your example, iOS both has way better real-world perf and way better Octane results than Android devices. BTW benchmarks I'd endorse include MotionMark, Speedometer and Jetstream from http://browserbench.org/ http://browserbench.org/ as well as JSBench http://plg.uwaterloo.ca/~dynjs/jsbench/ http://plg.uwaterloo.ca/~dynjs/jsbench/ These are all good at testing different aspects of browser performance including relevant testing of the JS engine.

pizlonator 10y ago

WebKit is currently king on Octane fwiw. We still think it's a great benchmark. V8 made the mistake of focusing only on this benchmark, so they didn't know that some of their Octane optimizations were bad for other scenarios. We didn't have that problem because we have always used a broader spectrum of benchmarks.

bterlson 10y ago
Agreed, Octane was good as far as benchmarks go but you have to balance with a much larger set of benchmarks and always ground your work in real-world scenarios (something we focused heavily on from day one e.g. when we realized having an interpreter really helps startup on lots of real-world apps but doesn't do much for Octane score). That said, FWIW I think Edge/Chakra gets the crown on Windows! (I work on Chakra)
- vanderZwan 10y ago
  Out of curiosity: does only having to target one OS make a difference here? Is it "easier" to get it to run fast? Any worries that you don't have compared to Firefox and Chrome, even though you purely work on the JS side of things?
  - bterlson 10y ago
    I'm sorry I missed your comment earlier, but hopefully replying later is better than never :) ChakraCore is cross-platform now so we no longer get the benefit of a single platform. However back when we were only Windows there were advantages, certainly, if only because all the work spent getting everything working on different OSes can instead be invested in improving performance.
cromwellian 10y ago
I don't think they only focused on Octane, it's possible they may have weighted it more highly, I can't speak for the v8 team (I don't work on v8), but that sounds like an oversimplification of what probably happened. One example could be focusing on throughput instead of startup latency which could yield distorted outcomes for real world apps, or mostly app code that is mostly early-bound style OO code/monomorphic functional code vs code that is hostile to class shape analysis/PIC. It seems in the last few years, especially with the idea of Web competing with native with stuff like asm.js for games, there was a focus on maximize throughput, but it's rare your typical interactive web app is running code that benefits from that.