5 ms·

Octane benchmark retired

90 points by natorion 9y ago

skrowl 9y ago
They probably got tired of FF & Edge beating them in their own benchmark. Can you imagine the conversation with some middle level manager when you were asked to explain that?
- Lerc 9y ago
  That does seem to be a concern here. They are replacing Octane with what amounts to a-collection-of-stuff. Without a specific workset (ie. a benchmark) there is no clear way to measure one thing against another. This seems like a mechanism to hide where you stand. Now they can just say "That's not what we find in our tests". I'm inclined to agree that becnhmarks do become obsolete as workloads change and development targets benchmark pointscoring. The solution is to make a new standardised benchmark, or indeed many of them. If they are clear about what they are measuring then the relative perfomance of different JavaScript engines can be understood by developers to give a nuanced view of what performs well.
  - codinghorror 9y ago
    We find that http://browserbench.org/Speedometer/ http://browserbench.org/Speedometer/ -- which is what v8 is optimizing against now -- is extremely close to real world Discourse (EmberJS, Glimmer) performance. We always wondered why Android was so crazy ridiculously slow at our real world JS vs. iOS starting in late 2013 onward.. getting worse and worse over 2014 and 2015.. and "gee, we optimized for the wrong thing aka Octane" was a big part of that. http://benediktmeurer.de/2016/12/16/the-truth-about-traditional-javascript-benchmarks/ http://benediktmeurer.de/2016/12/16/the-truth-about-traditio... You can read more about the history at https://meta.discourse.org/t/the-state-of-javascript-on-android-in-2015-is-poor/33889 https://meta.discourse.org/t/the-state-of-javascript-on-andr...
    - om2 9y ago
      JavaScriptCore (WebKit's JS engine) also optimized for Octane. But we additionally optimized for Speedometer, Sunspider (the not totally broken subset that made it into Jetstream) and JSBench. While Octane isn't the single answer to all JS performance measurement, using it in combination with better things seems better than not using it. To go back to your example, iOS both has way better real-world perf and way better Octane results than Android devices. BTW benchmarks I'd endorse include MotionMark, Speedometer and Jetstream from http://browserbench.org/ http://browserbench.org/ as well as JSBench http://plg.uwaterloo.ca/~dynjs/jsbench/ http://plg.uwaterloo.ca/~dynjs/jsbench/ These are all good at testing different aspects of browser performance including relevant testing of the JS engine.
- pizlonator 9y ago
  WebKit is currently king on Octane fwiw. We still think it's a great benchmark. V8 made the mistake of focusing only on this benchmark, so they didn't know that some of their Octane optimizations were bad for other scenarios. We didn't have that problem because we have always used a broader spectrum of benchmarks.
  - bterlson 9y ago
    Agreed, Octane was good as far as benchmarks go but you have to balance with a much larger set of benchmarks and always ground your work in real-world scenarios (something we focused heavily on from day one e.g. when we realized having an interpreter really helps startup on lots of real-world apps but doesn't do much for Octane score). That said, FWIW I think Edge/Chakra gets the crown on Windows! (I work on Chakra)
    - vanderZwan 9y ago
      Out of curiosity: does only having to target one OS make a difference here? Is it "easier" to get it to run fast? Any worries that you don't have compared to Firefox and Chrome, even though you purely work on the JS side of things?
      - bterlson 9y ago
        I'm sorry I missed your comment earlier, but hopefully replying later is better than never :) ChakraCore is cross-platform now so we no longer get the benefit of a single platform. However back when we were only Windows there were advantages, certainly, if only because all the work spent getting everything working on different OSes can instead be invested in improving performance.
  - cromwellian 9y ago
    I don't think they only focused on Octane, it's possible they may have weighted it more highly, I can't speak for the v8 team (I don't work on v8), but that sounds like an oversimplification of what probably happened. One example could be focusing on throughput instead of startup latency which could yield distorted outcomes for real world apps, or mostly app code that is mostly early-bound style OO code/monomorphic functional code vs code that is hostile to class shape analysis/PIC. It seems in the last few years, especially with the idea of Web competing with native with stuff like asm.js for games, there was a focus on maximize throughput, but it's rare your typical interactive web app is running code that benefits from that.
dallamaneni 9y ago
Looks like someone at Google realized that Servo (Quantum) is coming
- Etzos 9y ago
  Servo uses SpiderMonkey to provide Javascript support, so retiring Octane (a Javascript engine benchmark) would have nothing to do with Servo (a layout engine).
  - dallamaneni 9y ago
    Got it. I did'nt realize that.
bozonil 9y ago
First sunspider, now octane. This shows that browser vendors are not really good at writing JS benchmarks representative of real world, which is not surprising considering that they are not doing real web development. Ideally benchmarks would come from developers of popular websites like Facebook, Twitter, etc.
- ajross 9y ago
  That's just a truism about benchmarks. Once all the low hanging fruit is picked they stop showing interesting progress and just become a garden for crazy microoptimizations. The same thing happened with C compilers 10-15 years ago. Basically, Javascript performance is "done". It's not going to get much faster at this point, and higher performance will come from paradigm shifts (SIMD, WebAssembly, pick your fad).
franciscop 9y ago
This actually makes sense. When the low level stuff is at a terrible level they made Octane to make that better. Once that was fixed, optimizing for it is counter-productive. This has a big resemblance IMO to most formal study methods including University and natural languages. Once you know the basics you can optimize for learning or for grades, depending on your goals. They are correlated (but not strongly correlated), but the time investment is not negligible so you'll have to choose either of them.

90 points by natorion 9y ago

They probably got tired of FF & Edge beating them in their own benchmark. Can you imagine the conversation with some middle level manager when you were asked to explain that?

Lerc 9y ago
That does seem to be a concern here. They are replacing Octane with what amounts to a-collection-of-stuff. Without a specific workset (ie. a benchmark) there is no clear way to measure one thing against another. This seems like a mechanism to hide where you stand. Now they can just say "That's not what we find in our tests". I'm inclined to agree that becnhmarks do become obsolete as workloads change and development targets benchmark pointscoring. The solution is to make a new standardised benchmark, or indeed many of them. If they are clear about what they are measuring then the relative perfomance of different JavaScript engines can be understood by developers to give a nuanced view of what performs well.
- codinghorror 9y ago
  We find that http://browserbench.org/Speedometer/ http://browserbench.org/Speedometer/ -- which is what v8 is optimizing against now -- is extremely close to real world Discourse (EmberJS, Glimmer) performance. We always wondered why Android was so crazy ridiculously slow at our real world JS vs. iOS starting in late 2013 onward.. getting worse and worse over 2014 and 2015.. and "gee, we optimized for the wrong thing aka Octane" was a big part of that. http://benediktmeurer.de/2016/12/16/the-truth-about-traditional-javascript-benchmarks/ http://benediktmeurer.de/2016/12/16/the-truth-about-traditio... You can read more about the history at https://meta.discourse.org/t/the-state-of-javascript-on-android-in-2015-is-poor/33889 https://meta.discourse.org/t/the-state-of-javascript-on-andr...
  - om2 9y ago
    JavaScriptCore (WebKit's JS engine) also optimized for Octane. But we additionally optimized for Speedometer, Sunspider (the not totally broken subset that made it into Jetstream) and JSBench. While Octane isn't the single answer to all JS performance measurement, using it in combination with better things seems better than not using it. To go back to your example, iOS both has way better real-world perf and way better Octane results than Android devices. BTW benchmarks I'd endorse include MotionMark, Speedometer and Jetstream from http://browserbench.org/ http://browserbench.org/ as well as JSBench http://plg.uwaterloo.ca/~dynjs/jsbench/ http://plg.uwaterloo.ca/~dynjs/jsbench/ These are all good at testing different aspects of browser performance including relevant testing of the JS engine.
pizlonator 9y ago
WebKit is currently king on Octane fwiw. We still think it's a great benchmark. V8 made the mistake of focusing only on this benchmark, so they didn't know that some of their Octane optimizations were bad for other scenarios. We didn't have that problem because we have always used a broader spectrum of benchmarks.
- bterlson 9y ago
  Agreed, Octane was good as far as benchmarks go but you have to balance with a much larger set of benchmarks and always ground your work in real-world scenarios (something we focused heavily on from day one e.g. when we realized having an interpreter really helps startup on lots of real-world apps but doesn't do much for Octane score). That said, FWIW I think Edge/Chakra gets the crown on Windows! (I work on Chakra)
  - vanderZwan 9y ago
    Out of curiosity: does only having to target one OS make a difference here? Is it "easier" to get it to run fast? Any worries that you don't have compared to Firefox and Chrome, even though you purely work on the JS side of things?
    - bterlson 9y ago
      I'm sorry I missed your comment earlier, but hopefully replying later is better than never :) ChakraCore is cross-platform now so we no longer get the benefit of a single platform. However back when we were only Windows there were advantages, certainly, if only because all the work spent getting everything working on different OSes can instead be invested in improving performance.
- cromwellian 9y ago
  I don't think they only focused on Octane, it's possible they may have weighted it more highly, I can't speak for the v8 team (I don't work on v8), but that sounds like an oversimplification of what probably happened. One example could be focusing on throughput instead of startup latency which could yield distorted outcomes for real world apps, or mostly app code that is mostly early-bound style OO code/monomorphic functional code vs code that is hostile to class shape analysis/PIC. It seems in the last few years, especially with the idea of Web competing with native with stuff like asm.js for games, there was a focus on maximize throughput, but it's rare your typical interactive web app is running code that benefits from that.

dallamaneni 9y ago

Looks like someone at Google realized that Servo (Quantum) is coming

Etzos 9y ago
Servo uses SpiderMonkey to provide Javascript support, so retiring Octane (a Javascript engine benchmark) would have nothing to do with Servo (a layout engine).
- dallamaneni 9y ago
  Got it. I did'nt realize that.

bozonil 9y ago

First sunspider, now octane. This shows that browser vendors are not really good at writing JS benchmarks representative of real world, which is not surprising considering that they are not doing real web development. Ideally benchmarks would come from developers of popular websites like Facebook, Twitter, etc.

ajross 9y ago
That's just a truism about benchmarks. Once all the low hanging fruit is picked they stop showing interesting progress and just become a garden for crazy microoptimizations. The same thing happened with C compilers 10-15 years ago. Basically, Javascript performance is "done". It's not going to get much faster at this point, and higher performance will come from paradigm shifts (SIMD, WebAssembly, pick your fad).

franciscop 9y ago

This actually makes sense. When the low level stuff is at a terrible level they made Octane to make that better. Once that was fixed, optimizing for it is counter-productive. This has a big resemblance IMO to most formal study methods including University and natural languages. Once you know the basics you can optimize for learning or for grades, depending on your goals. They are correlated (but not strongly correlated), but the time investment is not negligible so you'll have to choose either of them.