3 ms·

Does Microsoft cheat in IE Test Drive?

12 points by wolframhempel 14y ago

kevingadd 14y ago
This article feels like it ends 10% of the way to actually answering the question it asks. "Hey, this thing is slow. I'll write something that might be equivalent, and see if it's not slow, and then I'll stop." If you're going to accuse someone of 'cheating' or otherwise rigging a benchmark, maybe you should at least do a proper benchmark yourself, or figure out how they cheated? EDIT: Some basic digging reveals that this is using web workers, and V8's profiler does't record samples for web workers. It appears to be much slower in the current Release build of Chrome than it is in Canary, so whatever is causing it to be slow in V8 is actively being worked on by Chrome's developers. Unfortunately Firefox's profiler only profiles the main thread as well. Furthermore the mandelbrot demo doesn't work right in Aurora or Nightly, so uh, yeah. Given the use of workers, one guess would be that IE10 offers a much faster implementation of postMessage, making it cheaper to send the rendered pixel data back from the workers when it's done. Another obvious candidate would be a much faster implementation of putImageData for rendering the pixels to the canvas. A third candidate would be the use of something that IE10's JIT optimizes that V8 doesn't and Firefox struggles with slightly. Try/catch would be an obvious one; floats would be another. The code for the demo is there, and it's got comments, so if you really think they're cheating, read it. It looks fine to me and it looks like something that V8 should be able to turn into highly performant assembly, so if it's not, the fault probably lies with V8, not Microsoft.
- Trezoid 14y ago
  Firefox nightly 20.0a1 (released today) runs the test...very well, actually.
Toshio 14y ago
Chrome has a very good built-in performance profiler for JavaScript. In the interest of scientific truth, someone (not me) should profile microsoft's fabricated example and show the world all the dirty browser-sniffing tricks.
- kevingadd 14y ago
  You don't have to browser sniff to make something run slowly in V8. You can just write ordinary JavaScript and discover that it runs awful because you hit one of the hundreds of JavaScript edge cases that each JIT treats differently.
verroq 14y ago
It doesn't matter how fast yours run. It's the comparison that matters.
Trezoid 14y ago
Running the test on the latest firefox nightly results in a time of ~3 seconds, so if microsoft IS cheating by doing some browser sniffing they might want to widen the net somewhat, or end up looking bad (in 18 weeks...) The other option, and still a little dodgy, is that they found operations that ran poorly in V8, optimized the hell out of them in chakra specifically to say "Look, we're faster then chrome!" while just letting the fact that spidermonkey is faster slide because mozilla is way less of a threat then google.
- kevingadd 14y ago
  Everyone optimizes for their own benchmarks. You can say it's dodgy, but that's really just an argument in favor of real-world tests instead of microbenchmarks like this one. You don't have to 'rig' a microbenchmark to come out the winner; you just pick the right microbenchmark, because it's not a real world scenario. Given the lack of evidence to support actual cheating you should probably just apply occam's razor here and assume it's actually faster. Or you could use the dark, hidden secrets of 'view source' to read the JS yourself. It has comments!
  - Trezoid 14y ago
    making a benchmark specifically to be slower on your competitors browser? Little dodgy. From everything I've read IE10 apparently is fast, and the numbers are believable from my experience doing similar things (web workers drawing fractal-like images on a canvas) but given that it's the only "pretty" benchmark in the lot, it just seems a little strange.
    - kevingadd 14y ago
      I think you misread my post. My point is that everyone optimizes their implementation so that their benchmarks run fast. Furthermore, people pick benchmarks that run fast on their implementation. Neither of these are a 'little dodgy': They are how benchmarks get used. You might note that they make for bad science, but benchmarks tend not to get used in anything resembling a scientific comparison. Benchmarks get used for comparative testing (did we make thing X faster than it was before?) and to try and prove that your product is better than the competition (in which case much larger lies are going to get trotted out either way).
opinali 14y ago
Googler here. There is no cheating in the benchmark. But we noticed this, and the V8 devs immediately diagnosed it as result of an optimizer failure (not exactly a bug, but a limitation of current stuff). There's already an in-progress patch that fixes this problem, so just wait for this to land: https://chromiumcodereview.appspot.com/10837165 https://chromiumcodereview.appspot.com/10837165

12 points by wolframhempel 14y ago

This article feels like it ends 10% of the way to actually answering the question it asks. "Hey, this thing is slow. I'll write something that might be equivalent, and see if it's not slow, and then I'll stop." If you're going to accuse someone of 'cheating' or otherwise rigging a benchmark, maybe you should at least do a proper benchmark yourself, or figure out how they cheated? EDIT: Some basic digging reveals that this is using web workers, and V8's profiler does't record samples for web workers. It appears to be much slower in the current Release build of Chrome than it is in Canary, so whatever is causing it to be slow in V8 is actively being worked on by Chrome's developers. Unfortunately Firefox's profiler only profiles the main thread as well. Furthermore the mandelbrot demo doesn't work right in Aurora or Nightly, so uh, yeah. Given the use of workers, one guess would be that IE10 offers a much faster implementation of postMessage, making it cheaper to send the rendered pixel data back from the workers when it's done. Another obvious candidate would be a much faster implementation of putImageData for rendering the pixels to the canvas. A third candidate would be the use of something that IE10's JIT optimizes that V8 doesn't and Firefox struggles with slightly. Try/catch would be an obvious one; floats would be another. The code for the demo is there, and it's got comments, so if you really think they're cheating, read it. It looks fine to me and it looks like something that V8 should be able to turn into highly performant assembly, so if it's not, the fault probably lies with V8, not Microsoft.

Trezoid 14y ago
Firefox nightly 20.0a1 (released today) runs the test...very well, actually.

Toshio 14y ago

Chrome has a very good built-in performance profiler for JavaScript. In the interest of scientific truth, someone (not me) should profile microsoft's fabricated example and show the world all the dirty browser-sniffing tricks.

kevingadd 14y ago
You don't have to browser sniff to make something run slowly in V8. You can just write ordinary JavaScript and discover that it runs awful because you hit one of the hundreds of JavaScript edge cases that each JIT treats differently.

verroq 14y ago

It doesn't matter how fast yours run. It's the comparison that matters.

Trezoid 14y ago

Running the test on the latest firefox nightly results in a time of ~3 seconds, so if microsoft IS cheating by doing some browser sniffing they might want to widen the net somewhat, or end up looking bad (in 18 weeks...) The other option, and still a little dodgy, is that they found operations that ran poorly in V8, optimized the hell out of them in chakra specifically to say "Look, we're faster then chrome!" while just letting the fact that spidermonkey is faster slide because mozilla is way less of a threat then google.

kevingadd 14y ago
Everyone optimizes for their own benchmarks. You can say it's dodgy, but that's really just an argument in favor of real-world tests instead of microbenchmarks like this one. You don't have to 'rig' a microbenchmark to come out the winner; you just pick the right microbenchmark, because it's not a real world scenario. Given the lack of evidence to support actual cheating you should probably just apply occam's razor here and assume it's actually faster. Or you could use the dark, hidden secrets of 'view source' to read the JS yourself. It has comments!
- Trezoid 14y ago
  making a benchmark specifically to be slower on your competitors browser? Little dodgy. From everything I've read IE10 apparently is fast, and the numbers are believable from my experience doing similar things (web workers drawing fractal-like images on a canvas) but given that it's the only "pretty" benchmark in the lot, it just seems a little strange.
  - kevingadd 14y ago
    I think you misread my post. My point is that everyone optimizes their implementation so that their benchmarks run fast. Furthermore, people pick benchmarks that run fast on their implementation. Neither of these are a 'little dodgy': They are how benchmarks get used. You might note that they make for bad science, but benchmarks tend not to get used in anything resembling a scientific comparison. Benchmarks get used for comparative testing (did we make thing X faster than it was before?) and to try and prove that your product is better than the competition (in which case much larger lies are going to get trotted out either way).

opinali 14y ago

Googler here. There is no cheating in the benchmark. But we noticed this, and the V8 devs immediately diagnosed it as result of an optimizer failure (not exactly a bug, but a limitation of current stuff). There's already an in-progress patch that fixes this problem, so just wait for this to land: https://chromiumcodereview.appspot.com/10837165 https://chromiumcodereview.appspot.com/10837165