7 ms·

Simplifying for my tiny brain: does this mean that IE9 takes more than 22x longer to complete a "real" test instead of the "optimized" one?

by seldo 16y ago

julian37 16y ago
I don't have IE9 installed and therefore can't verify the benchmark results. If they are genuine, I'm having a hard time coming up with a different conclusion. By the way, hilarious idea to make this into a bug report. EDIT: of course I find it much easier to believe that someone at Microsoft optimized for the benchmark than that someone at Mozilla would fudge the timing results, especially when it's so easy to verify the claims.
- JoachimSchipper 16y ago
  > hilarious idea to make this into a bug report. Actually, the MS guys requested that. I'm sure there's a lesson about soulless bureaucracies somewhere in there...
  - DougBTX 16y ago
    Bugs are more likely to get fixed if they are posted somewhere visible, instead of sitting in someone's inbox...
cheald 16y ago
Edit (yet again): My initial conclusions were wrong, and it's nearly certainly cheating. Dammit. I hate being wrong in front of people smarter than me. :< ---- I'm running the same benchmark independently right now. Core i7 in a Win7 64-bit install. For each test, I did 5 runs and averaged them. I increased the number of loops in each test from 25,000 to 250,000 as well. Chrome 9.0.576.0 Stock: 105.28ms With "true": 104.44ms MS IE 9.0.7930.16406 Stock: 10.98ms With "true": 181.16ms That's a pretty interesting jump. Here's a fun little observation: Chome: 8.6ms IE: 10.9ms --- tests/sunspider-0.9.1/math-cordic.js 2010-11-17 00:55:29.000000000 -0700 +++ tests/sunspider-0.9.1-deadcode/math-cordic.js 2010-11-17 01:10:36.000000000 -0700 @@ -63,6 +63,7 @@ TargetAngle = FIXED(28.027); CurrAngle = 0; for (Step = 0; Step < 12; Step++) { + return; var NewX; if (TargetAngle > CurrAngle) { NewX = X - (Y >> Step); By returning immediately out of the loop, Chrome's time drops by a factor of 12.1, whereas IE's stays pretty much constant. I suspect what's happening here is that the IE engine is somehow marking that entire function as deadcode, and thus, not running it; the ~10ms accounts for the time it takes to run that for loop 250k times, but the cordicsincos() code is not being run at all. Ironically, deadcode somewhere in the function causes the engine to NOT throw it all away, and its gets run. In fact, if we just kill that for loop all together: --- tests/sunspider-0.9.1/math-cordic.js 2010-11-17 00:55:29.000000000 -0700 +++ tests/sunspider-0.9.1-deadcode/math-cordic.js 2010-11-17 01:16:36.000000000 -0700 @@ -62,20 +62,6 @@ TargetAngle = FIXED(28.027); CurrAngle = 0; - for (Step = 0; Step < 12; Step++) { - *snip for brevity* - } } We get the following times: Chrome: 8.9ms IE: 10.6ms What I suspect is that the IE engine is seeing "Okay, nothing is returned, and nothing outside of the scope of this function is ever altered", so once it steps into it, it just immediately returns. This is arguably correct behavior! That code is, for all practical purposes, worthless. If we just move one of the variable references out of function scope (or just remove the var, making it effectively a global variable), IE takes the extra time to run: --- tests/sunspider-0.9.1/math-cordic.js 2010-11-17 00:55:29.000000000 -0700 +++ tests/sunspider-0.9.1-deadcode/math-cordic.js 2010-11-17 01:22:49.000000000 -0700 @@ -50,11 +50,11 @@ +var CurrAngle; function cordicsincos() { var X; var Y; var TargetAngle; - var CurrAngle; var Step; X = FIXED(AG_CONST); /* AG_CONST * cos(0) */ Chrome: 99.9ms IE: 217.1ms Sorry, guys. I like a good IE bash-fest (Hey, it's still slower than V8 when it actually runs the code!) as much as anyone, but I think it's legit here. The benchmark is poorly-conceived, and IE does the right thing with it, though it obviously distorts the scores in their favor. That's a problem with the benchmark, though, not IE. Edit (like...#14): It could well just be cheating on analysis in this particular case, which I stupidly overlooked. For example, this diff: --- tests/sunspider-0.9.1/math-cordic.js 2010-11-17 00:55:29.000000000 -0700 +++ tests/sunspider-0.9.1-deadcode/math-cordic.js 2010-11-17 02:09:34.000000000 -0700 @@ -62,7 +62,7 @@ TargetAngle = FIXED(28.027); CurrAngle = 0; - for (Step = 0; Step < 12; Step++) { + for (Step = 12; Step > 0; Step--) { var NewX; if (TargetAngle > CurrAngle) { NewX = X - (Y >> Step); Results in runtimes of: Chrome: 246.8ms IE: 956.5ms Replacing the for with a while also results in long runtimes: --- tests/sunspider-0.9.1/math-cordic.js 2010-11-17 00:55:29.000000000 -0700 +++ tests/sunspider-0.9.1-deadcode/math-cordic.js 2010-11-17 02:12:03.000000000 -0700 @@ -59,10 +59,11 @@ X = FIXED(AG_CONST); /* AG_CONST * cos(0) */ Y = 0; /* AG_CONST * sin(0) */ + Step = 0; TargetAngle = FIXED(28.027); CurrAngle = 0; - for (Step = 0; Step < 12; Step++) { + while(Step < 12) { var NewX; if (TargetAngle > CurrAngle) { NewX = X - (Y >> Step); @@ -75,6 +76,7 @@ X = NewX; CurrAngle -= Angles[Step]; } + Step++; } } Chrome: 103.4ms IE: 190.0ms So, my initial conclusions were wrong. Its dead code analysis is either incredibly narrow, or it was hand-crafted to optimize out that part of the benchmark. Either way it's rubbish.
- moconnor 16y ago
  The claim that IE is legit hinges on the diffs causing a valid bug in IE's optimization code (e.g. deadcode inside deadcode prevents the latter being optimized out), versus foul play in the engine (a hard-coded case for this benchmark). Can you find any variation on the benchmark code that still allows IE to optimize it, or does it only optimize the exact form of the code used in the benchmark suite?
  - cheald 16y ago
    I changed variable names and declaration order, the number of loops in that inner for loop, and other such things that could possibly change the bytecode (to what effect, I don't know - I'm not a JS VM engineer, obviously) without changing the operations actually performed. I don't have an explanation for this, though (maybe variable initialization counts as "run up until here"?): Runs fast (11ms): --- tests/sunspider-0.9.1/math-cordic.js 2010-11-17 00:55:29.000000000 -0700 +++ tests/sunspider-0.9.1-deadcode/math-cordic.js 2010-11-17 01:42:43.000000000 -0700 @@ -56,12 +56,14 @@ var TargetAngle; var CurrAngle; var Step; + var foo; X = FIXED(AG_CONST); /* AG_CONST * cos(0) */ Y = 0; /* AG_CONST * sin(0) */ TargetAngle = FIXED(28.027); CurrAngle = 0; + foo = 1; for (Step = 0; Step < 12; Step++) { var NewX; if (TargetAngle > CurrAngle) { But if I assign foo after the for loop, it runs slow: --- tests/sunspider-0.9.1/math-cordic.js 2010-11-17 00:55:29.000000000 -0700 +++ tests/sunspider-0.9.1-deadcode/math-cordic.js 2010-11-17 01:43:20.000000000 -0700 @@ -56,6 +56,7 @@ var TargetAngle; var CurrAngle; var Step; + var foo; X = FIXED(AG_CONST); /* AG_CONST * cos(0) */ Y = 0; /* AG_CONST * sin(0) */ @@ -76,6 +77,7 @@ CurrAngle -= Angles[Step]; } } + foo = 1; } If I assign foo inside the for loop, it runs fast: --- tests/sunspider-0.9.1/math-cordic.js 2010-11-17 00:55:29.000000000 -0700 +++ tests/sunspider-0.9.1-deadcode/math-cordic.js 2010-11-17 01:44:41.000000000 -0700 @@ -56,6 +56,7 @@ var TargetAngle; var CurrAngle; var Step; + var foo; X = FIXED(AG_CONST); /* AG_CONST * cos(0) */ Y = 0; /* AG_CONST * sin(0) */ @@ -63,6 +64,7 @@ TargetAngle = FIXED(28.027); CurrAngle = 0; for (Step = 0; Step < 12; Step++) { + foo = 1; var NewX; if (TargetAngle > CurrAngle) { NewX = X - (Y >> Step); Color me boggled.
detcader 16y ago
What if it just means that IE9 takes 21 more milliseconds to complete a "real" test instead of the "optimized" one?</silly question>
lty13800 16y ago
um.. Are you using the right version of IE 9? The current version is IE 9 PP7 (1.9.8023.6000), which produced a Sunspider result of 216ms. You are testing the beta version of IE 9, which was released in back in September. Of course the old version of IE 9 will have a much slower speed. By the way the Sunspider result for IE 9 beta was about 340 ms

Simplifying for my tiny brain: does this mean that IE9 takes more than 22x longer to complete a "real" test instead of the "optimized" one?

by seldo 16y ago

Simplifying for my tiny brain: does this mean that IE9 takes more than 22x longer to complete a "real" test instead of the "optimized" one?

julian37 16y ago

I don't have IE9 installed and therefore can't verify the benchmark results. If they are genuine, I'm having a hard time coming up with a different conclusion. By the way, hilarious idea to make this into a bug report. EDIT: of course I find it much easier to believe that someone at Microsoft optimized for the benchmark than that someone at Mozilla would fudge the timing results, especially when it's so easy to verify the claims.

JoachimSchipper 16y ago
> hilarious idea to make this into a bug report. Actually, the MS guys requested that. I'm sure there's a lesson about soulless bureaucracies somewhere in there...
- DougBTX 16y ago
  Bugs are more likely to get fixed if they are posted somewhere visible, instead of sitting in someone's inbox...

cheald 16y ago

Edit (yet again): My initial conclusions were wrong, and it's nearly certainly cheating. Dammit. I hate being wrong in front of people smarter than me. :< ---- I'm running the same benchmark independently right now. Core i7 in a Win7 64-bit install. For each test, I did 5 runs and averaged them. I increased the number of loops in each test from 25,000 to 250,000 as well. Chrome 9.0.576.0 Stock: 105.28ms With "true": 104.44ms MS IE 9.0.7930.16406 Stock: 10.98ms With "true": 181.16ms That's a pretty interesting jump. Here's a fun little observation: Chome: 8.6ms IE: 10.9ms --- tests/sunspider-0.9.1/math-cordic.js 2010-11-17 00:55:29.000000000 -0700 +++ tests/sunspider-0.9.1-deadcode/math-cordic.js 2010-11-17 01:10:36.000000000 -0700 @@ -63,6 +63,7 @@ TargetAngle = FIXED(28.027); CurrAngle = 0; for (Step = 0; Step < 12; Step++) { + return; var NewX; if (TargetAngle > CurrAngle) { NewX = X - (Y >> Step); By returning immediately out of the loop, Chrome's time drops by a factor of 12.1, whereas IE's stays pretty much constant. I suspect what's happening here is that the IE engine is somehow marking that entire function as deadcode, and thus, not running it; the ~10ms accounts for the time it takes to run that for loop 250k times, but the cordicsincos() code is not being run at all. Ironically, deadcode somewhere in the function causes the engine to NOT throw it all away, and its gets run. In fact, if we just kill that for loop all together: --- tests/sunspider-0.9.1/math-cordic.js 2010-11-17 00:55:29.000000000 -0700 +++ tests/sunspider-0.9.1-deadcode/math-cordic.js 2010-11-17 01:16:36.000000000 -0700 @@ -62,20 +62,6 @@ TargetAngle = FIXED(28.027); CurrAngle = 0; - for (Step = 0; Step < 12; Step++) { - *snip for brevity* - } } We get the following times: Chrome: 8.9ms IE: 10.6ms What I suspect is that the IE engine is seeing "Okay, nothing is returned, and nothing outside of the scope of this function is ever altered", so once it steps into it, it just immediately returns. This is arguably correct behavior! That code is, for all practical purposes, worthless. If we just move one of the variable references out of function scope (or just remove the var, making it effectively a global variable), IE takes the extra time to run: --- tests/sunspider-0.9.1/math-cordic.js 2010-11-17 00:55:29.000000000 -0700 +++ tests/sunspider-0.9.1-deadcode/math-cordic.js 2010-11-17 01:22:49.000000000 -0700 @@ -50,11 +50,11 @@ +var CurrAngle; function cordicsincos() { var X; var Y; var TargetAngle; - var CurrAngle; var Step; X = FIXED(AG_CONST); /* AG_CONST * cos(0) */ Chrome: 99.9ms IE: 217.1ms Sorry, guys. I like a good IE bash-fest (Hey, it's still slower than V8 when it actually runs the code!) as much as anyone, but I think it's legit here. The benchmark is poorly-conceived, and IE does the right thing with it, though it obviously distorts the scores in their favor. That's a problem with the benchmark, though, not IE. Edit (like...#14): It could well just be cheating on analysis in this particular case, which I stupidly overlooked. For example, this diff: --- tests/sunspider-0.9.1/math-cordic.js 2010-11-17 00:55:29.000000000 -0700 +++ tests/sunspider-0.9.1-deadcode/math-cordic.js 2010-11-17 02:09:34.000000000 -0700 @@ -62,7 +62,7 @@ TargetAngle = FIXED(28.027); CurrAngle = 0; - for (Step = 0; Step < 12; Step++) { + for (Step = 12; Step > 0; Step--) { var NewX; if (TargetAngle > CurrAngle) { NewX = X - (Y >> Step); Results in runtimes of: Chrome: 246.8ms IE: 956.5ms Replacing the for with a while also results in long runtimes: --- tests/sunspider-0.9.1/math-cordic.js 2010-11-17 00:55:29.000000000 -0700 +++ tests/sunspider-0.9.1-deadcode/math-cordic.js 2010-11-17 02:12:03.000000000 -0700 @@ -59,10 +59,11 @@ X = FIXED(AG_CONST); /* AG_CONST * cos(0) */ Y = 0; /* AG_CONST * sin(0) */ + Step = 0; TargetAngle = FIXED(28.027); CurrAngle = 0; - for (Step = 0; Step < 12; Step++) { + while(Step < 12) { var NewX; if (TargetAngle > CurrAngle) { NewX = X - (Y >> Step); @@ -75,6 +76,7 @@ X = NewX; CurrAngle -= Angles[Step]; } + Step++; } } Chrome: 103.4ms IE: 190.0ms So, my initial conclusions were wrong. Its dead code analysis is either incredibly narrow, or it was hand-crafted to optimize out that part of the benchmark. Either way it's rubbish.

moconnor 16y ago
The claim that IE is legit hinges on the diffs causing a valid bug in IE's optimization code (e.g. deadcode inside deadcode prevents the latter being optimized out), versus foul play in the engine (a hard-coded case for this benchmark). Can you find any variation on the benchmark code that still allows IE to optimize it, or does it only optimize the exact form of the code used in the benchmark suite?
- cheald 16y ago
  I changed variable names and declaration order, the number of loops in that inner for loop, and other such things that could possibly change the bytecode (to what effect, I don't know - I'm not a JS VM engineer, obviously) without changing the operations actually performed. I don't have an explanation for this, though (maybe variable initialization counts as "run up until here"?): Runs fast (11ms): --- tests/sunspider-0.9.1/math-cordic.js 2010-11-17 00:55:29.000000000 -0700 +++ tests/sunspider-0.9.1-deadcode/math-cordic.js 2010-11-17 01:42:43.000000000 -0700 @@ -56,12 +56,14 @@ var TargetAngle; var CurrAngle; var Step; + var foo; X = FIXED(AG_CONST); /* AG_CONST * cos(0) */ Y = 0; /* AG_CONST * sin(0) */ TargetAngle = FIXED(28.027); CurrAngle = 0; + foo = 1; for (Step = 0; Step < 12; Step++) { var NewX; if (TargetAngle > CurrAngle) { But if I assign foo after the for loop, it runs slow: --- tests/sunspider-0.9.1/math-cordic.js 2010-11-17 00:55:29.000000000 -0700 +++ tests/sunspider-0.9.1-deadcode/math-cordic.js 2010-11-17 01:43:20.000000000 -0700 @@ -56,6 +56,7 @@ var TargetAngle; var CurrAngle; var Step; + var foo; X = FIXED(AG_CONST); /* AG_CONST * cos(0) */ Y = 0; /* AG_CONST * sin(0) */ @@ -76,6 +77,7 @@ CurrAngle -= Angles[Step]; } } + foo = 1; } If I assign foo inside the for loop, it runs fast: --- tests/sunspider-0.9.1/math-cordic.js 2010-11-17 00:55:29.000000000 -0700 +++ tests/sunspider-0.9.1-deadcode/math-cordic.js 2010-11-17 01:44:41.000000000 -0700 @@ -56,6 +56,7 @@ var TargetAngle; var CurrAngle; var Step; + var foo; X = FIXED(AG_CONST); /* AG_CONST * cos(0) */ Y = 0; /* AG_CONST * sin(0) */ @@ -63,6 +64,7 @@ TargetAngle = FIXED(28.027); CurrAngle = 0; for (Step = 0; Step < 12; Step++) { + foo = 1; var NewX; if (TargetAngle > CurrAngle) { NewX = X - (Y >> Step); Color me boggled.

detcader 16y ago

What if it just means that IE9 takes 21 more milliseconds to complete a "real" test instead of the "optimized" one?</silly question>

lty13800 16y ago

um.. Are you using the right version of IE 9? The current version is IE 9 PP7 (1.9.8023.6000), which produced a Sunspider result of 216ms. You are testing the beta version of IE 9, which was released in back in September. Of course the old version of IE 9 will have a much slower speed. By the way the Sunspider result for IE 9 beta was about 340 ms