6 ms·

Chances are they have a script that hits a couple different websites that represent different archetypes: light/heavy text, light/heavy images, light/heavy js,

Chances are they have a script that hits a couple different websites that represent different archetypes: light/heavy text, light/heavy images, light/heavy js, light/heavy ajax, flash, rendering, dependencies, fonts. Instead of using the cache dozens of times, they refetch the pages (which would must more closely represent someone surfing ye olde interwebs). Seems pretty reasonable to me.

a2tech 10y ago
Thats correct-the funny bit is they were using a hidden developer feature-not one of the 'Developer' drop down features but an actual hidden flag. They also observed huge internal discrepancies with their testing methodology-but instead of reporting that to Apple or trying to figure out why their test was broken CR reported their results as 'facts'. Thats why people and Apple are irritated about this.
- c0nfused 10y ago
  The test is factual. The fact that turning off caching triggered a browser bug is not consumer reports problem nor do they have an obligation to figure out why. They exist to report on existing things even if they have bugs. The idea that battery life for a browsing workload should be tested from cached sites is silly, as is the idea that not using a check box on the menu makes them out to get Apple.
  - Terribledactyl 10y ago
    The browsing workload should be partly cached content and mostly new, like in the real world. Otherwise, what are we doing here?
    - slantyyz 10y ago
      Well, no battery test truly represents real world, so I don't think it's a big deal. The same testing methodology in the past gave the "Recommended" badge for Macbook Pros. Why should we question the methodology only when it produces a result that people don't like? -- edit, just noticed the article I posted this comment on got merged with another by the moderators, so it's a dupe within the overall thread.
  - lucb1e 10y ago
    > The fact that turning off caching triggered a browser bug is not consumer reports problem nor do they have an obligation to figure out why. Agree. Unfortunately this makes the test flawed, since it tries to simulate real-world behavior, but they couldn't know that unless they knew there was a bug in the cache thing. > The test is factual. Only if the report mentioned their testing methodology, i.e. if it mentions that they turn off caching for all laptops to better simulate browsing patterns. I expected Techcrunch to mention this but they neglected to.
  - kstrauser 10y ago
    > They exist to report on existing things even if they have bugs. On one level, sure. They found a legit bug. However, the bug was in a codepath that regular users would never experience. You and I might, but we're not the ones reading Consumer Reports to pick a laptop. My non-technical friends certainly aren't going into hidden settings to turn off browser performance features, but they're the ones who'll see that bad review and worry about it. > The idea that battery life for a browsing workload should be tested from cached sites is silly How many first-time-ever websites do you encounter every day? I guarantee that you don't download a new copy of jQuery on every website you visit, or reload every static asset on your favorite sites that you hit daily.
- criley2 10y ago
  I find it funny that Apple and its users are angry at the company who tested and reported results, not the company which coded and released a browser with bugs. We should aspire to attain such loyalty. I know when I release bugs, I get the blame!

a2tech 10y ago

Thats correct-the funny bit is they were using a hidden developer feature-not one of the 'Developer' drop down features but an actual hidden flag. They also observed huge internal discrepancies with their testing methodology-but instead of reporting that to Apple or trying to figure out why their test was broken CR reported their results as 'facts'. Thats why people and Apple are irritated about this.

c0nfused 10y ago
The test is factual. The fact that turning off caching triggered a browser bug is not consumer reports problem nor do they have an obligation to figure out why. They exist to report on existing things even if they have bugs. The idea that battery life for a browsing workload should be tested from cached sites is silly, as is the idea that not using a check box on the menu makes them out to get Apple.
- Terribledactyl 10y ago
  The browsing workload should be partly cached content and mostly new, like in the real world. Otherwise, what are we doing here?
  - slantyyz 10y ago
    Well, no battery test truly represents real world, so I don't think it's a big deal. The same testing methodology in the past gave the "Recommended" badge for Macbook Pros. Why should we question the methodology only when it produces a result that people don't like? -- edit, just noticed the article I posted this comment on got merged with another by the moderators, so it's a dupe within the overall thread.
- lucb1e 10y ago
  > The fact that turning off caching triggered a browser bug is not consumer reports problem nor do they have an obligation to figure out why. Agree. Unfortunately this makes the test flawed, since it tries to simulate real-world behavior, but they couldn't know that unless they knew there was a bug in the cache thing. > The test is factual. Only if the report mentioned their testing methodology, i.e. if it mentions that they turn off caching for all laptops to better simulate browsing patterns. I expected Techcrunch to mention this but they neglected to.
- kstrauser 10y ago
  > They exist to report on existing things even if they have bugs. On one level, sure. They found a legit bug. However, the bug was in a codepath that regular users would never experience. You and I might, but we're not the ones reading Consumer Reports to pick a laptop. My non-technical friends certainly aren't going into hidden settings to turn off browser performance features, but they're the ones who'll see that bad review and worry about it. > The idea that battery life for a browsing workload should be tested from cached sites is silly How many first-time-ever websites do you encounter every day? I guarantee that you don't download a new copy of jQuery on every website you visit, or reload every static asset on your favorite sites that you hit daily.
criley2 10y ago
I find it funny that Apple and its users are angry at the company who tested and reported results, not the company which coded and released a browser with bugs. We should aspire to attain such loyalty. I know when I release bugs, I get the blame!