5 ms·

Summary: Consumer Reports turned off Safari's page cache to ensure the browsers download fresh content. This hurt performance a lot, presumably because Apple ha

Summary: Consumer Reports turned off Safari's page cache to ensure the browsers download fresh content. This hurt performance a lot, presumably because Apple hadn't optimised this usage pattern. Apple has released an update which fixes the "bug". My opinion: Disabling the cache as Consumer Reports has done is an unrealistic testing pattern. Since Consumer Reports run their tests from local servers [1] they have a better option available. They can use server-generated content to control how much content is served from cache and how much is fresh. They can make the workload 0%–100% fresh by using the servers to generate unique ids for URLs that they want to be served fresh or reusing URLs if they want them to maybe be in the cache. 1: http://www.consumerreports.org/apple/apple-releases-fix-to-macbook-pros-in-response-to-consumer-reports-battery-test-results/ http://www.consumerreports.org/apple/apple-releases-fix-to-m...: EDIT: To be clear, I'm not saying the Consumer Reports didn't find something interesting here, I'm just saying they could make a better test for the future.

1_2__3 10y ago
If a laptop battery capacity is cut in half because you turn off caching in a browser that is important information to know, as it suggests the battery times are the result of carefully sterilized conditions that don't withstand the test of real world use. They didn't disable power management or something, they turned off caching in a single app. Come on.
- sdegutis 10y ago
  That's not true. The vast majority of people don't turn off caching. The default caching behavior Safari provides is probably tailored to real-life usage. I could be wrong, but I'm assuming Apple isn't that dense.
  - richdougherty 10y ago
    I agree, real world usage is a mix of cached and non-cached. The only time I turn of caching is when I'm debugging a website I'm developing.
- Joky 10y ago
  That's fine, as long as you don't say your benchmark is just "regular browsing". Enabling developer options put you outside of what the vendor really tested and optimized for. Also I find quite dubious your claim about `sterilized conditions that don't withstand the test of real world use` based on this. This PR mess also likely 'justifies' the approach of exposing as little knob as possible to the users.
djrogers 10y ago
Incorrect - the performance hit was from a bug related to that: "the variable battery performance we experienced is a result of a software bug in its Safari web browser that was triggered by our test conditions."
- heisenbit 10y ago
  There was a bug and that seemed to cause variation. But then turning on the cache would invalidate the results making it impossible to compare it with other laptops. So I guess they have to get the new version of Safari before doing a re-run.
- richdougherty 10y ago
  I said two separate things: a) That Safari's performance hadn't been tuned for when caching was disabled. b) That, in general, it's more realistic to test a mixed cached/uncached workload. Which bit was incorrect?
buildbuildbuild 10y ago
Changing a testing approach in my opinion is fine, but should result in the retroactive re-testing of previous models used in any in-article benchmark comparisons.

1_2__3 10y ago

If a laptop battery capacity is cut in half because you turn off caching in a browser that is important information to know, as it suggests the battery times are the result of carefully sterilized conditions that don't withstand the test of real world use. They didn't disable power management or something, they turned off caching in a single app. Come on.

sdegutis 10y ago
That's not true. The vast majority of people don't turn off caching. The default caching behavior Safari provides is probably tailored to real-life usage. I could be wrong, but I'm assuming Apple isn't that dense.
- richdougherty 10y ago
  I agree, real world usage is a mix of cached and non-cached. The only time I turn of caching is when I'm debugging a website I'm developing.
Joky 10y ago
That's fine, as long as you don't say your benchmark is just "regular browsing". Enabling developer options put you outside of what the vendor really tested and optimized for. Also I find quite dubious your claim about `sterilized conditions that don't withstand the test of real world use` based on this. This PR mess also likely 'justifies' the approach of exposing as little knob as possible to the users.

djrogers 10y ago

Incorrect - the performance hit was from a bug related to that: "the variable battery performance we experienced is a result of a software bug in its Safari web browser that was triggered by our test conditions."

heisenbit 10y ago
There was a bug and that seemed to cause variation. But then turning on the cache would invalidate the results making it impossible to compare it with other laptops. So I guess they have to get the new version of Safari before doing a re-run.
richdougherty 10y ago
I said two separate things: a) That Safari's performance hadn't been tuned for when caching was disabled. b) That, in general, it's more realistic to test a mixed cached/uncached workload. Which bit was incorrect?

buildbuildbuild 10y ago

Changing a testing approach in my opinion is fine, but should result in the retroactive re-testing of previous models used in any in-article benchmark comparisons.