3 ms·

>> Did Consumer Reports indicate in any previous reviews that this is an aspect of their methodology? Probably not. >> It's my understanding that this aspect

>> Did Consumer Reports indicate in any previous reviews that this is an aspect of their methodology? Probably not. >> It's my understanding that this aspect is just now being revealed. That's probably because nobody cared to ask before. The new Macbook Pros were already under a microscope even before Consumer Reports did their tests. Any less than stellar review from CR was going to get lots of eyeballs. In any case, the big reveal is not that caching was disabled, it's that there was a bug on Apple's side that is the likely root cause, and I'm guessing that bug didn't exist in the previous versions of Safari that were used to test the previous Macbook Pros that got the "Recommended" badge. CR's response today: http://www.consumerreports.org/apple/apple-releases-fix-to-macbook-pros-in-response-to-consumer-reports-battery-test-results/ http://www.consumerreports.org/apple/apple-releases-fix-to-m... Some key quotes: "We turn off caching as part of Consumer Reports' standard laptop test protocol" -- I can only take that at its word. It doesn't say "updated" standard protocol. "At Consumer Reports, we test every laptop from every manufacturer in a comparable way. Because people use laptops differently and because their usage can vary from day to day, our battery tests are not designed to be a direct simulation of a consumer’s experience. Rather, we look to control as many variables as possible, then perform a test that gives potential users a reasonable expectation of battery life when a computer’s processors, screen, memory, and antennas are under a light to moderate workload. This test has served as a good proxy for battery life on the hundreds of laptops in our ratings." They don't review hundreds of laptops per year, that's spread out over multiple years. So I don't think it's a huge leap of faith to assume that they turned off the cache on prior Macbook Pro models. FWIW, John Gruber does not "think it’s fair to say that disabling the caches is unfair or a flawed method" -- which says a lot.

intopieces 10y ago
I'm not sure, then, why the lack of scrutiny in previous years is valid criticism; nobody shines a lamp in her doctor's eyes when she gives her a clean bill of health. Of course an unusual review will invite unusual scrutiny. I just wish CR had been more open with the exact methodology from the start. What other settings do they activate that vary from a normal user's? They want us to take their word that their method sufficiently emulates light to moderate usage, but my own experience has been vastly different from that review.
- slantyyz 10y ago
  Put yourself in CR's shoes. You can't do every test of every laptop using the popular sites, because content payloads on various sites change over time. You won't get apples to apples comparison tests. The easiest way to do a repeatable browser test is to turn caching off on the browsers and hit a few web pages that never change and that you control on your own server. By repeating that test over and over again, it simulates hitting new sites even though you're hitting the same pages. If I was in the business of doing laptop reviews, that's how I'd do it. -- edit: This blog entry by Marco Arment on the topic is worth reading: https://marco.org/2017/01/10/cr-mbp-battery-update https://marco.org/2017/01/10/cr-mbp-battery-update

intopieces 10y ago

I'm not sure, then, why the lack of scrutiny in previous years is valid criticism; nobody shines a lamp in her doctor's eyes when she gives her a clean bill of health. Of course an unusual review will invite unusual scrutiny. I just wish CR had been more open with the exact methodology from the start. What other settings do they activate that vary from a normal user's? They want us to take their word that their method sufficiently emulates light to moderate usage, but my own experience has been vastly different from that review.

slantyyz 10y ago
Put yourself in CR's shoes. You can't do every test of every laptop using the popular sites, because content payloads on various sites change over time. You won't get apples to apples comparison tests. The easiest way to do a repeatable browser test is to turn caching off on the browsers and hit a few web pages that never change and that you control on your own server. By repeating that test over and over again, it simulates hitting new sites even though you're hitting the same pages. If I was in the business of doing laptop reviews, that's how I'd do it. -- edit: This blog entry by Marco Arment on the topic is worth reading: https://marco.org/2017/01/10/cr-mbp-battery-update https://marco.org/2017/01/10/cr-mbp-battery-update