13 ms·

Consumer Reports Retesting MacBook Pro Battery, Apple Says Safari Bug to Blame

hueving 10y ago
Does anyone here use consumer reports for general shopping guidance? Curious of the quality.
- acomjean 10y ago
  Generally I've found them to be pretty good with the reviews. The testing is thorough and they explain what to look for when buying something (I've used it for appliances). Its subscription only (they don't do ads), which means they're not beholden to the companies they're reviewing for money (a good thing) They generally make a grid of the items you're looking at with dots indicating the ratings. Its very easy to nagivate and intuitive. You'll have to pay for it (or find someone to give you their old copies, or many libraries have subscriptions).
  - jdmichal 10y ago
    I'm a big fan of the grids they do. Really allows you to pick the things that are most important to you and maximize them.
- rthomas6 10y ago
  I've used them for baby equipment, for mattresses, and for cars. They were spot on with cars, baby car seats and cribs, but their mattress ratings could have been better at the time. Overall, I've found them to be pretty high quality, and most importantly, independent. Their reviews will often differ from more public sources. And I could be mistaken, but I think they don't even accept free products from companies. They buy the products they test.
- no_wave 10y ago
  Not really... most people I know use things like the Wirecutter instead. This became Hacker News fare because lots of people were unhappy with the new MacBook Pros for other reasons.
  - gist 10y ago
    Never heard of that until you just mentioned. But geez it's apart of the New York Times and they earn affiliate fees for things you buy! http://thewirecutter.com/ http://thewirecutter.com/
    - slouch 10y ago
      Wirecutter is one of the best affiliate executions I have ever seen. I've used it to buy multiple things. It was only recently bought by the NYT.
- jxn 10y ago
  In general, they do well, but there are a few categories which they don't seem to "get". They do a good job of trying to remove a lot of barriers to objectivity, even when useful reviews require subjective knowledge. They also try hard to find equivalent, reproducible tests so that they can use them on multiple products, even if they're not exact equivalents. This means, for instance, that they might end up testing ice-braking tires by measuring the stop distance in an indoor ice rink where they can control the conditions... that makes it easy to measure the tires against other tires, but it's not real-world. They also purchase their products, rather than accepting them from the companies, which helps increase the odds that they're getting the same things off the shelf you do. Again, they seem to be clueless about a few categories, though, such as certain electronics. Just look at the ratings for the Google Home and Amazon Alexa--very poor--because they get lumped in with wireless speakers where sound quality is the utmost importance.
- slouch 10y ago
  My mother and sister are very huge fans of Consumer Reports.
- DavidAdams 10y ago
  Consumer Reports is pretty good for household products that they have expertise in. For example, I'd trust their dishwasher or toaster recommendation. However, when they test something that's outside of their sphere of expertise, they can miss the mark by failing to focus on criteria that are known to be important by real experts. For example, from time to time they'll review bicycles. Anyone who really knows about bikes will be amused/exasperated by the aspects of the bikes being tested that they make their judgement on. They've been testing computers for long enough that I'm sure they're not completely clueless, but I think their judgement criteria are based more on a casual computer-as-appliance home user, and a power user might feel that they focus on superficial things.
  - devopsproject 10y ago
    > a power user might feel that they focus on superficial things battery life affects everyone and they had the competency and integrity to call apple out multiple times
- tomcam 10y ago
  I found them useless for high-end dishwashers (both Mieles they recommended worked poorly), audio equipment, and high-end vacuums. I specify high-end because I am very willing to pay top dollar for appliances that will last many years, not because I'm a snob.
  - slantyyz 10y ago
    >> I found them useless for high-end dishwashers (both Mieles they recommended worked poorly) That seems to apply to all of their highly rated dishwashers -- when you read the CR user comments, most of them got a lot of horrible user ratings. I don't know if it's the dishwasher segment, but it seems like dishwashers in general have gone downhill. I was also going to buy a higher end dishwasher but it was so hard to choose because of the negative user comments that I ended up getting a lower priced one that got decent reliability ratings. I didn't want to get a Bosch or Miele after reading the user reviews on CR. Having said all of that, if you've been using dishwashers since the 80s, it turns out your old habits can result in bad washes. I was used to scraping my dishes clean, but according to my research, you actually need your dishes to be soiled enough so the newer detergent pods can clean them properly. I was lucky I found that out or I'd be getting inconsistent wash results.
    - DashRattlesnake 10y ago
      Also, it appears newer ones have less effective drying hardware which forces them to rely on "rinse aid," so your choice is either still-wet dishes or dishes with residue.
      - slantyyz 10y ago
        When I was researching, I found that the SweetHome (Wirecutter's sister site) had very useful information on this stuff. I originally had my eyes set on a European style dishwasher which dries very differently from an American style dishwasher. I ended up going with an American style dishwasher which has a heating element for drying (compared to passive drying in European models).
    - 10y ago
mmastrac 10y ago
I guess this really just confirms what I've suspected: the only way to get decent battery life out of a Mac is if you are literally using it for nothing but simple web browsing or watching a movie and doing nothing else. Non-Apple apps are death for battery life. I think I'd be lucky to get two hours of productive development time on my laptops. 1.5 hours is probably closer to reality. I have two BatteryBox devices that add about 45 minutes to 1 hour each to runtime and it's impossible to get productive work done without them. Perhaps Consumer Reports should try testing battery life by running other apps?
- geerlingguy 10y ago
  I put my MacBook Pros through a grueling battery test (basically build and destroy a VM successively, sustaining 100%+ CPU, tons of disk I/O, and maxing out WiFi bandwidth intermittently) for my blog post elsewhere on HN. Worst-case, the 13" Pro with Touch Bar gets 3.5 hours. The Function Key model gets 4 hours. And the 2015 Pro gets 5 hours. In light usage (e.g. while writing said blog post, and watching an occasional HTML5 YouTube video, and listening to some music in iTunes through AirPods), I can get 8-10 hours without issue. 1-2 hours sounds like you may have an app that's more buggy than the worst that I'd allow to run on my laptop... Edit: Link to said test script: https://github.com/geerlingguy/macbook-pro-battery-test https://github.com/geerlingguy/macbook-pro-battery-test
  - caycep 10y ago
    pretty much. I have a 2 year old MBP 13" and it lasted most of a 9 hr flight recently.
- zitterbewegung 10y ago
  Can you recommend a company that has good battery life for high intensity workloads? I'm genuinely curious.
  - mmastrac 10y ago
    No, as I've been in the Apple ecosystem for about 10 years and haven't shopped around. I used to have a Dell beast of a laptop back then that was 1) very heavy, 2) very hot and 3) would last for hours of development time. I would actually trade whatever weight would be necessary to get my useful portable development time up to 3-4 hours.
cddotdotslash 10y ago
So Consumer Reports had Safari Developer Mode enabled which forced full page loads (plus some other bug). And Apple claims "this is not a setting used by customers and does not reflect real-world usage." While I agree most people wouldn't have that option enabled, I would argue that the average user is somewhere between the extremes. For example, most users are not disabling cache and browsing to 20 MB webpages and clicking refresh hundreds of times. But at the same time, neither are users leaving cache on and reloading the same page hundreds of times; they're browsing from site to site, refreshing Facebook feeds, etc. If Apple can make this statement, I'd be curious as to what tests they are performing. I'm almost certain the battery specs released by Apple are not indicative of how average users would use the MacBook (i.e. brigtness 10%, no movement, WiFi off, all background apps disabled, etc.). EDIT: as a reply mentions, Apple does provide (some) conditions used for their tests. However I'd still imagine that the conditions they do choose are the "most optimal" (for example - who do you know that only browses 25 sites in a single charge?)
- IBM 10y ago
  This is Apple's battery life test: >The wireless web test measures battery life by wirelessly browsing 25 popular websites with display brightness set to 12 clicks from bottom or 75 percent. The iTunes movie playback test measures battery life by playing back HD 1080p content with display brightness set to 12 clicks from bottom or 75 percent. The standby test measures battery life by allowing a system, connected to a wireless network and signed in to an iCloud account, to enter standby mode with Safari and Mail applications launched and all system settings left at default. Battery life varies by use and configuration. http://www.apple.com/macbook-pro/specs/ http://www.apple.com/macbook-pro/specs/
  - r00fus 10y ago
    Thing is, with dynamic brightness (Macs have long been better at that than PCs), it's often the case that 75% brightness or less is used - at night for me it's more like 25% bright.
- r00fus 10y ago
  So the test if flawed. You can't simply disable cache and call it a good test. Do they do this on other browsers? Browse the same site over and over while disabling cache? Why did this not cause a problem when they ran Chrome (did they also disable cache there)? The CR tests raise more concerns about how CR runs their tests (and specifically how they run this test) as opposed to the MacbookPro.
richdougherty 10y ago
Summary: Consumer Reports turned off Safari's page cache to ensure the browsers download fresh content. This hurt performance a lot, presumably because Apple hadn't optimised this usage pattern. Apple has released an update which fixes the "bug". My opinion: Disabling the cache as Consumer Reports has done is an unrealistic testing pattern. Since Consumer Reports run their tests from local servers [1] they have a better option available. They can use server-generated content to control how much content is served from cache and how much is fresh. They can make the workload 0%–100% fresh by using the servers to generate unique ids for URLs that they want to be served fresh or reusing URLs if they want them to maybe be in the cache. 1: http://www.consumerreports.org/apple/apple-releases-fix-to-macbook-pros-in-response-to-consumer-reports-battery-test-results/ http://www.consumerreports.org/apple/apple-releases-fix-to-m...: EDIT: To be clear, I'm not saying the Consumer Reports didn't find something interesting here, I'm just saying they could make a better test for the future.
- 1_2__3 10y ago
  If a laptop battery capacity is cut in half because you turn off caching in a browser that is important information to know, as it suggests the battery times are the result of carefully sterilized conditions that don't withstand the test of real world use. They didn't disable power management or something, they turned off caching in a single app. Come on.
  - sdegutis 10y ago
    That's not true. The vast majority of people don't turn off caching. The default caching behavior Safari provides is probably tailored to real-life usage. I could be wrong, but I'm assuming Apple isn't that dense.
    - richdougherty 10y ago
      I agree, real world usage is a mix of cached and non-cached. The only time I turn of caching is when I'm debugging a website I'm developing.
  - 10y ago
lynndylanhurley 10y ago
Here's a question - if the results were due to an obscure Safari bug, then why did Apple remove the battery life estimate with the latest OS X update? I've been getting 2-3 hours max out of my new 15" 2016 MBP and I don't even use Safari. I'm just running webpack, vim and Chrome.
- pat2man 10y ago
  Chrome is pretty bad for battery life.
- caconym_ 10y ago
  I think they removed it because the battery life of the new MBPs has been getting a lot of negative attention, deserved or not, and in general people don't seem to be capable of understanding why the estimate would change for different workloads. Maybe the best PR move was just to remove it altogether. I wish they hadn't. It's not the same machine, but I routinely get > 10 hours on my 2016 12" Macbook with a workflow that includes moderate to heavy browsing in Opera, coding in Tmux/Vim, and various compilers and interpreters invoked periodically. Have you had a look in Activity Monitor to see where all your power is going? I'm using two non-Apple apps as the core of my workflow (Opera and Alacritty for terminal: https://github.com/jwilm/alacritty https://github.com/jwilm/alacritty) but my battery life is still excellent. However, I chose them carefully on the basis of their efficiency. Things written in Javascript seem to be real resource hogs (Atom comes to mind), and Chrome is notorious for inefficiency in memory, at least.
- rrock 10y ago
  Click on the battery icon in your menu bar, and see if there are any programs using "Significant Energy". Some programs will switch on the discrete graphics card without needing it, for example.
- acchow 10y ago
  I'd suggest switching to Safari. I easily doubled my battery life while on El Capitan doing that. Sierra has wrecked my battery life tho...
EnFinlay 10y ago
Chances are they have a script that hits a couple different websites that represent different archetypes: light/heavy text, light/heavy images, light/heavy js, light/heavy ajax, flash, rendering, dependencies, fonts. Instead of using the cache dozens of times, they refetch the pages (which would must more closely represent someone surfing ye olde interwebs). Seems pretty reasonable to me.
- a2tech 10y ago
  Thats correct-the funny bit is they were using a hidden developer feature-not one of the 'Developer' drop down features but an actual hidden flag. They also observed huge internal discrepancies with their testing methodology-but instead of reporting that to Apple or trying to figure out why their test was broken CR reported their results as 'facts'. Thats why people and Apple are irritated about this.
  - c0nfused 10y ago
    The test is factual. The fact that turning off caching triggered a browser bug is not consumer reports problem nor do they have an obligation to figure out why. They exist to report on existing things even if they have bugs. The idea that battery life for a browsing workload should be tested from cached sites is silly, as is the idea that not using a check box on the menu makes them out to get Apple.
    - Terribledactyl 10y ago
      The browsing workload should be partly cached content and mostly new, like in the real world. Otherwise, what are we doing here?
      - slantyyz 10y ago
        Well, no battery test truly represents real world, so I don't think it's a big deal. The same testing methodology in the past gave the "Recommended" badge for Macbook Pros. Why should we question the methodology only when it produces a result that people don't like? -- edit, just noticed the article I posted this comment on got merged with another by the moderators, so it's a dupe within the overall thread.
Will_Do 10y ago
I guess the way to benchmark browsing to get most realistic numbers would be to record normal browser behavior of a decently-sized sample of people (say 1000) and automatically repeat the behavior of a representative subsample (10-50 may be enough). This would take forever and cost quite a lot of money, however, so the much more reasonable solution is to simply turn off the cache for everything. I don't agree with Apple or Techcrunch that CR's methodology is wrong -- pretty much anyone would make that same decision because the alternative is too much work for too little reward.
- Johnny555 10y ago
  Websites change over time -- they'd have to download a static copy of the websites and store them locally for repeatable tests (but then it no longer reflects the real world, since websites often tailor themselves for each user). And since browsing activity changes over time too, they'd have to regularly update the test, which means that tests are not comparable over time.
  - pfranz 10y ago
    I don't think that's unreasonable to address. For CPU and GPU benchmarks they often run BenchMark2014 so you can see improvements compared to previous years and BenchMark2017 to see the latest and greatest because hardware architecture, game engine design, and graphics APIs change. You just need to figure out how best to present that to the user.
  - slantyyz 10y ago
    IIRC, the sites that are visited during the test are off a CR server used specifically for the tests, not the Internet at large.
bluthru 10y ago
Consumer Reports's original methodology is dumbfounding.
- Oletros 10y ago
  Why?
  - bluthru 10y ago
    The vast majority of users don't browse with caching disabled.
    - Oletros 10y ago
      The vast majority of users browse different pages. And why it wasn't a problem until now? CR was praised when they acknowledge that Macbooks had almost always the higher battery life
purephase 10y ago
I dunno here. CR's validation around the fact that users will often visit different sites seems on point to me. Some users visit the same sites over and over, true. But, a more accurate browsing pattern would be different sites, new content etc. A more in-depth detail on how the browsing is done (automation) might be relevant. If they hit the same 5 sites 1000 times vs. 1000 sites 5 times would alter whether they disable cache or not. I'd say the former makes sense, not the latter.
Oletros 10y ago
The Techcrunch article is very bad. CR has disabled the cache for all of their tests in all the computers for years. And now is bad because there was a bug in Safari?
tener 10y ago
They turn off the browser cache for every notebook they test. If their methodology were flawed because of this they would have noticed that long time ago.
sailfast 10y ago
Speaking personally, I would want any stress test to be the MOST stressful possible across all laptops for the baseline. Turn off cache (how often do you have the console open), run a VM, sit on a Hangout, all at the same time. Make the fans whirr hard. If you want an "econo" benchmark go right ahead but I don't know why one would criticize Consumer Reports for being hard on a laptop browser. (It obviously makes sense for hardware makers to dispute the technique as it's harder to hit "X Hours" markers that are tested)
- lucb1e 10y ago
  I can see where you're coming from but I disagree. In that case you are testing the battery hardware, while the software actually makes a big difference. Say you have two phones: one with crap software, and one from a company that spends time on writing proper, efficient software. If you run a stress benchmark, they will both be running at 100% CPU all the time and you'll only be measuring how much mAh is in the battery divided by how much power the CPU draws. All of this is information is already available in the product specification. The only reason my Galaxy Note 2's battery life is any good (still the original battery which is, what, 5 years old now?) is because I did software tweaks to keep it from holding wake locks when apps are inactive.
  - sailfast 10y ago
    That makes sense and you're right, if all you're doing is a head-on stress test then just let the math do the talking. I think testing heavy use is probably different than a straight up stress test in a number of ways that are more applicable to my daily use and more useful. It's good to know if Safari's caching strategy saves you battery, but as a non-Safari user that often does not refresh from cache I'm not sure I would appreciate buying a laptop advertising X battery and finding it to have changed significantly or be caveated by lots of required settings.
    - lucb1e 10y ago
      Alright, that's a good point.
deleted 10y ago
[deleted]
DoodleBuggy 10y ago
"You're testing it wrong." It couldn't be that the battery is actually quite literally smaller. No, it is the tester and tests fault.
revelation 10y ago
It doesn't really matter what the test is as long as they run the same one across the devices.
tedivm 10y ago
Consumer Reports did nothing different with the MacBook Pro tests than they do with their other tests. So using the benchmarks as a comparison point between Apple and others, such as HP, should still be completely valid.
- ralfd 10y ago
  Read Apples statement to TechCrunch in the article.
leejoramo 10y ago
The issue is more complex. Turning off the cache also triggered a bug. http://www.imore.com/consumer-reports-fails-earn-macbook-pro-credibility http://www.imore.com/consumer-reports-fails-earn-macbook-pro... > Consumer Reports uses a hidden Safari setting for developing web sites which turns off the browser cache. This is not a setting used by customers and does not reflect real-world usage. Their use of this developer setting also triggered an obscure and intermittent bug reloading icons which created inconsistent results in their lab.
- mturmon 10y ago
  This is a good point, which seems to be ignored by other commenters nearby. Additionally, this seems to be a case where CR (a venerable publication with many good qualities) used results in a home-brew single-point test to characterize a system's quality. This is not an isolated problem with this particular review. There's a lot of snake oil in audio systems, but CR reviews of audio equipment were a joke for many years because of over-reliance on specific lab measurements. CR has had issues over the years with mistaken testing of child safety seats, pet food, and other products over the years (see their wiki page). This caveat is apparent after subscribing for a few years.

Does anyone here use consumer reports for general shopping guidance? Curious of the quality.

acomjean 10y ago
Generally I've found them to be pretty good with the reviews. The testing is thorough and they explain what to look for when buying something (I've used it for appliances). Its subscription only (they don't do ads), which means they're not beholden to the companies they're reviewing for money (a good thing) They generally make a grid of the items you're looking at with dots indicating the ratings. Its very easy to nagivate and intuitive. You'll have to pay for it (or find someone to give you their old copies, or many libraries have subscriptions).
- jdmichal 10y ago
  I'm a big fan of the grids they do. Really allows you to pick the things that are most important to you and maximize them.
rthomas6 10y ago
I've used them for baby equipment, for mattresses, and for cars. They were spot on with cars, baby car seats and cribs, but their mattress ratings could have been better at the time. Overall, I've found them to be pretty high quality, and most importantly, independent. Their reviews will often differ from more public sources. And I could be mistaken, but I think they don't even accept free products from companies. They buy the products they test.
no_wave 10y ago
Not really... most people I know use things like the Wirecutter instead. This became Hacker News fare because lots of people were unhappy with the new MacBook Pros for other reasons.
- gist 10y ago
  Never heard of that until you just mentioned. But geez it's apart of the New York Times and they earn affiliate fees for things you buy! http://thewirecutter.com/ http://thewirecutter.com/
  - slouch 10y ago
    Wirecutter is one of the best affiliate executions I have ever seen. I've used it to buy multiple things. It was only recently bought by the NYT.
jxn 10y ago
In general, they do well, but there are a few categories which they don't seem to "get". They do a good job of trying to remove a lot of barriers to objectivity, even when useful reviews require subjective knowledge. They also try hard to find equivalent, reproducible tests so that they can use them on multiple products, even if they're not exact equivalents. This means, for instance, that they might end up testing ice-braking tires by measuring the stop distance in an indoor ice rink where they can control the conditions... that makes it easy to measure the tires against other tires, but it's not real-world. They also purchase their products, rather than accepting them from the companies, which helps increase the odds that they're getting the same things off the shelf you do. Again, they seem to be clueless about a few categories, though, such as certain electronics. Just look at the ratings for the Google Home and Amazon Alexa--very poor--because they get lumped in with wireless speakers where sound quality is the utmost importance.
slouch 10y ago
My mother and sister are very huge fans of Consumer Reports.
DavidAdams 10y ago
Consumer Reports is pretty good for household products that they have expertise in. For example, I'd trust their dishwasher or toaster recommendation. However, when they test something that's outside of their sphere of expertise, they can miss the mark by failing to focus on criteria that are known to be important by real experts. For example, from time to time they'll review bicycles. Anyone who really knows about bikes will be amused/exasperated by the aspects of the bikes being tested that they make their judgement on. They've been testing computers for long enough that I'm sure they're not completely clueless, but I think their judgement criteria are based more on a casual computer-as-appliance home user, and a power user might feel that they focus on superficial things.
- devopsproject 10y ago
  > a power user might feel that they focus on superficial things battery life affects everyone and they had the competency and integrity to call apple out multiple times
tomcam 10y ago
I found them useless for high-end dishwashers (both Mieles they recommended worked poorly), audio equipment, and high-end vacuums. I specify high-end because I am very willing to pay top dollar for appliances that will last many years, not because I'm a snob.
- slantyyz 10y ago
  >> I found them useless for high-end dishwashers (both Mieles they recommended worked poorly) That seems to apply to all of their highly rated dishwashers -- when you read the CR user comments, most of them got a lot of horrible user ratings. I don't know if it's the dishwasher segment, but it seems like dishwashers in general have gone downhill. I was also going to buy a higher end dishwasher but it was so hard to choose because of the negative user comments that I ended up getting a lower priced one that got decent reliability ratings. I didn't want to get a Bosch or Miele after reading the user reviews on CR. Having said all of that, if you've been using dishwashers since the 80s, it turns out your old habits can result in bad washes. I was used to scraping my dishes clean, but according to my research, you actually need your dishes to be soiled enough so the newer detergent pods can clean them properly. I was lucky I found that out or I'd be getting inconsistent wash results.
  - DashRattlesnake 10y ago
    Also, it appears newer ones have less effective drying hardware which forces them to rely on "rinse aid," so your choice is either still-wet dishes or dishes with residue.
    - slantyyz 10y ago
      When I was researching, I found that the SweetHome (Wirecutter's sister site) had very useful information on this stuff. I originally had my eyes set on a European style dishwasher which dries very differently from an American style dishwasher. I ended up going with an American style dishwasher which has a heating element for drying (compared to passive drying in European models).
  - 10y ago

I guess this really just confirms what I've suspected: the only way to get decent battery life out of a Mac is if you are literally using it for nothing but simple web browsing or watching a movie and doing nothing else. Non-Apple apps are death for battery life. I think I'd be lucky to get two hours of productive development time on my laptops. 1.5 hours is probably closer to reality. I have two BatteryBox devices that add about 45 minutes to 1 hour each to runtime and it's impossible to get productive work done without them. Perhaps Consumer Reports should try testing battery life by running other apps?

geerlingguy 10y ago
I put my MacBook Pros through a grueling battery test (basically build and destroy a VM successively, sustaining 100%+ CPU, tons of disk I/O, and maxing out WiFi bandwidth intermittently) for my blog post elsewhere on HN. Worst-case, the 13" Pro with Touch Bar gets 3.5 hours. The Function Key model gets 4 hours. And the 2015 Pro gets 5 hours. In light usage (e.g. while writing said blog post, and watching an occasional HTML5 YouTube video, and listening to some music in iTunes through AirPods), I can get 8-10 hours without issue. 1-2 hours sounds like you may have an app that's more buggy than the worst that I'd allow to run on my laptop... Edit: Link to said test script: https://github.com/geerlingguy/macbook-pro-battery-test https://github.com/geerlingguy/macbook-pro-battery-test
- caycep 10y ago
  pretty much. I have a 2 year old MBP 13" and it lasted most of a 9 hr flight recently.
zitterbewegung 10y ago
Can you recommend a company that has good battery life for high intensity workloads? I'm genuinely curious.
- mmastrac 10y ago
  No, as I've been in the Apple ecosystem for about 10 years and haven't shopped around. I used to have a Dell beast of a laptop back then that was 1) very heavy, 2) very hot and 3) would last for hours of development time. I would actually trade whatever weight would be necessary to get my useful portable development time up to 3-4 hours.

cddotdotslash 10y ago

So Consumer Reports had Safari Developer Mode enabled which forced full page loads (plus some other bug). And Apple claims "this is not a setting used by customers and does not reflect real-world usage." While I agree most people wouldn't have that option enabled, I would argue that the average user is somewhere between the extremes. For example, most users are not disabling cache and browsing to 20 MB webpages and clicking refresh hundreds of times. But at the same time, neither are users leaving cache on and reloading the same page hundreds of times; they're browsing from site to site, refreshing Facebook feeds, etc. If Apple can make this statement, I'd be curious as to what tests they are performing. I'm almost certain the battery specs released by Apple are not indicative of how average users would use the MacBook (i.e. brigtness 10%, no movement, WiFi off, all background apps disabled, etc.). EDIT: as a reply mentions, Apple does provide (some) conditions used for their tests. However I'd still imagine that the conditions they do choose are the "most optimal" (for example - who do you know that only browses 25 sites in a single charge?)

IBM 10y ago
This is Apple's battery life test: >The wireless web test measures battery life by wirelessly browsing 25 popular websites with display brightness set to 12 clicks from bottom or 75 percent. The iTunes movie playback test measures battery life by playing back HD 1080p content with display brightness set to 12 clicks from bottom or 75 percent. The standby test measures battery life by allowing a system, connected to a wireless network and signed in to an iCloud account, to enter standby mode with Safari and Mail applications launched and all system settings left at default. Battery life varies by use and configuration. http://www.apple.com/macbook-pro/specs/ http://www.apple.com/macbook-pro/specs/
- r00fus 10y ago
  Thing is, with dynamic brightness (Macs have long been better at that than PCs), it's often the case that 75% brightness or less is used - at night for me it's more like 25% bright.
r00fus 10y ago
So the test if flawed. You can't simply disable cache and call it a good test. Do they do this on other browsers? Browse the same site over and over while disabling cache? Why did this not cause a problem when they ran Chrome (did they also disable cache there)? The CR tests raise more concerns about how CR runs their tests (and specifically how they run this test) as opposed to the MacbookPro.

richdougherty 10y ago

Summary: Consumer Reports turned off Safari's page cache to ensure the browsers download fresh content. This hurt performance a lot, presumably because Apple hadn't optimised this usage pattern. Apple has released an update which fixes the "bug". My opinion: Disabling the cache as Consumer Reports has done is an unrealistic testing pattern. Since Consumer Reports run their tests from local servers [1] they have a better option available. They can use server-generated content to control how much content is served from cache and how much is fresh. They can make the workload 0%–100% fresh by using the servers to generate unique ids for URLs that they want to be served fresh or reusing URLs if they want them to maybe be in the cache. 1: http://www.consumerreports.org/apple/apple-releases-fix-to-macbook-pros-in-response-to-consumer-reports-battery-test-results/ http://www.consumerreports.org/apple/apple-releases-fix-to-m...: EDIT: To be clear, I'm not saying the Consumer Reports didn't find something interesting here, I'm just saying they could make a better test for the future.

1_2__3 10y ago
If a laptop battery capacity is cut in half because you turn off caching in a browser that is important information to know, as it suggests the battery times are the result of carefully sterilized conditions that don't withstand the test of real world use. They didn't disable power management or something, they turned off caching in a single app. Come on.
- sdegutis 10y ago
  That's not true. The vast majority of people don't turn off caching. The default caching behavior Safari provides is probably tailored to real-life usage. I could be wrong, but I'm assuming Apple isn't that dense.
  - richdougherty 10y ago
    I agree, real world usage is a mix of cached and non-cached. The only time I turn of caching is when I'm debugging a website I'm developing.
- 10y ago

lynndylanhurley 10y ago

Here's a question - if the results were due to an obscure Safari bug, then why did Apple remove the battery life estimate with the latest OS X update? I've been getting 2-3 hours max out of my new 15" 2016 MBP and I don't even use Safari. I'm just running webpack, vim and Chrome.

pat2man 10y ago
Chrome is pretty bad for battery life.
caconym_ 10y ago
I think they removed it because the battery life of the new MBPs has been getting a lot of negative attention, deserved or not, and in general people don't seem to be capable of understanding why the estimate would change for different workloads. Maybe the best PR move was just to remove it altogether. I wish they hadn't. It's not the same machine, but I routinely get > 10 hours on my 2016 12" Macbook with a workflow that includes moderate to heavy browsing in Opera, coding in Tmux/Vim, and various compilers and interpreters invoked periodically. Have you had a look in Activity Monitor to see where all your power is going? I'm using two non-Apple apps as the core of my workflow (Opera and Alacritty for terminal: https://github.com/jwilm/alacritty https://github.com/jwilm/alacritty) but my battery life is still excellent. However, I chose them carefully on the basis of their efficiency. Things written in Javascript seem to be real resource hogs (Atom comes to mind), and Chrome is notorious for inefficiency in memory, at least.
rrock 10y ago
Click on the battery icon in your menu bar, and see if there are any programs using "Significant Energy". Some programs will switch on the discrete graphics card without needing it, for example.
acchow 10y ago
I'd suggest switching to Safari. I easily doubled my battery life while on El Capitan doing that. Sierra has wrecked my battery life tho...

EnFinlay 10y ago

Chances are they have a script that hits a couple different websites that represent different archetypes: light/heavy text, light/heavy images, light/heavy js, light/heavy ajax, flash, rendering, dependencies, fonts. Instead of using the cache dozens of times, they refetch the pages (which would must more closely represent someone surfing ye olde interwebs). Seems pretty reasonable to me.

a2tech 10y ago
Thats correct-the funny bit is they were using a hidden developer feature-not one of the 'Developer' drop down features but an actual hidden flag. They also observed huge internal discrepancies with their testing methodology-but instead of reporting that to Apple or trying to figure out why their test was broken CR reported their results as 'facts'. Thats why people and Apple are irritated about this.
- c0nfused 10y ago
  The test is factual. The fact that turning off caching triggered a browser bug is not consumer reports problem nor do they have an obligation to figure out why. They exist to report on existing things even if they have bugs. The idea that battery life for a browsing workload should be tested from cached sites is silly, as is the idea that not using a check box on the menu makes them out to get Apple.
  - Terribledactyl 10y ago
    The browsing workload should be partly cached content and mostly new, like in the real world. Otherwise, what are we doing here?
    - slantyyz 10y ago
      Well, no battery test truly represents real world, so I don't think it's a big deal. The same testing methodology in the past gave the "Recommended" badge for Macbook Pros. Why should we question the methodology only when it produces a result that people don't like? -- edit, just noticed the article I posted this comment on got merged with another by the moderators, so it's a dupe within the overall thread.

Will_Do 10y ago

I guess the way to benchmark browsing to get most realistic numbers would be to record normal browser behavior of a decently-sized sample of people (say 1000) and automatically repeat the behavior of a representative subsample (10-50 may be enough). This would take forever and cost quite a lot of money, however, so the much more reasonable solution is to simply turn off the cache for everything. I don't agree with Apple or Techcrunch that CR's methodology is wrong -- pretty much anyone would make that same decision because the alternative is too much work for too little reward.

Johnny555 10y ago
Websites change over time -- they'd have to download a static copy of the websites and store them locally for repeatable tests (but then it no longer reflects the real world, since websites often tailor themselves for each user). And since browsing activity changes over time too, they'd have to regularly update the test, which means that tests are not comparable over time.
- pfranz 10y ago
  I don't think that's unreasonable to address. For CPU and GPU benchmarks they often run BenchMark2014 so you can see improvements compared to previous years and BenchMark2017 to see the latest and greatest because hardware architecture, game engine design, and graphics APIs change. You just need to figure out how best to present that to the user.
- slantyyz 10y ago
  IIRC, the sites that are visited during the test are off a CR server used specifically for the tests, not the Internet at large.

bluthru 10y ago

Consumer Reports's original methodology is dumbfounding.

Oletros 10y ago
Why?
- bluthru 10y ago
  The vast majority of users don't browse with caching disabled.
  - Oletros 10y ago
    The vast majority of users browse different pages. And why it wasn't a problem until now? CR was praised when they acknowledge that Macbooks had almost always the higher battery life

purephase 10y ago

I dunno here. CR's validation around the fact that users will often visit different sites seems on point to me. Some users visit the same sites over and over, true. But, a more accurate browsing pattern would be different sites, new content etc. A more in-depth detail on how the browsing is done (automation) might be relevant. If they hit the same 5 sites 1000 times vs. 1000 sites 5 times would alter whether they disable cache or not. I'd say the former makes sense, not the latter.

Oletros 10y ago

The Techcrunch article is very bad. CR has disabled the cache for all of their tests in all the computers for years. And now is bad because there was a bug in Safari?

tener 10y ago

They turn off the browser cache for every notebook they test. If their methodology were flawed because of this they would have noticed that long time ago.

sailfast 10y ago

Speaking personally, I would want any stress test to be the MOST stressful possible across all laptops for the baseline. Turn off cache (how often do you have the console open), run a VM, sit on a Hangout, all at the same time. Make the fans whirr hard. If you want an "econo" benchmark go right ahead but I don't know why one would criticize Consumer Reports for being hard on a laptop browser. (It obviously makes sense for hardware makers to dispute the technique as it's harder to hit "X Hours" markers that are tested)

lucb1e 10y ago
I can see where you're coming from but I disagree. In that case you are testing the battery hardware, while the software actually makes a big difference. Say you have two phones: one with crap software, and one from a company that spends time on writing proper, efficient software. If you run a stress benchmark, they will both be running at 100% CPU all the time and you'll only be measuring how much mAh is in the battery divided by how much power the CPU draws. All of this is information is already available in the product specification. The only reason my Galaxy Note 2's battery life is any good (still the original battery which is, what, 5 years old now?) is because I did software tweaks to keep it from holding wake locks when apps are inactive.
- sailfast 10y ago
  That makes sense and you're right, if all you're doing is a head-on stress test then just let the math do the talking. I think testing heavy use is probably different than a straight up stress test in a number of ways that are more applicable to my daily use and more useful. It's good to know if Safari's caching strategy saves you battery, but as a non-Safari user that often does not refresh from cache I'm not sure I would appreciate buying a laptop advertising X battery and finding it to have changed significantly or be caveated by lots of required settings.
  - lucb1e 10y ago
    Alright, that's a good point.

deleted 10y ago

[deleted]

DoodleBuggy 10y ago

"You're testing it wrong." It couldn't be that the battery is actually quite literally smaller. No, it is the tester and tests fault.

revelation 10y ago

It doesn't really matter what the test is as long as they run the same one across the devices.

tedivm 10y ago

Consumer Reports did nothing different with the MacBook Pro tests than they do with their other tests. So using the benchmarks as a comparison point between Apple and others, such as HP, should still be completely valid.

ralfd 10y ago
Read Apples statement to TechCrunch in the article.

leejoramo 10y ago

The issue is more complex. Turning off the cache also triggered a bug. http://www.imore.com/consumer-reports-fails-earn-macbook-pro-credibility http://www.imore.com/consumer-reports-fails-earn-macbook-pro... > Consumer Reports uses a hidden Safari setting for developing web sites which turns off the browser cache. This is not a setting used by customers and does not reflect real-world usage. Their use of this developer setting also triggered an obscure and intermittent bug reloading icons which created inconsistent results in their lab.

mturmon 10y ago
This is a good point, which seems to be ignored by other commenters nearby. Additionally, this seems to be a case where CR (a venerable publication with many good qualities) used results in a home-brew single-point test to characterize a system's quality. This is not an isolated problem with this particular review. There's a lot of snake oil in audio systems, but CR reviews of audio equipment were a joke for many years because of over-reliance on specific lab measurements. CR has had issues over the years with mistaken testing of child safety seats, pet food, and other products over the years (see their wiki page). This caveat is apparent after subscribing for a few years.