6 ms·

Weird that the Safari team gets worked up over this when https://apple.com/safari https://apple.com/safari regularly shows numbers based on years-old versions o

by bla3 5y ago

Weird that the Safari team gets worked up over this when https://apple.com/safari https://apple.com/safari regularly shows numbers based on years-old versions of Chrome. They just updated it so it's now only Safari prerelease versus Chrome 94, but it used to compare against a one-year old version. Right after the update, it's still comparing prerelease to an outdated stable Chrome version. Apple is very willing to intentionally make skewed comparisons to make themselves look good, but Apple apparently also gets very upset over an apparently honest mistake where a presentation that even tried to make them look good didn't make them look good enough.

ezfe 5y ago
I mean, it would've been the newest version at publication a year ago. Nobody expects Google to update these slides over the next year either.
- darkwater 5y ago
  Being a dynamic website vs a static slide from a presentation happened at a specific point in time, I think you are comparing apples (err..) to oranges.
  - FemmeAndroid 5y ago
    They do have a clickable footnote on one of their claims which currently reads: > Testing conducted by Apple in September 2021 by measuring page load performance of snapshot versions of 10 popular websites under simulated network conditions. Tested on production 13-inch MacBook Pro systems with Apple M1, 8GB RAM, 256GB SSD, and prerelease macOS Monterey. Tested with prerelease Safari 15.0 and Chrome v94.0.4606.61. Performance will vary based on usage, system configuration, network connection, and other factors. And in August 2021, it read: > Testing conducted by Apple in October 2020 by measuring page load performance of snapshot versions of 10 popular websites under simulated network conditions. Tested on production 1.4GHz quad-core Intel Core i5-based 13-inch MacBook Pro systems with 8GB RAM, 256GB SSD, and prerelease macOS Big Sur. Tested with prerelease Safari 14.0.1 and Chrome v85.0.4183.121. Performance will vary based on usage, system configuration, network connection, and other factors. In September 2020, they posted a clickable footnote with: > Testing conducted by Apple in August 2019 using Jetstream 2, MotionMark 1.1, and Speedometer 2.0 performance benchmarks. Tested on production 1.4GHz quad-core Intel Core i5-based 13-inch MacBook Pro systems with 8GB RAM, 256GB SSD, and prerelease macOS Catalina, and Windows 10 Home, version 1903, running in Boot Camp. Scores represent browsers that completed the test. Tested with prerelease Safari 13, Chrome v76.0.3809.100, and Firefox v68.0.2 on macOS, as well as Chrome v76.0.3809.100, Microsoft Edge v44.18362.267.0, and Firefox v68.0.2 on Windows Home, with WPA2 Wi-Fi network connection. Performance will vary based on system configuration, network connection, and other factors. Could they constantly update this piece of information? Sure. But I think they're being pretty up front and clear about what they're saying. And it seems like they do similar testing every year, at a fairly regular interval. I see no evidence that they're doing anything wrong here. I think it's a pretty high bar to expect that all web pages must always show the latest possible data. Edit: I actually pulled slightly different footnotes accidentally, since the pages have updated over time. But in general, all testing clearly points to the months where the testing occurred. I think the worst thing I saw was in 2018, I think they had some tests which used data in August and October. Which I don't love. But I didn't notice that happening recently, and they did clearly disclose it.
    - jefftk 5y ago
      > September 2021 ... prerelease Safari 15.0 and Chrome v94 It looks like in September 2021 Chrome v94 was in either Beta or Stable depending on when in the month they were testing. [1] Assuming they picked their versions early in the month, this seems very reasonable. > October 2020 ... prerelease Safari 14.0.1 and Chrome v85 In October 2020 Chrome Beta was v86 and then v87 (2020-10-15), so this is less reasonable: prerelease Safari vs stable Chrome. > August 2019 ... prerelease Safari 13, Chrome v76 In August 2019 Chrome beta was v76 and then v77 (2019-08-08). Assuming they picked their versions at the beginning of the month, this is fine. [1] https://chromiumdash.appspot.com/schedule https://chromiumdash.appspot.com/schedule (Disclosure: I work for Google, speaking only for myself)
      - ezfe 5y ago
        They say "prerelease" Safari because it's the version they're marketing, but they test it before they officially release it so the marketing materials are ready. That version of Safari would've been current at publication time.
        jefftk 5y ago
        That's a good point: if they are literally about to release it, it's much more similar to stable than beta. I tried to look at historical timing, and it's kind of confusing. For example, Apple said they tested "prerelease Safari 14.0.1" in October 2020. But the initial Safari 14 came out on 2020-09-16, an 14.0.1 was a security update released on 2020-11-12 (which I would expect not to have any web compatibility or performance changes). My overall takeaway is that what Apple is doing here is unremarkable and not misleading.
    - Nullabillity 5y ago
      > But I think they're being pretty up front and clear about what they're saying. And it seems like they do similar testing every year, at a fairly regular interval. I see no evidence that they're doing anything wrong here. Is this a failed attempt at sarcasm? Apple takes the small print to an art, in addition to being several pages of scrolling away from the actual claim. Even presented with normal typography here on HN, the text is so unreadable that my eyes try to glance off of it, and I'm a technical users that knows what the text means! If you want to publish this kind of "snapshot" information, either publish somewhere where that is clear from context (like a blog), or at the very least make sure that the inline content makes it very clear! For example: "In October 2020, we found that Safari caused the battery to last 10% longer the then-latest version of Chrome". Publishing this kind of claim as evergreen and then pointing at the small print when anyone complains (as if anyone in the actual target group actually read that) is effectively fraud.
soraminazuki 5y ago
So I read your comment expecting to be mad at Apple, but a quick look at the Wayback Machine [1] disproved almost everything you wrote. > They just updated it Last month would be a more accurate description [2]. > but it used to compare against a one-year old version What actually happened is that they hadn't updated the data on their website for a year. That's very different from "comparing it against a one-year old version." They were at no point comparing the latest version of Safari against a year old version of Chrome. > Apple is very willing to intentionally make skewed comparisons It looks more like you're willing to make intentionally skewed projection of facts. [1]: https://web.archive.org/web/*/https://www.apple.com/safari https://web.archive.org/web/*/https://www.apple.com/safari [2]: https://web.archive.org/web/20211026001113/https://www.apple.com/safari/ https://web.archive.org/web/20211026001113/https://www.apple...
- madeofpalk 5y ago
  Maybe both of you are right? I'm sure Apple's page was factual and up-to-date at the time it was posted, but Chrome's faster dev cycle means that the Chrome data is outdated long before the next year when Apple updates Safari.
- onion2k 5y ago
  That's very different from "comparing it against a one-year old version. The outcome of Apple not keeping their page up to date is that they were misleading the user by comparing Safari to a version of Chrome that is no longer current, or relevant. The action (or inaction strictly speaking) might be different, but the result is the same. Apple have enough staff that they could keep a page up to date if they wanted to. They chose not to.
  - jefftk 5y ago
    > misleading the user by comparing Safari to a version of Chrome that is no longer current, or relevant If they had continued to update the version of Safari in their comparisons without updating the version of Chrome I agree that would be misleading. But leaving the comparison frozen because they haven't updated their page is just the normal difficulty of keeping things up-to-date. (Disclosure: I work at Google, speaking only for myself)
deleted 5y ago
[deleted]
phillipseamore 5y ago
I like their honesty though it's in the small print "Not all features are available on all devices."

by bla3 5y ago

ezfe 5y ago

I mean, it would've been the newest version at publication a year ago. Nobody expects Google to update these slides over the next year either.

darkwater 5y ago
Being a dynamic website vs a static slide from a presentation happened at a specific point in time, I think you are comparing apples (err..) to oranges.
- FemmeAndroid 5y ago
  They do have a clickable footnote on one of their claims which currently reads: > Testing conducted by Apple in September 2021 by measuring page load performance of snapshot versions of 10 popular websites under simulated network conditions. Tested on production 13-inch MacBook Pro systems with Apple M1, 8GB RAM, 256GB SSD, and prerelease macOS Monterey. Tested with prerelease Safari 15.0 and Chrome v94.0.4606.61. Performance will vary based on usage, system configuration, network connection, and other factors. And in August 2021, it read: > Testing conducted by Apple in October 2020 by measuring page load performance of snapshot versions of 10 popular websites under simulated network conditions. Tested on production 1.4GHz quad-core Intel Core i5-based 13-inch MacBook Pro systems with 8GB RAM, 256GB SSD, and prerelease macOS Big Sur. Tested with prerelease Safari 14.0.1 and Chrome v85.0.4183.121. Performance will vary based on usage, system configuration, network connection, and other factors. In September 2020, they posted a clickable footnote with: > Testing conducted by Apple in August 2019 using Jetstream 2, MotionMark 1.1, and Speedometer 2.0 performance benchmarks. Tested on production 1.4GHz quad-core Intel Core i5-based 13-inch MacBook Pro systems with 8GB RAM, 256GB SSD, and prerelease macOS Catalina, and Windows 10 Home, version 1903, running in Boot Camp. Scores represent browsers that completed the test. Tested with prerelease Safari 13, Chrome v76.0.3809.100, and Firefox v68.0.2 on macOS, as well as Chrome v76.0.3809.100, Microsoft Edge v44.18362.267.0, and Firefox v68.0.2 on Windows Home, with WPA2 Wi-Fi network connection. Performance will vary based on system configuration, network connection, and other factors. Could they constantly update this piece of information? Sure. But I think they're being pretty up front and clear about what they're saying. And it seems like they do similar testing every year, at a fairly regular interval. I see no evidence that they're doing anything wrong here. I think it's a pretty high bar to expect that all web pages must always show the latest possible data. Edit: I actually pulled slightly different footnotes accidentally, since the pages have updated over time. But in general, all testing clearly points to the months where the testing occurred. I think the worst thing I saw was in 2018, I think they had some tests which used data in August and October. Which I don't love. But I didn't notice that happening recently, and they did clearly disclose it.
  - jefftk 5y ago
    > September 2021 ... prerelease Safari 15.0 and Chrome v94 It looks like in September 2021 Chrome v94 was in either Beta or Stable depending on when in the month they were testing. [1] Assuming they picked their versions early in the month, this seems very reasonable. > October 2020 ... prerelease Safari 14.0.1 and Chrome v85 In October 2020 Chrome Beta was v86 and then v87 (2020-10-15), so this is less reasonable: prerelease Safari vs stable Chrome. > August 2019 ... prerelease Safari 13, Chrome v76 In August 2019 Chrome beta was v76 and then v77 (2019-08-08). Assuming they picked their versions at the beginning of the month, this is fine. [1] https://chromiumdash.appspot.com/schedule https://chromiumdash.appspot.com/schedule (Disclosure: I work for Google, speaking only for myself)
    - ezfe 5y ago
      They say "prerelease" Safari because it's the version they're marketing, but they test it before they officially release it so the marketing materials are ready. That version of Safari would've been current at publication time.
      - jefftk 5y ago
        That's a good point: if they are literally about to release it, it's much more similar to stable than beta. I tried to look at historical timing, and it's kind of confusing. For example, Apple said they tested "prerelease Safari 14.0.1" in October 2020. But the initial Safari 14 came out on 2020-09-16, an 14.0.1 was a security update released on 2020-11-12 (which I would expect not to have any web compatibility or performance changes). My overall takeaway is that what Apple is doing here is unremarkable and not misleading.
  - Nullabillity 5y ago
    > But I think they're being pretty up front and clear about what they're saying. And it seems like they do similar testing every year, at a fairly regular interval. I see no evidence that they're doing anything wrong here. Is this a failed attempt at sarcasm? Apple takes the small print to an art, in addition to being several pages of scrolling away from the actual claim. Even presented with normal typography here on HN, the text is so unreadable that my eyes try to glance off of it, and I'm a technical users that knows what the text means! If you want to publish this kind of "snapshot" information, either publish somewhere where that is clear from context (like a blog), or at the very least make sure that the inline content makes it very clear! For example: "In October 2020, we found that Safari caused the battery to last 10% longer the then-latest version of Chrome". Publishing this kind of claim as evergreen and then pointing at the small print when anyone complains (as if anyone in the actual target group actually read that) is effectively fraud.

soraminazuki 5y ago

So I read your comment expecting to be mad at Apple, but a quick look at the Wayback Machine [1] disproved almost everything you wrote. > They just updated it Last month would be a more accurate description [2]. > but it used to compare against a one-year old version What actually happened is that they hadn't updated the data on their website for a year. That's very different from "comparing it against a one-year old version." They were at no point comparing the latest version of Safari against a year old version of Chrome. > Apple is very willing to intentionally make skewed comparisons It looks more like you're willing to make intentionally skewed projection of facts. [1]: https://web.archive.org/web/*/https://www.apple.com/safari https://web.archive.org/web/*/https://www.apple.com/safari [2]: https://web.archive.org/web/20211026001113/https://www.apple.com/safari/ https://web.archive.org/web/20211026001113/https://www.apple...

madeofpalk 5y ago
Maybe both of you are right? I'm sure Apple's page was factual and up-to-date at the time it was posted, but Chrome's faster dev cycle means that the Chrome data is outdated long before the next year when Apple updates Safari.
onion2k 5y ago
That's very different from "comparing it against a one-year old version. The outcome of Apple not keeping their page up to date is that they were misleading the user by comparing Safari to a version of Chrome that is no longer current, or relevant. The action (or inaction strictly speaking) might be different, but the result is the same. Apple have enough staff that they could keep a page up to date if they wanted to. They chose not to.
- jefftk 5y ago
  > misleading the user by comparing Safari to a version of Chrome that is no longer current, or relevant If they had continued to update the version of Safari in their comparisons without updating the version of Chrome I agree that would be misleading. But leaving the comparison frozen because they haven't updated their page is just the normal difficulty of keeping things up-to-date. (Disclosure: I work at Google, speaking only for myself)

deleted 5y ago

[deleted]

phillipseamore 5y ago

I like their honesty though it's in the small print "Not all features are available on all devices."