3 ms·

They do have a clickable footnote on one of their claims which currently reads: > Testing conducted by Apple in September 2021 by measuring page load performan

They do have a clickable footnote on one of their claims which currently reads: > Testing conducted by Apple in September 2021 by measuring page load performance of snapshot versions of 10 popular websites under simulated network conditions. Tested on production 13-inch MacBook Pro systems with Apple M1, 8GB RAM, 256GB SSD, and prerelease macOS Monterey. Tested with prerelease Safari 15.0 and Chrome v94.0.4606.61. Performance will vary based on usage, system configuration, network connection, and other factors. And in August 2021, it read: > Testing conducted by Apple in October 2020 by measuring page load performance of snapshot versions of 10 popular websites under simulated network conditions. Tested on production 1.4GHz quad-core Intel Core i5-based 13-inch MacBook Pro systems with 8GB RAM, 256GB SSD, and prerelease macOS Big Sur. Tested with prerelease Safari 14.0.1 and Chrome v85.0.4183.121. Performance will vary based on usage, system configuration, network connection, and other factors. In September 2020, they posted a clickable footnote with: > Testing conducted by Apple in August 2019 using Jetstream 2, MotionMark 1.1, and Speedometer 2.0 performance benchmarks. Tested on production 1.4GHz quad-core Intel Core i5-based 13-inch MacBook Pro systems with 8GB RAM, 256GB SSD, and prerelease macOS Catalina, and Windows 10 Home, version 1903, running in Boot Camp. Scores represent browsers that completed the test. Tested with prerelease Safari 13, Chrome v76.0.3809.100, and Firefox v68.0.2 on macOS, as well as Chrome v76.0.3809.100, Microsoft Edge v44.18362.267.0, and Firefox v68.0.2 on Windows Home, with WPA2 Wi-Fi network connection. Performance will vary based on system configuration, network connection, and other factors. Could they constantly update this piece of information? Sure. But I think they're being pretty up front and clear about what they're saying. And it seems like they do similar testing every year, at a fairly regular interval. I see no evidence that they're doing anything wrong here. I think it's a pretty high bar to expect that all web pages must always show the latest possible data. Edit: I actually pulled slightly different footnotes accidentally, since the pages have updated over time. But in general, all testing clearly points to the months where the testing occurred. I think the worst thing I saw was in 2018, I think they had some tests which used data in August and October. Which I don't love. But I didn't notice that happening recently, and they did clearly disclose it.

jefftk 5y ago
> September 2021 ... prerelease Safari 15.0 and Chrome v94 It looks like in September 2021 Chrome v94 was in either Beta or Stable depending on when in the month they were testing. [1] Assuming they picked their versions early in the month, this seems very reasonable. > October 2020 ... prerelease Safari 14.0.1 and Chrome v85 In October 2020 Chrome Beta was v86 and then v87 (2020-10-15), so this is less reasonable: prerelease Safari vs stable Chrome. > August 2019 ... prerelease Safari 13, Chrome v76 In August 2019 Chrome beta was v76 and then v77 (2019-08-08). Assuming they picked their versions at the beginning of the month, this is fine. [1] https://chromiumdash.appspot.com/schedule https://chromiumdash.appspot.com/schedule (Disclosure: I work for Google, speaking only for myself)
- ezfe 5y ago
  They say "prerelease" Safari because it's the version they're marketing, but they test it before they officially release it so the marketing materials are ready. That version of Safari would've been current at publication time.
  - jefftk 5y ago
    That's a good point: if they are literally about to release it, it's much more similar to stable than beta. I tried to look at historical timing, and it's kind of confusing. For example, Apple said they tested "prerelease Safari 14.0.1" in October 2020. But the initial Safari 14 came out on 2020-09-16, an 14.0.1 was a security update released on 2020-11-12 (which I would expect not to have any web compatibility or performance changes). My overall takeaway is that what Apple is doing here is unremarkable and not misleading.
Nullabillity 5y ago
> But I think they're being pretty up front and clear about what they're saying. And it seems like they do similar testing every year, at a fairly regular interval. I see no evidence that they're doing anything wrong here. Is this a failed attempt at sarcasm? Apple takes the small print to an art, in addition to being several pages of scrolling away from the actual claim. Even presented with normal typography here on HN, the text is so unreadable that my eyes try to glance off of it, and I'm a technical users that knows what the text means! If you want to publish this kind of "snapshot" information, either publish somewhere where that is clear from context (like a blog), or at the very least make sure that the inline content makes it very clear! For example: "In October 2020, we found that Safari caused the battery to last 10% longer the then-latest version of Chrome". Publishing this kind of claim as evergreen and then pointing at the small print when anyone complains (as if anyone in the actual target group actually read that) is effectively fraud.

jefftk 5y ago

> September 2021 ... prerelease Safari 15.0 and Chrome v94 It looks like in September 2021 Chrome v94 was in either Beta or Stable depending on when in the month they were testing. [1] Assuming they picked their versions early in the month, this seems very reasonable. > October 2020 ... prerelease Safari 14.0.1 and Chrome v85 In October 2020 Chrome Beta was v86 and then v87 (2020-10-15), so this is less reasonable: prerelease Safari vs stable Chrome. > August 2019 ... prerelease Safari 13, Chrome v76 In August 2019 Chrome beta was v76 and then v77 (2019-08-08). Assuming they picked their versions at the beginning of the month, this is fine. [1] https://chromiumdash.appspot.com/schedule https://chromiumdash.appspot.com/schedule (Disclosure: I work for Google, speaking only for myself)

ezfe 5y ago
They say "prerelease" Safari because it's the version they're marketing, but they test it before they officially release it so the marketing materials are ready. That version of Safari would've been current at publication time.
- jefftk 5y ago
  That's a good point: if they are literally about to release it, it's much more similar to stable than beta. I tried to look at historical timing, and it's kind of confusing. For example, Apple said they tested "prerelease Safari 14.0.1" in October 2020. But the initial Safari 14 came out on 2020-09-16, an 14.0.1 was a security update released on 2020-11-12 (which I would expect not to have any web compatibility or performance changes). My overall takeaway is that what Apple is doing here is unremarkable and not misleading.

Nullabillity 5y ago

> But I think they're being pretty up front and clear about what they're saying. And it seems like they do similar testing every year, at a fairly regular interval. I see no evidence that they're doing anything wrong here. Is this a failed attempt at sarcasm? Apple takes the small print to an art, in addition to being several pages of scrolling away from the actual claim. Even presented with normal typography here on HN, the text is so unreadable that my eyes try to glance off of it, and I'm a technical users that knows what the text means! If you want to publish this kind of "snapshot" information, either publish somewhere where that is clear from context (like a blog), or at the very least make sure that the inline content makes it very clear! For example: "In October 2020, we found that Safari caused the battery to last 10% longer the then-latest version of Chrome". Publishing this kind of claim as evergreen and then pointing at the small print when anyone complains (as if anyone in the actual target group actually read that) is effectively fraud.