12 ms·

If you're going to talk about the performance of a production site, use real-user data. To do that I'd have to implement features that track users, or at least

If you're going to talk about the performance of a production site, use real-user data. To do that I'd have to implement features that track users, or at least report back to a server with data I could use to track users, and I don't want to do that. As unfortunate as lab-data-only tests are, I'd rather accept those limitations than gather data I don't have any genuine need for. Heck, I'd rather deliver a slightly slower website than do that. Some websites probably do have a need for super accurate perf metrics, and the devs who build those sites will have to wrestle with the question of what data get to from users for themselves, but I believe we ought to be normalising limiting what data we gather in the real world for everything else. Ultimately, when I ask myself whether performance is more important than user privacy, I end up answering no.

maven29 5y ago
A beta/early-access program?
- onion2k 5y ago
  Tracking people almost always a bad idea in my opinion. Even if they get something in return like early access to something. Even if they're happy to accept being tracked. There just isn't a good justification for it in most cases. For a start, most sites that track users don't even use the data they gather. They're just gathering it because they're think they're expected to. So many sites implement Google Analytics, Hotjar, etc and then use the data as nothing more than a web counter. That's beyond infuriating to anyone who thinks privacy is worth fighting for. If you're going to violate my privacy, at least do it for a good reason. In the case of the article there could be solid reasons for hoovering up lots of information about users. But really, unless you're in a massive company with significant resources, all of that perf data is just going to sit in a database and a senior might occasionally look at the dashboard overview and think "We should make our images smaller." or "Huh, users have a sucky time when they're on Android. Oh well.". Those are not good reasons to gather real-world user data instead of relying on Lighthouse scores. Lighthouse is "good enough" for most people. I imagine that a frustrating part of working on web perf is that small companies and startups don't have the resources to do it well, so they end up cargo-culting what Google does but without spending time or effort to make good on what they might learn. To be fair to the author, I'm much more at fault here than they are. I have no doubt that the article is aimed it at a much smaller audience (the web perf community), and now I'm here saying it's not applicable in a much broader context because it was posted to HN. I apologize to them for that. This is something I care about and it was an opportunity to rant. :)
  - lelandfe 5y ago
    I’m not sure we are in agreement on what tracking means. This article advocates logging performance timings. You can’t track someone with that.
    - onion2k 5y ago
      This article advocates logging performance timings. You can’t track someone with that. Firstly, yes you can - https://www.researchgate.net/publication/2441386_Timing_Attacks_on_Web_Privacy https://www.researchgate.net/publication/2441386_Timing_Atta... - but that's not the point here. Secondly, web perf testing benefits from user tracking. It is very common for perf and tracking to be used together. In fact, I don't actually know of a time where I've seen perf metrics being used without tracking. I'd agree that it's possible to track perf metrics without tying them to a specific user, but even then you still need to be sending user data back to your server. I'm not in favour of that for something as trivial (in most case) as just making a website load slightly faster. Site owners should start with the assumption that they don't actually need data and then only track things that are critical to the functioning of their website. There will be very few cases where knowing how fast the page loaded actually meets that criteria.
      - lelandfe 5y ago
        > Firstly, yes you can 20 year old paper. "Domain tagging," Felten and Schneider's proposed countermeasure, was implemented as "cache partitioning" in Firefox, Safari, and Chrome. If you're aware of extant issues please let me know. > In fact, I don't actually know of a time where I've seen perf metrics being used without tracking Google's own library: https://github.com/GoogleChrome/web-vitals#overview https://github.com/GoogleChrome/web-vitals#overview, which is recommended in OP's article. > I'm not in favour of that for something as trivial (in most case) as just making a website load slightly faster No performance RUM means no knowledge of performance regressions. I have deployed changes that looked great on our CI and then gravely impacted performance for a segment of customers. Blinding a business to user experience because you've read papers on historic exploits seems like odd logic to me. You can't do anything malicious with a DB of LCP timings. HN's anti-telemetry mindset is baffling to me.
Cthulhu_ 5y ago
The trick here is to ask for explicit consent, and use "first party" tracking solutions that run on your own servers instead of phone home. You can probably get perf only tooling as well, for which I don't believe you even need permission (GDPR wise).
- southerntofu 5y ago
  It's true GDPR is so broad you don't need to gather consent for such metrics, as it should be covered by contractual obligations (providing a best-effort service) and the legitimate interest of the operator (improving their service), both of which are exemptions to explicit consent. However, if you want good performance, do actual UX/performance studies without treating your users like lab rats (hell, even poor rats don't deserve that). Just nitpicking, but "first-party tracking" really falls into "phoning home" (just a different "home").
klntsky 5y ago
I believe that given that one knows the user agents of users, it is possible to create a test suite with browser automation tools to measure performance in some interactive scenarios in different browsers. It's still 'lab data', but not as crude as just time-to-first-render or whatever you can measure without considering user actions. I doubt the difference with real data will matter at this point. Am I wrong?
- scinos 5y ago
  For starters, you lab data doesn't account for latency from different locations or client caches.
  - southerntofu 5y ago
    Good point! But there's an abundance of network tools to simulate latency and packet loss. Last i checked, even my Tor Browser has that in the debugging tools (F12) and can also disable caching.
scinos 5y ago
You don't need to track users to track performance. You only need to know that a page/feature took X ms to load, there is no user-related data there. If you want to add other dimensions like user agent or country, then you may step a bit into user data, but it's as easy as avoid those metrics.
- onion2k 5y ago
  Once you have tracking code in the client the temptation to use it to answer problems will be very tempting, and the pressure from other stakeholders to add 'useful' data will pile on. If your product manager wants to know user screen resolutions it's much harder for them to argue "We track nothing, we need to implement tracking to get screen resolutions." than it is for them to say "We already track loading times; it's a few additional lines of code to add screen resolutions".
- southerntofu 5y ago
  What's funny about this is that client-side "performance metrics" scripts is really close to the definition of malware (only the data points gathered differ with an actual tracker), doesn't work in all cases (older browsers, NoScript..). But what's really hilarious about it all is that in my experience this kind of sites doing metrics gathering are the worst offenders in terms of performance. Please, just do what you need to do and don't add more bloat in the name of performance. If you need to do tests, setup a test VM with 512MB RAM, 1s artificial latency and 20% packet loss. By testing for the "worst" case (which is close to the reality of many people) you ensure the average case is great.
  - gherkinnn 5y ago
    > doesn't work in all cases (older browsers, NoScript..) And that is fine. Even if old browsers and noscript were supported, it's usually a small percentage. > kind of sites doing metrics gathering are the worst offenders in terms of performance. You have a point. I have a feeling it's an org problem, bloated org builds bloated products, but struggle to articulate it further. > setup a test VM with 512MB RAM, 1s artificial latency and 20% packet loss. A synthetic approach will get you a long way already. Though I would like to see it as part of the CI/CD pipeline. Testing perf has to be passive and continuous or it's useless.
tkzed49 5y ago
You don't really need to track users. Send back a few timings: LCP, FID, whatever, and nothing else. That's all you need to look at the distribution of real world performance data and compute useful statistics. You don't need to associate data points with any kind of user profile. Sure, you _could_ maybe mine some noisy timings for user fingerprint entropy but... if you're building something more complicated than a blog, real performance data is very useful and the privacy implication is relatively minimal.
- edoceo 5y ago
  What are LCP and FID?
  - kretaceous 5y ago
    LCP: Largest Contentful Paint: "render time of the largest image or text block visible within the viewport, relative to when the page first started loading" https://web.dev/lcp/ https://web.dev/lcp/ FID: First Input Delay: "the time from when a user first interacts with a page to the time when the browser is actually able to begin processing event handlers in response to that interaction" https://web.dev/fid/ https://web.dev/fid/
  - PudgePacket 5y ago
    https://web.dev/vitals/ https://web.dev/vitals/
southerntofu 5y ago
> Ultimately, when I ask myself whether performance is more important than user privacy, I end up answering no. Thank you for this! We need more people like you building websites. Or less people unlike you, not sure which :) Like others pointed out, it's fairly easy to test in a "lab" without affecting actual users. I wish more developers did just that to see how their software/website fares on low-resource hardware or lossy connection.
poulsbohemian 5y ago
>Ultimately, when I ask myself whether performance is more important than user privacy, I end up answering no. FWIW: In the past when I sold and supported these kinds of tools, users were completely anonymized - you saw them as "requests" rather than users. In fact, the best tools I saw didn't even capture "all" users, but rather did sampling based on various attributes, IE: request was outside the standard deviation.

maven29 5y ago

A beta/early-access program?

onion2k 5y ago
Tracking people almost always a bad idea in my opinion. Even if they get something in return like early access to something. Even if they're happy to accept being tracked. There just isn't a good justification for it in most cases. For a start, most sites that track users don't even use the data they gather. They're just gathering it because they're think they're expected to. So many sites implement Google Analytics, Hotjar, etc and then use the data as nothing more than a web counter. That's beyond infuriating to anyone who thinks privacy is worth fighting for. If you're going to violate my privacy, at least do it for a good reason. In the case of the article there could be solid reasons for hoovering up lots of information about users. But really, unless you're in a massive company with significant resources, all of that perf data is just going to sit in a database and a senior might occasionally look at the dashboard overview and think "We should make our images smaller." or "Huh, users have a sucky time when they're on Android. Oh well.". Those are not good reasons to gather real-world user data instead of relying on Lighthouse scores. Lighthouse is "good enough" for most people. I imagine that a frustrating part of working on web perf is that small companies and startups don't have the resources to do it well, so they end up cargo-culting what Google does but without spending time or effort to make good on what they might learn. To be fair to the author, I'm much more at fault here than they are. I have no doubt that the article is aimed it at a much smaller audience (the web perf community), and now I'm here saying it's not applicable in a much broader context because it was posted to HN. I apologize to them for that. This is something I care about and it was an opportunity to rant. :)
- lelandfe 5y ago
  I’m not sure we are in agreement on what tracking means. This article advocates logging performance timings. You can’t track someone with that.
  - onion2k 5y ago
    This article advocates logging performance timings. You can’t track someone with that. Firstly, yes you can - https://www.researchgate.net/publication/2441386_Timing_Attacks_on_Web_Privacy https://www.researchgate.net/publication/2441386_Timing_Atta... - but that's not the point here. Secondly, web perf testing benefits from user tracking. It is very common for perf and tracking to be used together. In fact, I don't actually know of a time where I've seen perf metrics being used without tracking. I'd agree that it's possible to track perf metrics without tying them to a specific user, but even then you still need to be sending user data back to your server. I'm not in favour of that for something as trivial (in most case) as just making a website load slightly faster. Site owners should start with the assumption that they don't actually need data and then only track things that are critical to the functioning of their website. There will be very few cases where knowing how fast the page loaded actually meets that criteria.
    - lelandfe 5y ago
      > Firstly, yes you can 20 year old paper. "Domain tagging," Felten and Schneider's proposed countermeasure, was implemented as "cache partitioning" in Firefox, Safari, and Chrome. If you're aware of extant issues please let me know. > In fact, I don't actually know of a time where I've seen perf metrics being used without tracking Google's own library: https://github.com/GoogleChrome/web-vitals#overview https://github.com/GoogleChrome/web-vitals#overview, which is recommended in OP's article. > I'm not in favour of that for something as trivial (in most case) as just making a website load slightly faster No performance RUM means no knowledge of performance regressions. I have deployed changes that looked great on our CI and then gravely impacted performance for a segment of customers. Blinding a business to user experience because you've read papers on historic exploits seems like odd logic to me. You can't do anything malicious with a DB of LCP timings. HN's anti-telemetry mindset is baffling to me.

Cthulhu_ 5y ago

The trick here is to ask for explicit consent, and use "first party" tracking solutions that run on your own servers instead of phone home. You can probably get perf only tooling as well, for which I don't believe you even need permission (GDPR wise).

southerntofu 5y ago
It's true GDPR is so broad you don't need to gather consent for such metrics, as it should be covered by contractual obligations (providing a best-effort service) and the legitimate interest of the operator (improving their service), both of which are exemptions to explicit consent. However, if you want good performance, do actual UX/performance studies without treating your users like lab rats (hell, even poor rats don't deserve that). Just nitpicking, but "first-party tracking" really falls into "phoning home" (just a different "home").

klntsky 5y ago

I believe that given that one knows the user agents of users, it is possible to create a test suite with browser automation tools to measure performance in some interactive scenarios in different browsers. It's still 'lab data', but not as crude as just time-to-first-render or whatever you can measure without considering user actions. I doubt the difference with real data will matter at this point. Am I wrong?

scinos 5y ago
For starters, you lab data doesn't account for latency from different locations or client caches.
- southerntofu 5y ago
  Good point! But there's an abundance of network tools to simulate latency and packet loss. Last i checked, even my Tor Browser has that in the debugging tools (F12) and can also disable caching.

scinos 5y ago

You don't need to track users to track performance. You only need to know that a page/feature took X ms to load, there is no user-related data there. If you want to add other dimensions like user agent or country, then you may step a bit into user data, but it's as easy as avoid those metrics.

onion2k 5y ago
Once you have tracking code in the client the temptation to use it to answer problems will be very tempting, and the pressure from other stakeholders to add 'useful' data will pile on. If your product manager wants to know user screen resolutions it's much harder for them to argue "We track nothing, we need to implement tracking to get screen resolutions." than it is for them to say "We already track loading times; it's a few additional lines of code to add screen resolutions".
southerntofu 5y ago
What's funny about this is that client-side "performance metrics" scripts is really close to the definition of malware (only the data points gathered differ with an actual tracker), doesn't work in all cases (older browsers, NoScript..). But what's really hilarious about it all is that in my experience this kind of sites doing metrics gathering are the worst offenders in terms of performance. Please, just do what you need to do and don't add more bloat in the name of performance. If you need to do tests, setup a test VM with 512MB RAM, 1s artificial latency and 20% packet loss. By testing for the "worst" case (which is close to the reality of many people) you ensure the average case is great.
- gherkinnn 5y ago
  > doesn't work in all cases (older browsers, NoScript..) And that is fine. Even if old browsers and noscript were supported, it's usually a small percentage. > kind of sites doing metrics gathering are the worst offenders in terms of performance. You have a point. I have a feeling it's an org problem, bloated org builds bloated products, but struggle to articulate it further. > setup a test VM with 512MB RAM, 1s artificial latency and 20% packet loss. A synthetic approach will get you a long way already. Though I would like to see it as part of the CI/CD pipeline. Testing perf has to be passive and continuous or it's useless.

tkzed49 5y ago

You don't really need to track users. Send back a few timings: LCP, FID, whatever, and nothing else. That's all you need to look at the distribution of real world performance data and compute useful statistics. You don't need to associate data points with any kind of user profile. Sure, you _could_ maybe mine some noisy timings for user fingerprint entropy but... if you're building something more complicated than a blog, real performance data is very useful and the privacy implication is relatively minimal.

edoceo 5y ago
What are LCP and FID?
- kretaceous 5y ago
  LCP: Largest Contentful Paint: "render time of the largest image or text block visible within the viewport, relative to when the page first started loading" https://web.dev/lcp/ https://web.dev/lcp/ FID: First Input Delay: "the time from when a user first interacts with a page to the time when the browser is actually able to begin processing event handlers in response to that interaction" https://web.dev/fid/ https://web.dev/fid/
- PudgePacket 5y ago
  https://web.dev/vitals/ https://web.dev/vitals/

southerntofu 5y ago

> Ultimately, when I ask myself whether performance is more important than user privacy, I end up answering no. Thank you for this! We need more people like you building websites. Or less people unlike you, not sure which :) Like others pointed out, it's fairly easy to test in a "lab" without affecting actual users. I wish more developers did just that to see how their software/website fares on low-resource hardware or lossy connection.

poulsbohemian 5y ago

>Ultimately, when I ask myself whether performance is more important than user privacy, I end up answering no. FWIW: In the past when I sold and supported these kinds of tools, users were completely anonymized - you saw them as "requests" rather than users. In fact, the best tools I saw didn't even capture "all" users, but rather did sampling based on various attributes, IE: request was outside the standard deviation.