11 ms·

Lies, damn lies, and front-end tracking

183 points by AlexeyMK 6y ago

dntrkv 6y ago
We ran into the same exact issue at my previous employer. It's scary how similar it was to your situation given we also moved our site over to Next.js and we were a competitor to Opendoor. It also wasn't the first time I've run into bad metrics in a legacy app making it appear to convert better than a new version.
EdJiang 6y ago
Wouldn't the B variant show higher session count? If your A/B testing tool doesn't detect imbalances in cohort size I would imagine you have a bigger problem, since it's easy to accidentally measure the A and B groups differently.
- jart 6y ago
  Important takeaway is they rewrote their website from scratch before even having the A/B testing. Probably one of the best arguments I've ever seen against writing bloated website code. We now know that software can hit a point where it's so slow, that it produces false metrics about its own slowness. Imagine how many people are still out there, who didn't write website B, and are thinking, oh my bounce rate is fine, I don't need to invest money in performance, since the numbers are telling me people don't care. Folks who trust numbers at face value, don't always know what they don't know.
  - AlexeyMK 6y ago
    author here > We explicitly only changed the infra which served our landing pages, and kept the content - the HTML/CSS/JS - identical. Once the new infra was shown to work, we would begin to experiment with the website itself. - http://alexeymk.com/2020/07/14/lies-damn-list-and-front-end-tracking.html#fn:1 http://alexeymk.com/2020/07/14/lies-damn-list-and-front-end-...
ddevault 6y ago
There's a fourth reason not to do this that isn't mentioned in the article: tracking is unethical. Don't do it.
- jtsiskin 6y ago
  I think it is possible to do tracking responsibly. I don’t really see a problem with server-side anonymized tracking used to improve the product. (But yes, the blog author also talks about FB and Google pixel tracking, which is very different)
  - mrmonkeyman 6y ago
    BS. It's just monitoring marketing performance. That's cool, it's alright, but just say it, don't lie about it.
  - d0gbread 6y ago
    It's against Google's terms of service to collect personally identifiable information so it's anonymized there, too. It's the same -- you can collect data server side via the measurement protocol via the same hit parameters. Either way, you can be responsible about measurement.
- XCSme 6y ago
  So tracking your bounce-rates/conversion rates to know if your marketing budget is spent wisely is unethical?
  - throwaway2048 6y ago
    If tracking wasn't useful nobody would do it.
    - Veen 6y ago
      That’s like saying if homeopathy wasn’t useful no one would do it. Both homeopathy and ad tech are “useful” to someone, but most benefit accrues to the seller, not the buyer.
ve55 6y ago
>The less JavaScript (especially third-party) you have on your landing pages, the better. It’s a better customer experience, and it improves your page’s conversion and quality score. Really wish I read this more. And not just for the landing page.
- messe 6y ago
  I'm surprised we haven't run into this before now. JS performance on android has been stagnant for a while now, yet we still see the same awful JS-heavy applications. This doesn't bode well for the end of Moore's law. That is Moore's law in the performance sense, i.e. guaranteed future single-core performance increases [even if small], not Moore's law as in the the doubling transistor density.
  - stepstop 6y ago
    > yet we still see the same awful JS-heavy applications. Isn't Google's AMP just a bunch of Javascript too? So even their "fix" is just perpetuating the issue
    - scoot_718 6y ago
      third-party js no less.
    - ffpip 6y ago
      I fucking hate AMP so much. Wish someone ended it
      - forgotmypw17 6y ago
        I don't like the idea of AMP, and I was really concerned when I first heard about it. However, ::knock on wood::, I don't seem to have encountered it that much. Maybe it's because of how I browse, and that I don't follow "football" news, but I almost never see the URL in my browser, nor links to it anywhere. How do people typically come across AMP content?
XCSme 6y ago
I don't think this is an issue with the on-site analytics, but with the quality of the traffic and website performance. If you make sure that: 1) Your site loads in under 3-4 seconds for any user. 2) The user is interested enough to wait 3-4 seconds until the page loads. Then most issues will be solved. The problem with ads in many cases is that the traffic they send is of very low quality or just bots. In the end you already know from your ads provider how many users they say they sent and you should always use that when calculating ad conversion rate. Also note that using Cloudflare will count as bounced users who never actually even tried to load your page (bots, crawlers, scrapers, all HTTP requests).
- dogfoods 6y ago
  It's amazing that 3-4 seconds is considered a reasonable target in 2020 when 8 seconds was a target on freaking dialup. It's like having a Ferrari and considering outrunning a horse a success.
  - amluto 6y ago
    Every time I see an article about the growth of data usage, I thing that data usage in bytes is increasing much, much faster than data usage in terms of end-user utility.
    - dogfoods 6y ago
      Twitter weighs about a MB, a tweet is at most 280 characters, and typically of negative utility to the user and society. So yes.
  - ianamartin 6y ago
    No one thinks that's a reasonable target. That's insane. I don't know who that person is or what he's in charge of, but that's the fastest way to being a dead company. 100 ms max on backend processing, 500 ms max on first time to interaction. Client connection speeds matter, obviously. 3-4 seconds is get your fired ass out of here in my world.
m463 6y ago
> inform Google/Facebook via their server-side APIs when feasible, instead of trying to load a pixel I was wondering when this was going to be the norm. Can't block third parties for privacy if the first party talks to them behind the scenes. (If I understand what this means)
- amluto 6y ago
  I imagine that the vast majority of users of Facebook SDKs are not actually motivated to inform Facebook about anything. They’re using the SDK’s to enable login, likes, etc. If this moved to the server side, they might only inform Facebook about actions that involve Facebook. edit: for the specific case of conversion tracking as mentioned in the article, I would hope that only referred by Facebook/Google would be reported.
  - Nextgrid 6y ago
    A good portion of the SDK usage in mobile apps is not nefarious. They're embedding it for advertising conversion tracking which would respect the "limit ad tracking" setting in your device's settings. The problem is that beyond whatever functionality the developer of the app intended to use the SDK for, it also does its own thing of stalking the user even if the app doesn't end up using any SDK features at all.
    - amluto 6y ago
      Exactly my point. The third-party app is not nefarious, but the SDK is. A server side API would have a better chance of doing the useful bits without the nefarious parts. I doubt many competent server operators would be willing to run Facebook-supplied binaries on their precious servers, especially given Facebook’s track record of SDK crashes. (Not to mention PCI requirements, etc.)
malisper 6y ago
Can someone explain to me why third party JS has such a big impact on load time? I thought browsers deprioritize third party JS and load it after all first party assests have been downloaded and the site has been rendered. If that's the case why does third party JS still have such a big impact on page performance?
- defjosiah 6y ago
  You can (usually accidentally) end up with a render blocking script tag. In this article example it was client-side optimizely. The parsing and execution of third-party javascript is definitely non-trivial if you profile it, especially on lower end devices. Finally, browser download priority requires async and defer attributes on scripts (usually), or other clever ways of deferring loading.
  - ss3000 6y ago
    One thing important to note is for certain use cases of Optimizely like A/B testing, it's actually desirable to block rendering until it's initialized as otherwise you end up with a flash of the default variation that then gets switched to the appropriate one afterwards. Though after working with a few of these A/B testing vendors, I'm firmly convinced that you'd be better off long term implementing your own A/B testing service on top of some internal admin UI builder like Retool.
  - malisper 6y ago
    > You can (usually accidentally) end up with a render blocking script tag. In this article example it was client-side optimizely. This makes sense for a tool like Optimizely which has to block rendering. > The parsing and execution of third-party javascript is definitely non-trivial if you profile it, especially on lower end devices. Given that this is supposed to happen after the page has already loaded, does this matter? If the thing you are optimizing for is page load time, and it takes a few 100ms after the page has already loaded to run the JS, does that actually negatively impact the user experience? > Finally, browser download priority requires async and defer attributes on scripts (usually), or other clever ways of deferring loading. When you add third party JS, shouldn't this be taken care of for you? Scripts for adding third party JS should already make use of async and defer as needed. What's a case where you would need to actually think about async and defer when making use of third party JS?
itisit 6y ago
*damned
- npteljes 6y ago
  And the manager peeked in, not peaked in.
  - AlexeyMK 6y ago
    Thanks! Updated on both fronts.
pachico 6y ago
Although interesting story, it seems the main issue was that the author never sanitized their analytics in the first place and blindly relied on data that was never confirmed before. I've seen this as a quite common phenomenon. No piece of software does magic and they all require certain certain conditions to work as expected. In client JavaScript, these conditions are simply harder to grasp but they can be studied.
ravivyas 6y ago
The headline, the problem and the solutions are all different things. - Front end tracking did not lie, the people tracking it were not aware. If opendoor was spending on Ads, the marketing team would be the first to see a disparity between the clicks on an ad network and pageviews on their end. - As such this is also a reason why you need to check your server hit logs with your Analytics. - Where is the author is right it Frontend numbers being incorrect largely due to blockers - His understanding of bounces is also incorrect. Bounce is when there is no second "interaction event", many marketing folks fake-fix the bounce rate, by sending a scroll event as an interaction event. - I always tell people that analytics numbers are "signals", not "metrics", they are not accurate enough to be called "metrics"
- d0gbread 6y ago
  If these numbers were significantly changed by a server side page view, I'd bet their page view wasn't firing any earlier than the top or bottom of the body tag, or maybe even on window ready. Anyone that's spent any real time outside of dropping in a snippet understands the implications of users navigating before hits fire, and it sounds like the team just didn't have that experience. Like you, I also work to make sure people understand that client side data is not necessarily accurate, but still refer to it as metrics and KPIs, just with the caviot that we should bank on consistancy over accuracy, keeping our analysis geared towards behavior and trends versus most business people's immediate desire to take it to an operational/business intelligence type place.
dccoolgai 6y ago
Sshhhh. So many people will lose so many bullshit jobs if this gets discussed widely. There are entire tertiary industries built on top of front-end tracking and most people with a modicum of analytical ability know it's turtles all the way down.
- jkdfslgjklgdf 6y ago
  What do you mean by "turtles"?
  - dccoolgai 6y ago
    A turn of phrase: https://en.m.wikiquote.org/wiki/Turtles_all_the_way_down https://en.m.wikiquote.org/wiki/Turtles_all_the_way_down In this context, I'm referring to the "trust us, it works" attitude that people who sell/provide analytics on front-end tracking use to justify their effort/expense. Based on experience, when you point out obvious problems doing analytics with JS, they keep repeating the mantra "but you _need_ front-end analytics" infinitely.
- d0gbread 6y ago
  I thought this wasn't talked about because we all understand the limitations, accept them, and still find value. This sounds like a maturity thing, not a tool thing. Lots of user experience professionals find utility in data that serves discovery and design sprints. Lots of product management professionals find utility in data that serves validating incremental feature engagement.

183 points by AlexeyMK 6y ago

We ran into the same exact issue at my previous employer. It's scary how similar it was to your situation given we also moved our site over to Next.js and we were a competitor to Opendoor. It also wasn't the first time I've run into bad metrics in a legacy app making it appear to convert better than a new version.

EdJiang 6y ago

Wouldn't the B variant show higher session count? If your A/B testing tool doesn't detect imbalances in cohort size I would imagine you have a bigger problem, since it's easy to accidentally measure the A and B groups differently.

jart 6y ago
Important takeaway is they rewrote their website from scratch before even having the A/B testing. Probably one of the best arguments I've ever seen against writing bloated website code. We now know that software can hit a point where it's so slow, that it produces false metrics about its own slowness. Imagine how many people are still out there, who didn't write website B, and are thinking, oh my bounce rate is fine, I don't need to invest money in performance, since the numbers are telling me people don't care. Folks who trust numbers at face value, don't always know what they don't know.
- AlexeyMK 6y ago
  author here > We explicitly only changed the infra which served our landing pages, and kept the content - the HTML/CSS/JS - identical. Once the new infra was shown to work, we would begin to experiment with the website itself. - http://alexeymk.com/2020/07/14/lies-damn-list-and-front-end-tracking.html#fn:1 http://alexeymk.com/2020/07/14/lies-damn-list-and-front-end-...

ddevault 6y ago

There's a fourth reason not to do this that isn't mentioned in the article: tracking is unethical. Don't do it.

jtsiskin 6y ago
I think it is possible to do tracking responsibly. I don’t really see a problem with server-side anonymized tracking used to improve the product. (But yes, the blog author also talks about FB and Google pixel tracking, which is very different)
- mrmonkeyman 6y ago
  BS. It's just monitoring marketing performance. That's cool, it's alright, but just say it, don't lie about it.
- d0gbread 6y ago
  It's against Google's terms of service to collect personally identifiable information so it's anonymized there, too. It's the same -- you can collect data server side via the measurement protocol via the same hit parameters. Either way, you can be responsible about measurement.
XCSme 6y ago
So tracking your bounce-rates/conversion rates to know if your marketing budget is spent wisely is unethical?
- throwaway2048 6y ago
  If tracking wasn't useful nobody would do it.
  - Veen 6y ago
    That’s like saying if homeopathy wasn’t useful no one would do it. Both homeopathy and ad tech are “useful” to someone, but most benefit accrues to the seller, not the buyer.

ve55 6y ago

>The less JavaScript (especially third-party) you have on your landing pages, the better. It’s a better customer experience, and it improves your page’s conversion and quality score. Really wish I read this more. And not just for the landing page.

messe 6y ago
I'm surprised we haven't run into this before now. JS performance on android has been stagnant for a while now, yet we still see the same awful JS-heavy applications. This doesn't bode well for the end of Moore's law. That is Moore's law in the performance sense, i.e. guaranteed future single-core performance increases [even if small], not Moore's law as in the the doubling transistor density.
- stepstop 6y ago
  > yet we still see the same awful JS-heavy applications. Isn't Google's AMP just a bunch of Javascript too? So even their "fix" is just perpetuating the issue
  - scoot_718 6y ago
    third-party js no less.
  - ffpip 6y ago
    I fucking hate AMP so much. Wish someone ended it
    - forgotmypw17 6y ago
      I don't like the idea of AMP, and I was really concerned when I first heard about it. However, ::knock on wood::, I don't seem to have encountered it that much. Maybe it's because of how I browse, and that I don't follow "football" news, but I almost never see the URL in my browser, nor links to it anywhere. How do people typically come across AMP content?

XCSme 6y ago

I don't think this is an issue with the on-site analytics, but with the quality of the traffic and website performance. If you make sure that: 1) Your site loads in under 3-4 seconds for any user. 2) The user is interested enough to wait 3-4 seconds until the page loads. Then most issues will be solved. The problem with ads in many cases is that the traffic they send is of very low quality or just bots. In the end you already know from your ads provider how many users they say they sent and you should always use that when calculating ad conversion rate. Also note that using Cloudflare will count as bounced users who never actually even tried to load your page (bots, crawlers, scrapers, all HTTP requests).

dogfoods 6y ago
It's amazing that 3-4 seconds is considered a reasonable target in 2020 when 8 seconds was a target on freaking dialup. It's like having a Ferrari and considering outrunning a horse a success.
- amluto 6y ago
  Every time I see an article about the growth of data usage, I thing that data usage in bytes is increasing much, much faster than data usage in terms of end-user utility.
  - dogfoods 6y ago
    Twitter weighs about a MB, a tweet is at most 280 characters, and typically of negative utility to the user and society. So yes.
- ianamartin 6y ago
  No one thinks that's a reasonable target. That's insane. I don't know who that person is or what he's in charge of, but that's the fastest way to being a dead company. 100 ms max on backend processing, 500 ms max on first time to interaction. Client connection speeds matter, obviously. 3-4 seconds is get your fired ass out of here in my world.

m463 6y ago

> inform Google/Facebook via their server-side APIs when feasible, instead of trying to load a pixel I was wondering when this was going to be the norm. Can't block third parties for privacy if the first party talks to them behind the scenes. (If I understand what this means)

amluto 6y ago
I imagine that the vast majority of users of Facebook SDKs are not actually motivated to inform Facebook about anything. They’re using the SDK’s to enable login, likes, etc. If this moved to the server side, they might only inform Facebook about actions that involve Facebook. edit: for the specific case of conversion tracking as mentioned in the article, I would hope that only referred by Facebook/Google would be reported.
- Nextgrid 6y ago
  A good portion of the SDK usage in mobile apps is not nefarious. They're embedding it for advertising conversion tracking which would respect the "limit ad tracking" setting in your device's settings. The problem is that beyond whatever functionality the developer of the app intended to use the SDK for, it also does its own thing of stalking the user even if the app doesn't end up using any SDK features at all.
  - amluto 6y ago
    Exactly my point. The third-party app is not nefarious, but the SDK is. A server side API would have a better chance of doing the useful bits without the nefarious parts. I doubt many competent server operators would be willing to run Facebook-supplied binaries on their precious servers, especially given Facebook’s track record of SDK crashes. (Not to mention PCI requirements, etc.)

malisper 6y ago

Can someone explain to me why third party JS has such a big impact on load time? I thought browsers deprioritize third party JS and load it after all first party assests have been downloaded and the site has been rendered. If that's the case why does third party JS still have such a big impact on page performance?

defjosiah 6y ago
You can (usually accidentally) end up with a render blocking script tag. In this article example it was client-side optimizely. The parsing and execution of third-party javascript is definitely non-trivial if you profile it, especially on lower end devices. Finally, browser download priority requires async and defer attributes on scripts (usually), or other clever ways of deferring loading.
- ss3000 6y ago
  One thing important to note is for certain use cases of Optimizely like A/B testing, it's actually desirable to block rendering until it's initialized as otherwise you end up with a flash of the default variation that then gets switched to the appropriate one afterwards. Though after working with a few of these A/B testing vendors, I'm firmly convinced that you'd be better off long term implementing your own A/B testing service on top of some internal admin UI builder like Retool.
- malisper 6y ago
  > You can (usually accidentally) end up with a render blocking script tag. In this article example it was client-side optimizely. This makes sense for a tool like Optimizely which has to block rendering. > The parsing and execution of third-party javascript is definitely non-trivial if you profile it, especially on lower end devices. Given that this is supposed to happen after the page has already loaded, does this matter? If the thing you are optimizing for is page load time, and it takes a few 100ms after the page has already loaded to run the JS, does that actually negatively impact the user experience? > Finally, browser download priority requires async and defer attributes on scripts (usually), or other clever ways of deferring loading. When you add third party JS, shouldn't this be taken care of for you? Scripts for adding third party JS should already make use of async and defer as needed. What's a case where you would need to actually think about async and defer when making use of third party JS?

itisit 6y ago

*damned

npteljes 6y ago
And the manager peeked in, not peaked in.
- AlexeyMK 6y ago
  Thanks! Updated on both fronts.

pachico 6y ago

Although interesting story, it seems the main issue was that the author never sanitized their analytics in the first place and blindly relied on data that was never confirmed before. I've seen this as a quite common phenomenon. No piece of software does magic and they all require certain certain conditions to work as expected. In client JavaScript, these conditions are simply harder to grasp but they can be studied.

ravivyas 6y ago

The headline, the problem and the solutions are all different things. - Front end tracking did not lie, the people tracking it were not aware. If opendoor was spending on Ads, the marketing team would be the first to see a disparity between the clicks on an ad network and pageviews on their end. - As such this is also a reason why you need to check your server hit logs with your Analytics. - Where is the author is right it Frontend numbers being incorrect largely due to blockers - His understanding of bounces is also incorrect. Bounce is when there is no second "interaction event", many marketing folks fake-fix the bounce rate, by sending a scroll event as an interaction event. - I always tell people that analytics numbers are "signals", not "metrics", they are not accurate enough to be called "metrics"

d0gbread 6y ago
If these numbers were significantly changed by a server side page view, I'd bet their page view wasn't firing any earlier than the top or bottom of the body tag, or maybe even on window ready. Anyone that's spent any real time outside of dropping in a snippet understands the implications of users navigating before hits fire, and it sounds like the team just didn't have that experience. Like you, I also work to make sure people understand that client side data is not necessarily accurate, but still refer to it as metrics and KPIs, just with the caviot that we should bank on consistancy over accuracy, keeping our analysis geared towards behavior and trends versus most business people's immediate desire to take it to an operational/business intelligence type place.

dccoolgai 6y ago

Sshhhh. So many people will lose so many bullshit jobs if this gets discussed widely. There are entire tertiary industries built on top of front-end tracking and most people with a modicum of analytical ability know it's turtles all the way down.

jkdfslgjklgdf 6y ago
What do you mean by "turtles"?
- dccoolgai 6y ago
  A turn of phrase: https://en.m.wikiquote.org/wiki/Turtles_all_the_way_down https://en.m.wikiquote.org/wiki/Turtles_all_the_way_down In this context, I'm referring to the "trust us, it works" attitude that people who sell/provide analytics on front-end tracking use to justify their effort/expense. Based on experience, when you point out obvious problems doing analytics with JS, they keep repeating the mantra "but you _need_ front-end analytics" infinitely.
d0gbread 6y ago
I thought this wasn't talked about because we all understand the limitations, accept them, and still find value. This sounds like a maturity thing, not a tool thing. Lots of user experience professionals find utility in data that serves discovery and design sprints. Lots of product management professionals find utility in data that serves validating incremental feature engagement.