9 ms·

Shadow traffic: site visits that are not captured by typical analytics providers

58 points by ahstilde 6y ago

ThePhysicist 6y ago
Absolute numbers tend to be overrated in analytics. Often relative numbers, like the number of conversions per tracked user matter more. Also, if your product is targeted at privacy-savvy individuals like developers that often use blockers you might be better off using server-side tracking. That seems to have become a lost art though, especially since many sites use CDNs that hide a lot of visits for cacheable content.
- thekyle 6y ago
  If you use Cloudfront (AWS) they have the option for server-side tracking built-in. You just tell them which S3 bucket to dump the logs into and you get the raw HTTP requests with timestamps. I personally use a service called s3stat which takes those dumps and turns them into pretty graphs.
deleted 6y ago
[deleted]
gentleman11 6y ago
Obvious question: how do you filter out bot traffic with server side logs? What percent of visitors are bots anyway?
- bleepblorp 6y ago
  Most legitimate bots identify themselves with specific user agent strings. Script kiddie attack bots are generally fairly obvious as they hammer away at things like /wp-login.php for days on end regardless of what error codes the server returns. Most other bots are pretty evident just by looking at access patterns. Just identify their IPs and drop them from your analytics.
- toast0 6y ago
  User-Agent for nice bots, and client ASN for naughty bots will get you pretty far. Fake chrome from residential IPs is hard to detect though.
- compumike 6y ago
  There are off-the-shelf open-source libraries for this that are pretty decent and are kept up-to-date by the community. For example, you can just do browser.is_bot? after you install https://github.com/fnando/browser#bots https://github.com/fnando/browser#bots
- ratsbane 6y ago
  Watching the bot traffic is an interesting exercise in itself. The trick is not to filter it out; it's to identify it (to the greatest degree possible.)
ghgr 6y ago
The other side of the coin are the inflated traffic statistics that include all kind of bots & crawlers with spoofed user-agent, and for low traffic sites like niche personal blogs with a custom domain it can be 90+% of the server-side logged visits. How proud was the 15-year old me with my first .com domain, having over 100 visitors per day. Little did I know that the actual number of visitors was much, much less than that.
dddddaviddddd 6y ago
I had an article on the front page of Hacker News last year that had about 17,000 real visits, as determined by analysing my server log files. I was also using Google Analytics at the time, which told me I had 10,000 visitors (of which only 7 were using Firefox!). Obviously there's a gap between what trackers say and reality, bigger for some demographics than for others.
- ta17711771 6y ago
  > of which only 7 were using Firefox Were reporting using Firefox. Also, not surprising, Firefox security leaves a lot to be desired.
  - marcinzm 6y ago
    >Also, not surprising, Firefox security leaves a lot to be desired. Huh? Blocking google analytics tracking is a positive, not a negative regarding security.
    - vlovich123 6y ago
      He's talking about user agent spoofing.
      - inetknght 6y ago
        I'm not so sure. It shouldn't matter whatsoever what user agent I'm using. In fact, not sending the user agent field would be massively better if only that in and of itself wasn't a unique datapoint.
      - marcinzm 6y ago
        Still not sure how that ties into Firefox having worse security. User agent spoofing is a privacy item and a positive privacy item.
        vlovich123 6y ago
jklinger410 6y ago
This is pretty much a parsely ad. I think the HN crowd is pretty aware that ad blockers, vpns, etc can break analytics.
- acdha 6y ago
  I think you're too quick to dismiss it. It's one thing to know that it exists and another to recognize that it's somewhere between 20-40% of your total traffic, especially unprompted. I've had many conversations where people assumed their traffic numbers was real until this point was raised, at which point everyone metaphorically slapped their foreheads and realized that they had forgotten to take this into account.
  - dylz 6y ago
    another quite fun thing is that this depends on your site vertical and demographic: on some of the esports-related properties i've worked on in the past, adblock rates (that also block analytics) may exceed 80%. i have seen 90% before (along with per-device differences, like you suddenly think almost all of your traffic is mobile - because all desktop users are blocking client side analytics)
  - throwaway287391 6y ago
    When does this actually matter though? Isn't growth (or shrinkage) what you'd normally really care about (e.g. this month we had 10% more DAUs than last month)? I suppose if you changed something to attract tons of new users who disproportionately use AdBlock (for example) this becomes an issue as it wouldn't show up in your metrics, but is that sort of thing common? I suppose if nothing else it's good to know so you can immediately beef up your numbers +20% in your slide deck for VCs.
    - acdha 6y ago
      What if you're doing anything which doesn't involve logins — public information, advertising, etc. – where users don't otherwise trigger something like account creation / logins? What if you're trying to get stats about people who don't convert or otherwise give you a signal that they're using the site? I've run into sites where things like signup or checkout are blocked behind an analytics tracker (Adobe used to recommend running theirs in a synchronous navigation-blocking mode) which meant that any problem with that service was completely invisible unless they contacted you to complain. I also remember people wondering why Firefox users stopped using their site when they shipped the release which enabled tracking protection by default.
choeger 6y ago
You should use Server-Side tracking. There is no reason to design your website in a way that makes your legitimate analysis use cases depend on Client-Side computations. If Server-Side tracking looks too complex for you, you might want to reevaluate the balance of technical knowledge in your enterprise.
- XCSme 6y ago
  What if your site is a SPA? You would not know, for example, the time spent on site, what pages are visited, where exactly users leave, if there are client-side errors, right?
  - wlll 6y ago
    Then now you have two problems ;) I guess your options here are, do collect metrics in JS and hope that whatever the reason that 20% of visitors don't show up in Google analytics isn't preventing them from using your site.
  - acdha 6y ago
    If you're using a SPA you need to build that instrumentation in to match the native behaviour, along with robust error handling using something like https://github.com/getsentry/sentry/ https://github.com/getsentry/sentry/ so you can tell when your code is broken client-side where you would otherwise not have visibility. This is much less likely to be blocked if you self-host it — breaking requests to your server will break the app, whereas blocking common cross-site tracking services is popular because there are few drawbacks for the user.
    - XCSme 6y ago
      You can self-host sentry?
      - acdha 6y ago
        Yes – it's pretty easy to run the open source app in your favorite container runtime: https://hub.docker.com/_/sentry/ https://hub.docker.com/_/sentry/
DevX101 6y ago
I recognized this a couple years ago at a startup I work with. Comparing Google Analytics numbers to validated event logs, the numbers were off by ~20-30%. Surely there must be a quick workaround I thought, there's no way there's an entire multi-billion $ industry of 3rd party analytics software giving bogus numbers to websites?! But that's indeed the case. I immediately made top priority building out an in-house analytics platform where event logs were sent via the API and thus didn't get blocked. And for those saying relative direction is all that matters, I guarantee you the behavior of users with adblocker installed is very different from those who can't be bothered or don't know how.
- sharkweek 6y ago
  Let me tell you what was fun - trying to explain to a former boss why Facebook ads showed one number for amount of traffic sent, Google Analytics showed a different number for traffic from those ads, then lastly our server logs showing an entirely different number!
- tiffanyh 6y ago
  For clarity, so you’d recommend web server logs over client side analytics? If so, what open source web server analytics tools do you believe is best? E.g. https://goaccess.io https://goaccess.io
pkaye 6y ago
What does parse.ly do differently to account for this discrepancy in the analytics?
- pixelmonkey 6y ago
  Good question. I explain some of the technical approaches we've taken with customers in another comment here: https://news.ycombinator.com/item?id=24205803 https://news.ycombinator.com/item?id=24205803
pixelmonkey 6y ago
I'm one of Parse.ly's co-founders. This post was written by one of our product managers about a project and investigation we've been doing for the past few months. It first got on my team's radar when I posted this set of tweets back in 2019: https://twitter.com/amontalenti/status/1165262620959617025 https://twitter.com/amontalenti/status/1165262620959617025 Specifically: I noticed a huge difference between the metrics we were reporting on my blog post in Parse.ly, and the metrics being reported by my personal blog's Cloudflare CDN (caching the content). Ironically enough, this traffic was all coming from HN and the post was itself about modern JavaScript[1]. Since then, we've also been hearing from a lot of customers about various scenarios where traffic is either under-counted or mis-counted. For example, something that has been tripping us up lately is that our Twitter integration relies (partially) upon the official t.co link shortener[2], and yet, due to modern browser rules related to W3C Referrer Policy[3], the t.co link's path segment is often not transmitted to the analytics provider, and thus the source tweet for traffic cannot be easily ascertained. I firmly believe in privacy and analytics without compromise[4], so the team is trying to come up with ways to at least quantify shadow traffic at an aggregate level, and to ensure legitimate user privacy interests are honored, while making sure they don't break legitimate privacy-safe first-party analytics use cases. As a developer, something that concerned me recently was realizing that Sentry, the open source error tracking tool with a SaaS reporting frontend and a JavaScript SDK[5], gets blocked in many conservative browser privacy setups. Though the interest to user privacy is legitimate, I think we can all agree it'd be better for site/app operators to know when certain browsers are hitting JavaScript stack traces. [1]: https://news.ycombinator.com/item?id=20785616 https://news.ycombinator.com/item?id=20785616 [2]: https://help.twitter.com/en/using-twitter/url-shortener https://help.twitter.com/en/using-twitter/url-shortener [3]: https://www.w3.org/TR/referrer-policy/ https://www.w3.org/TR/referrer-policy/ [4]: https://blog.parse.ly/post/3394/analytics-privacy-without-compromise/ https://blog.parse.ly/post/3394/analytics-privacy-without-co... [5]: https://sentry.io/for/javascript/ https://sentry.io/for/javascript/
- MauranKilom 6y ago
  So what do you actually (want to) do in regards to measuring shadow traffic? The blog post tries to convince the reader they should care about shadow traffic and then handwaves away existing solutions, but "we're developing a solution" is as concrete as the post gets as to why I should turn to parse.ly. Now you say "the team is trying to come up with ways to at least quantify shadow traffic at an aggregate level", so it appears that you don't even have a solution yet. In light of that, presenting "existing analytics services like parse.ly" as one of three solutions on "how to measure shadow traffic" seems borderline disingenuous. If you can do it, why not say so plain and clear? If you can't do it, why do you mention yourself as a solution? Or is it only other services like parse.ly that can do it? It also rubs me the wrong way how both the blog post and your comment has an undertone of "it would be better if users didn't have as much tracking protection". Just take the framing of your last sentence as an example...
thinkloop 6y ago
Does hosting client-side tracking on your own domain circumvent all the problems? How come that hasn't become the standard and killed 3rd-party trackers? If it's a question of having to manage an analytics platform, can't that still be deferred to a 3rd-party but through your own subdomain?
- derivagral 6y ago
  My interpretation of how Google Analytics does this is that pulling in a 3rd party dep lets them keep a lot of control over versioning and similar things. I assume they do this so out-of-date or vulnerable code (or extra tracking, sure) can be cleanly and quickly applied through the network and not rely on the clients to properly update "ga.js" a few times a day.
ChuckMcM 6y ago
Okay, the cynic in me wants to write "New Age web designers are stumped by lack of analytics while still refusing to look at their HTTP server log data." I remember when the ONLY analytics were those you could derive by analyzing your http logs. Which have useful information in them. Things like source IP address (which can be geo tagged), a bunch of HTTP headers (which are full of information too), and a timestamp which tells you when it came in and from where. Not to mention session cookies which take zero javascript to implement. I've been retooling my site slowly to only use these analytics (less the cookies) because I value people's privacy while browsing as much as my own. During the transition I've been comparing what I can pull out of the logs vs what Google's analytics gives me. Sure, Google can do wonders, especially if the person is coming from a browser where they are logged into Google. But, as the article points out, they miss everyone running noscript and/or other privacy enhancers like Privacy Badger from EFF. I don't feel like I'm going to miss the Google added insights.
- neonate 6y ago
  This is pretty funny, buried in the middle of the article: > Option 2 – Server-Side Tracking
  - ChuckMcM 6y ago
    With advice that this is so technically complex, don't even look there. These guys can write java script that animates a web page so that it looks like a turning page and its "too technical" to pull data out of a server log?
  - LaundroMat 6y ago
    The article talks about sending first-party analytics events to an analytics provider from your own servers. So the server-side tracking the article refers to is similar to the server-side tagging Google recently announced[1], not the analysis of server logs. [1] https://developers.google.com/tag-manager/serverside https://developers.google.com/tag-manager/serverside
    - neonate 6y ago
paulchap 6y ago
Funnily enough, I got a warning telling me uBlock prevented this page from loading...
billyhoffman 6y ago
I get wanting to create valuable thought leadership content, but this is the worst example of Bad product marketing: 1- present a new concept to readers (shadow visitors) 2- show how this concept is scary and bad for your business (your analytics is off by 20%!!!) 3- present 2 options, which by the way are free, but immediately shit all over them (Server logs! But that’s hard and complicated! Edge logs! But getting these are hard!) 4- present your company’s product as option 3, which surprisingly, have no downsides and isnt shit upon 5- profit What disingenuous garbage. You should be ashamed Parse.ly. The right way to do this is do steps 1 and 2. Then show in detail step 3: how to solve the problem with easy options, ideally with free and open software. It’s ok to show edge cases, corner cases, or just shear scale issues that makes these options challenging. The difference is that good product marketing pieces show people how to solve the problem and offer a solution to do that at scale or in a automated/hosted way so the customer doesn’t have to deal with it. If your product marketing content’s message is “you are screwed unless you buy our product “ you are doing it wrong
- MauranKilom 6y ago
  Yeah, this article just read super weird to anyone with even a basic understanding of the field. It's very clearly targeted at managerial folk, and even then it is as shallow in its arguments as it is transparent in its motives.
indymike 6y ago
Server side tracking is useful for logging http requests. Client side tracking is useful for logging user interactions. Used to be there would be a small difference between server and client due to caches and user settings... but... Modern apps (i.e. React, Vue, Angular) often only load one page, and then all interaction is managed by client side code, so often client side tracking is the only thing that works.

58 points by ahstilde 6y ago

Absolute numbers tend to be overrated in analytics. Often relative numbers, like the number of conversions per tracked user matter more. Also, if your product is targeted at privacy-savvy individuals like developers that often use blockers you might be better off using server-side tracking. That seems to have become a lost art though, especially since many sites use CDNs that hide a lot of visits for cacheable content.

thekyle 6y ago
If you use Cloudfront (AWS) they have the option for server-side tracking built-in. You just tell them which S3 bucket to dump the logs into and you get the raw HTTP requests with timestamps. I personally use a service called s3stat which takes those dumps and turns them into pretty graphs.

deleted 6y ago

[deleted]

gentleman11 6y ago

Obvious question: how do you filter out bot traffic with server side logs? What percent of visitors are bots anyway?

bleepblorp 6y ago
Most legitimate bots identify themselves with specific user agent strings. Script kiddie attack bots are generally fairly obvious as they hammer away at things like /wp-login.php for days on end regardless of what error codes the server returns. Most other bots are pretty evident just by looking at access patterns. Just identify their IPs and drop them from your analytics.
toast0 6y ago
User-Agent for nice bots, and client ASN for naughty bots will get you pretty far. Fake chrome from residential IPs is hard to detect though.
compumike 6y ago
There are off-the-shelf open-source libraries for this that are pretty decent and are kept up-to-date by the community. For example, you can just do browser.is_bot? after you install https://github.com/fnando/browser#bots https://github.com/fnando/browser#bots
ratsbane 6y ago
Watching the bot traffic is an interesting exercise in itself. The trick is not to filter it out; it's to identify it (to the greatest degree possible.)

ghgr 6y ago

The other side of the coin are the inflated traffic statistics that include all kind of bots & crawlers with spoofed user-agent, and for low traffic sites like niche personal blogs with a custom domain it can be 90+% of the server-side logged visits. How proud was the 15-year old me with my first .com domain, having over 100 visitors per day. Little did I know that the actual number of visitors was much, much less than that.

dddddaviddddd 6y ago

I had an article on the front page of Hacker News last year that had about 17,000 real visits, as determined by analysing my server log files. I was also using Google Analytics at the time, which told me I had 10,000 visitors (of which only 7 were using Firefox!). Obviously there's a gap between what trackers say and reality, bigger for some demographics than for others.

ta17711771 6y ago
> of which only 7 were using Firefox Were reporting using Firefox. Also, not surprising, Firefox security leaves a lot to be desired.
- marcinzm 6y ago
  >Also, not surprising, Firefox security leaves a lot to be desired. Huh? Blocking google analytics tracking is a positive, not a negative regarding security.
  - vlovich123 6y ago
    He's talking about user agent spoofing.
    - inetknght 6y ago
      I'm not so sure. It shouldn't matter whatsoever what user agent I'm using. In fact, not sending the user agent field would be massively better if only that in and of itself wasn't a unique datapoint.
    - marcinzm 6y ago
      Still not sure how that ties into Firefox having worse security. User agent spoofing is a privacy item and a positive privacy item.
      - vlovich123 6y ago

jklinger410 6y ago

This is pretty much a parsely ad. I think the HN crowd is pretty aware that ad blockers, vpns, etc can break analytics.

acdha 6y ago
I think you're too quick to dismiss it. It's one thing to know that it exists and another to recognize that it's somewhere between 20-40% of your total traffic, especially unprompted. I've had many conversations where people assumed their traffic numbers was real until this point was raised, at which point everyone metaphorically slapped their foreheads and realized that they had forgotten to take this into account.
- dylz 6y ago
  another quite fun thing is that this depends on your site vertical and demographic: on some of the esports-related properties i've worked on in the past, adblock rates (that also block analytics) may exceed 80%. i have seen 90% before (along with per-device differences, like you suddenly think almost all of your traffic is mobile - because all desktop users are blocking client side analytics)
- throwaway287391 6y ago
  When does this actually matter though? Isn't growth (or shrinkage) what you'd normally really care about (e.g. this month we had 10% more DAUs than last month)? I suppose if you changed something to attract tons of new users who disproportionately use AdBlock (for example) this becomes an issue as it wouldn't show up in your metrics, but is that sort of thing common? I suppose if nothing else it's good to know so you can immediately beef up your numbers +20% in your slide deck for VCs.
  - acdha 6y ago
    What if you're doing anything which doesn't involve logins — public information, advertising, etc. – where users don't otherwise trigger something like account creation / logins? What if you're trying to get stats about people who don't convert or otherwise give you a signal that they're using the site? I've run into sites where things like signup or checkout are blocked behind an analytics tracker (Adobe used to recommend running theirs in a synchronous navigation-blocking mode) which meant that any problem with that service was completely invisible unless they contacted you to complain. I also remember people wondering why Firefox users stopped using their site when they shipped the release which enabled tracking protection by default.

choeger 6y ago

You should use Server-Side tracking. There is no reason to design your website in a way that makes your legitimate analysis use cases depend on Client-Side computations. If Server-Side tracking looks too complex for you, you might want to reevaluate the balance of technical knowledge in your enterprise.

XCSme 6y ago
What if your site is a SPA? You would not know, for example, the time spent on site, what pages are visited, where exactly users leave, if there are client-side errors, right?
- wlll 6y ago
  Then now you have two problems ;) I guess your options here are, do collect metrics in JS and hope that whatever the reason that 20% of visitors don't show up in Google analytics isn't preventing them from using your site.
- acdha 6y ago
  If you're using a SPA you need to build that instrumentation in to match the native behaviour, along with robust error handling using something like https://github.com/getsentry/sentry/ https://github.com/getsentry/sentry/ so you can tell when your code is broken client-side where you would otherwise not have visibility. This is much less likely to be blocked if you self-host it — breaking requests to your server will break the app, whereas blocking common cross-site tracking services is popular because there are few drawbacks for the user.
  - XCSme 6y ago
    You can self-host sentry?
    - acdha 6y ago
      Yes – it's pretty easy to run the open source app in your favorite container runtime: https://hub.docker.com/_/sentry/ https://hub.docker.com/_/sentry/

DevX101 6y ago

I recognized this a couple years ago at a startup I work with. Comparing Google Analytics numbers to validated event logs, the numbers were off by ~20-30%. Surely there must be a quick workaround I thought, there's no way there's an entire multi-billion $ industry of 3rd party analytics software giving bogus numbers to websites?! But that's indeed the case. I immediately made top priority building out an in-house analytics platform where event logs were sent via the API and thus didn't get blocked. And for those saying relative direction is all that matters, I guarantee you the behavior of users with adblocker installed is very different from those who can't be bothered or don't know how.

sharkweek 6y ago
Let me tell you what was fun - trying to explain to a former boss why Facebook ads showed one number for amount of traffic sent, Google Analytics showed a different number for traffic from those ads, then lastly our server logs showing an entirely different number!
tiffanyh 6y ago
For clarity, so you’d recommend web server logs over client side analytics? If so, what open source web server analytics tools do you believe is best? E.g. https://goaccess.io https://goaccess.io

pkaye 6y ago

What does parse.ly do differently to account for this discrepancy in the analytics?

pixelmonkey 6y ago
Good question. I explain some of the technical approaches we've taken with customers in another comment here: https://news.ycombinator.com/item?id=24205803 https://news.ycombinator.com/item?id=24205803

pixelmonkey 6y ago

I'm one of Parse.ly's co-founders. This post was written by one of our product managers about a project and investigation we've been doing for the past few months. It first got on my team's radar when I posted this set of tweets back in 2019: https://twitter.com/amontalenti/status/1165262620959617025 https://twitter.com/amontalenti/status/1165262620959617025 Specifically: I noticed a huge difference between the metrics we were reporting on my blog post in Parse.ly, and the metrics being reported by my personal blog's Cloudflare CDN (caching the content). Ironically enough, this traffic was all coming from HN and the post was itself about modern JavaScript[1]. Since then, we've also been hearing from a lot of customers about various scenarios where traffic is either under-counted or mis-counted. For example, something that has been tripping us up lately is that our Twitter integration relies (partially) upon the official t.co link shortener[2], and yet, due to modern browser rules related to W3C Referrer Policy[3], the t.co link's path segment is often not transmitted to the analytics provider, and thus the source tweet for traffic cannot be easily ascertained. I firmly believe in privacy and analytics without compromise[4], so the team is trying to come up with ways to at least quantify shadow traffic at an aggregate level, and to ensure legitimate user privacy interests are honored, while making sure they don't break legitimate privacy-safe first-party analytics use cases. As a developer, something that concerned me recently was realizing that Sentry, the open source error tracking tool with a SaaS reporting frontend and a JavaScript SDK[5], gets blocked in many conservative browser privacy setups. Though the interest to user privacy is legitimate, I think we can all agree it'd be better for site/app operators to know when certain browsers are hitting JavaScript stack traces. [1]: https://news.ycombinator.com/item?id=20785616 https://news.ycombinator.com/item?id=20785616 [2]: https://help.twitter.com/en/using-twitter/url-shortener https://help.twitter.com/en/using-twitter/url-shortener [3]: https://www.w3.org/TR/referrer-policy/ https://www.w3.org/TR/referrer-policy/ [4]: https://blog.parse.ly/post/3394/analytics-privacy-without-compromise/ https://blog.parse.ly/post/3394/analytics-privacy-without-co... [5]: https://sentry.io/for/javascript/ https://sentry.io/for/javascript/

MauranKilom 6y ago
So what do you actually (want to) do in regards to measuring shadow traffic? The blog post tries to convince the reader they should care about shadow traffic and then handwaves away existing solutions, but "we're developing a solution" is as concrete as the post gets as to why I should turn to parse.ly. Now you say "the team is trying to come up with ways to at least quantify shadow traffic at an aggregate level", so it appears that you don't even have a solution yet. In light of that, presenting "existing analytics services like parse.ly" as one of three solutions on "how to measure shadow traffic" seems borderline disingenuous. If you can do it, why not say so plain and clear? If you can't do it, why do you mention yourself as a solution? Or is it only other services like parse.ly that can do it? It also rubs me the wrong way how both the blog post and your comment has an undertone of "it would be better if users didn't have as much tracking protection". Just take the framing of your last sentence as an example...

thinkloop 6y ago

Does hosting client-side tracking on your own domain circumvent all the problems? How come that hasn't become the standard and killed 3rd-party trackers? If it's a question of having to manage an analytics platform, can't that still be deferred to a 3rd-party but through your own subdomain?

derivagral 6y ago
My interpretation of how Google Analytics does this is that pulling in a 3rd party dep lets them keep a lot of control over versioning and similar things. I assume they do this so out-of-date or vulnerable code (or extra tracking, sure) can be cleanly and quickly applied through the network and not rely on the clients to properly update "ga.js" a few times a day.

ChuckMcM 6y ago

Okay, the cynic in me wants to write "New Age web designers are stumped by lack of analytics while still refusing to look at their HTTP server log data." I remember when the ONLY analytics were those you could derive by analyzing your http logs. Which have useful information in them. Things like source IP address (which can be geo tagged), a bunch of HTTP headers (which are full of information too), and a timestamp which tells you when it came in and from where. Not to mention session cookies which take zero javascript to implement. I've been retooling my site slowly to only use these analytics (less the cookies) because I value people's privacy while browsing as much as my own. During the transition I've been comparing what I can pull out of the logs vs what Google's analytics gives me. Sure, Google can do wonders, especially if the person is coming from a browser where they are logged into Google. But, as the article points out, they miss everyone running noscript and/or other privacy enhancers like Privacy Badger from EFF. I don't feel like I'm going to miss the Google added insights.

neonate 6y ago
This is pretty funny, buried in the middle of the article: > Option 2 – Server-Side Tracking
- ChuckMcM 6y ago
  With advice that this is so technically complex, don't even look there. These guys can write java script that animates a web page so that it looks like a turning page and its "too technical" to pull data out of a server log?
- LaundroMat 6y ago
  The article talks about sending first-party analytics events to an analytics provider from your own servers. So the server-side tracking the article refers to is similar to the server-side tagging Google recently announced[1], not the analysis of server logs. [1] https://developers.google.com/tag-manager/serverside https://developers.google.com/tag-manager/serverside
  - neonate 6y ago

paulchap 6y ago

Funnily enough, I got a warning telling me uBlock prevented this page from loading...

billyhoffman 6y ago

I get wanting to create valuable thought leadership content, but this is the worst example of Bad product marketing: 1- present a new concept to readers (shadow visitors) 2- show how this concept is scary and bad for your business (your analytics is off by 20%!!!) 3- present 2 options, which by the way are free, but immediately shit all over them (Server logs! But that’s hard and complicated! Edge logs! But getting these are hard!) 4- present your company’s product as option 3, which surprisingly, have no downsides and isnt shit upon 5- profit What disingenuous garbage. You should be ashamed Parse.ly. The right way to do this is do steps 1 and 2. Then show in detail step 3: how to solve the problem with easy options, ideally with free and open software. It’s ok to show edge cases, corner cases, or just shear scale issues that makes these options challenging. The difference is that good product marketing pieces show people how to solve the problem and offer a solution to do that at scale or in a automated/hosted way so the customer doesn’t have to deal with it. If your product marketing content’s message is “you are screwed unless you buy our product “ you are doing it wrong

MauranKilom 6y ago
Yeah, this article just read super weird to anyone with even a basic understanding of the field. It's very clearly targeted at managerial folk, and even then it is as shallow in its arguments as it is transparent in its motives.

indymike 6y ago

Server side tracking is useful for logging http requests. Client side tracking is useful for logging user interactions. Used to be there would be a small difference between server and client due to caches and user settings... but... Modern apps (i.e. React, Vue, Angular) often only load one page, and then all interaction is managed by client side code, so often client side tracking is the only thing that works.