9 ms·

I wish we could just go back to "analytics" being an moderately complicated awk invocation on whatever logs your webserver produces.

frenkel 5y ago
Why can’t we?
- gentleman11 5y ago
  What percent of visits are bots and how do you detect them serverside? That’s the only part I’m unsure of
  - yjftsjthsd-h 5y ago
    Can you detect bots client side? I sorta assumed they used headless Chrome and looked like real users nowadays, although I guess with effort you can probably fingerprint?
    - thayne 5y ago
      You can if you track all of the user's online activity and use machine learning to determine if the behavior seems genuine. That's how invisible captcha works, I'd be surprised if google analytics didn't do something similar. Of course, that does mean you have to track all the user's online activity....
      - NmAmDa 5y ago
        What about GDPR potential violation?
        adrianN 5y ago
        GDPR has exceptions for fraud detection.
        Nextgrid 5y ago
        A bot does not immediately imply fraud. If you're processing personal data to detect bots to prevent let's say card fraud, fair enough. If you're processing personal data to detect bots for analytics purposes, this might not fly. Of course, this assumes someone actually gives a shit about enforcing the GDPR, which is currently not the case.
        adrianN 5y ago
        I have literally worked on bot detection and the high powered company lawyers told me that it's okay for my code to collect and analyze PII for these purposes because GDPR has these exceptions. I'm not a lawyer myself though, so that might have been wrong.
    - jerven 5y ago
      For one of the sites I work with. Headless chrome defenitly inflates the GA user numbers. We can spot this because they tend to come from big cloud data centre locations when doing ip to geo mapping.
      - yjftsjthsd-h 5y ago
        > We can spot this because they tend to come from big cloud data centre locations when doing ip to geo mapping. Of course, you can do that with server side logs, so in this case client side is less accurate:)
    - NavinF 5y ago
      The default build of chromedriver is pretty easy to detect client side because it injects js variables with known names. Most bots don't really care if they're detected and excluded from analytics. Unfortunately a few sites unfairly throttled me to 1 request/hour despite the cost of my requests being fuck-all. So a couple of years ago I had to randomize those variable names, distribute requests over 64 IPs, and screw up all their analytics numbers in the process. I hope it was worth saving $0.01 for them.
    - adrianN 5y ago
      Most bots are pretty stupid, so it's easy to detect them. Bot detection need not be 100% accurate. It's an arms race with the goal to make writing bots financially unattractive.
  - digitalengineer 5y ago
    This can help: https://www.darwin.so/features/bots https://www.darwin.so/features/bots
  - inetknght 5y ago
    > What percent of visits are bots It doesn't matter. > how do you detect them serverside? Instead of trying to detect whether something isn't human you should be trying to detect whether something is malicious.
- darig 5y ago
  because developers don't know how to build anything anymore. they paint themselves into smaller and smaller focuses relying on someone else to provide them with everything they need. no more web developers... now we have front-end and back-end all fighting over who needs to do the architecture and build the necessary tooling.
rognjen 5y ago
If I need accurate traffic numbers (for performance or server sizing for instance) I use GoAccess (https://goaccess.io/ https://goaccess.io/)
deleted 5y ago
[deleted]
foobar33333 5y ago
And then a user sends a support request with a screenshot showing an in browser error and demands you fix it but your server logs show nothing because this was a JS issue. While people who use frontend logging/tracking just search the user ID and see what went wrong.
- tinus_hn 5y ago
  In Google Analytics?
- matsemann 5y ago
  That has nothing to do with external visitor analytics, though. I've implemented log sending from frontend to backend multiple times. First party, same endpoints as other requests. Not getting blocked.
- lucb1e 5y ago
  To me, one doesn't have to pre-emptively record every user session on the off chance that they run into a problem large enough to bother contacting support for. Having a 'copy error' button and/or logging the errors to a server seems sufficient here, no need to hire the biggest brother out there to track your users for that.
Tenoke 5y ago
This doesn't work too well nowadays since a lot of visits don't hit my server as Cloudflare itself is serving them the page.
- Nextgrid 5y ago
  In this case you'd get your logs from CF instead of your origin server.
maple3142 5y ago
It doesn't work if the website is hosted on GitHub Pages or Netlify. Having a CDN such as CloudFlare will make log files on your server useless too.
- bruce343434 5y ago
  How so? Isn't that just for pictures and other media?
  - maple3142 5y ago
    Hosting a (static) website on GitHub pages means you put all your resources including html files into a repo, then the whole website will be served from GitHub's server. You don't even have a server to view your access log. Netlify is pretty similar. As for CloudFlare, all the requests will go through CloudFlare's server for caching, so some request won't reach your server.
    - xuki 5y ago
      > As for CloudFlare, all the requests will go through CloudFlare's server for caching, so some request won't reach your server. Only for static files, they don't cache HTML and dynamic contents. So you can still make sense of unique visitors to your site.
      - viraptor 5y ago
        That really depends on your setup. The cache whatever you tell them to cache. HTML and dynamic content is also cacheable with good results. For example you can vary the cache by the user ID cookie. Or for dynamic content you can in most cases do caching for 5min or so.
- realityking 5y ago
  Netlify offers an add-on for server-side analytics: https://www.netlify.com/products/analytics/ https://www.netlify.com/products/analytics/
wdr1 5y ago
Turning impression logs into people counts turns out to be a pretty difficult problem. Even with cookies it's still very complicated. Facebook has publicly acknowledged screwing it up several times. We could go back to the days of talking about "hits" on a website, but for most things where you'd care about these types of metrics, it's a pretty crap metric.

frenkel 5y ago

Why can’t we?

gentleman11 5y ago
What percent of visits are bots and how do you detect them serverside? That’s the only part I’m unsure of
- yjftsjthsd-h 5y ago
  Can you detect bots client side? I sorta assumed they used headless Chrome and looked like real users nowadays, although I guess with effort you can probably fingerprint?
  - thayne 5y ago
    You can if you track all of the user's online activity and use machine learning to determine if the behavior seems genuine. That's how invisible captcha works, I'd be surprised if google analytics didn't do something similar. Of course, that does mean you have to track all the user's online activity....
    - NmAmDa 5y ago
      What about GDPR potential violation?
      - adrianN 5y ago
        GDPR has exceptions for fraud detection.
        Nextgrid 5y ago
        A bot does not immediately imply fraud. If you're processing personal data to detect bots to prevent let's say card fraud, fair enough. If you're processing personal data to detect bots for analytics purposes, this might not fly. Of course, this assumes someone actually gives a shit about enforcing the GDPR, which is currently not the case.
        adrianN 5y ago
        I have literally worked on bot detection and the high powered company lawyers told me that it's okay for my code to collect and analyze PII for these purposes because GDPR has these exceptions. I'm not a lawyer myself though, so that might have been wrong.
  - jerven 5y ago
    For one of the sites I work with. Headless chrome defenitly inflates the GA user numbers. We can spot this because they tend to come from big cloud data centre locations when doing ip to geo mapping.
    - yjftsjthsd-h 5y ago
      > We can spot this because they tend to come from big cloud data centre locations when doing ip to geo mapping. Of course, you can do that with server side logs, so in this case client side is less accurate:)
  - NavinF 5y ago
    The default build of chromedriver is pretty easy to detect client side because it injects js variables with known names. Most bots don't really care if they're detected and excluded from analytics. Unfortunately a few sites unfairly throttled me to 1 request/hour despite the cost of my requests being fuck-all. So a couple of years ago I had to randomize those variable names, distribute requests over 64 IPs, and screw up all their analytics numbers in the process. I hope it was worth saving $0.01 for them.
  - adrianN 5y ago
    Most bots are pretty stupid, so it's easy to detect them. Bot detection need not be 100% accurate. It's an arms race with the goal to make writing bots financially unattractive.
- digitalengineer 5y ago
  This can help: https://www.darwin.so/features/bots https://www.darwin.so/features/bots
- inetknght 5y ago
  > What percent of visits are bots It doesn't matter. > how do you detect them serverside? Instead of trying to detect whether something isn't human you should be trying to detect whether something is malicious.
darig 5y ago
because developers don't know how to build anything anymore. they paint themselves into smaller and smaller focuses relying on someone else to provide them with everything they need. no more web developers... now we have front-end and back-end all fighting over who needs to do the architecture and build the necessary tooling.

rognjen 5y ago

If I need accurate traffic numbers (for performance or server sizing for instance) I use GoAccess (https://goaccess.io/ https://goaccess.io/)

deleted 5y ago

[deleted]

foobar33333 5y ago

And then a user sends a support request with a screenshot showing an in browser error and demands you fix it but your server logs show nothing because this was a JS issue. While people who use frontend logging/tracking just search the user ID and see what went wrong.

tinus_hn 5y ago
In Google Analytics?
matsemann 5y ago
That has nothing to do with external visitor analytics, though. I've implemented log sending from frontend to backend multiple times. First party, same endpoints as other requests. Not getting blocked.
lucb1e 5y ago
To me, one doesn't have to pre-emptively record every user session on the off chance that they run into a problem large enough to bother contacting support for. Having a 'copy error' button and/or logging the errors to a server seems sufficient here, no need to hire the biggest brother out there to track your users for that.

Tenoke 5y ago

This doesn't work too well nowadays since a lot of visits don't hit my server as Cloudflare itself is serving them the page.

Nextgrid 5y ago
In this case you'd get your logs from CF instead of your origin server.

maple3142 5y ago

It doesn't work if the website is hosted on GitHub Pages or Netlify. Having a CDN such as CloudFlare will make log files on your server useless too.

bruce343434 5y ago
How so? Isn't that just for pictures and other media?
- maple3142 5y ago
  Hosting a (static) website on GitHub pages means you put all your resources including html files into a repo, then the whole website will be served from GitHub's server. You don't even have a server to view your access log. Netlify is pretty similar. As for CloudFlare, all the requests will go through CloudFlare's server for caching, so some request won't reach your server.
  - xuki 5y ago
    > As for CloudFlare, all the requests will go through CloudFlare's server for caching, so some request won't reach your server. Only for static files, they don't cache HTML and dynamic contents. So you can still make sense of unique visitors to your site.
    - viraptor 5y ago
      That really depends on your setup. The cache whatever you tell them to cache. HTML and dynamic content is also cacheable with good results. For example you can vary the cache by the user ID cookie. Or for dynamic content you can in most cases do caching for 5min or so.
realityking 5y ago
Netlify offers an add-on for server-side analytics: https://www.netlify.com/products/analytics/ https://www.netlify.com/products/analytics/

wdr1 5y ago

Turning impression logs into people counts turns out to be a pretty difficult problem. Even with cookies it's still very complicated. Facebook has publicly acknowledged screwing it up several times. We could go back to the days of talking about "hits" on a website, but for most things where you'd care about these types of metrics, it's a pretty crap metric.