4 ms·

I get a LOT of bot traffic with bogus user agents. Google Analytics does an okayish job dealing with it. Looking at web server logs and trying to reconstruct tr

by eli 6y ago

I get a LOT of bot traffic with bogus user agents. Google Analytics does an okayish job dealing with it. Looking at web server logs and trying to reconstruct traffic patterns from just real users would be real chore.

Seirdy 6y ago
I mean, that sounds like more of a feature than a bug. You shouldn't know exactly what people are doing on your site (too invasive), but this approach gives you a rough idea of which requests are being made. I personally find it to still be too revealing in some cases, which is why tools like the nginx IPscrub module[0] should be the upstream default IMO. [0]: https://github.com/masonicboom/ipscrub https://github.com/masonicboom/ipscrub
- sofixa 6y ago
  > mean, that sounds like more of a feature than a bug. You shouldn't know exactly what people are doing on your site (too invasive), but this approach gives you a rough idea of which requests are being made Why not? Why wouldn't you know what people are doing on your website, within limits? Which section is getting most visits, which is the slowest, how many elements people click on in your lists, etc etc can have immense value in deciding what to do with your site, what needs work, what doesn't work at all, etc.
  - Seirdy 6y ago
    I typically see software from the perspective of software freedom. When a user downloads a web page from my server, I can temporarily record that an anonymous user made a request because I'm only logging what the software I'm running on my machine is doing. After that, the user should own that downloaded copy of the page and use it freely without surveillance from me or anyone else. The only restrictions in place are those set by the license (hopefully AGPLv3). Beyond those restrictions, the software doesn't belong to me anymore so I don't collect metrics. It's true that I won't be able to get detailed metrics about what the user is doing, but it's also true that user's copy of my software will only be serving the user directly. Web apps aren't different from other software; telemetry and user studies should be opt-in, transparent, and sent back only with informed consent. Only then can I consider running analytics.
kevin_thibedeau 6y ago
I abandoned GA because they wouldn't do anything to fix the problem with referer spam. Unless they've finally fixed it, a lot of the stats they produce are from non-users.
- eli 6y ago
  It’s still not awesome if I’m honest. But at least you’re starting from clients that speak JS instead of literally every GET request.
  - account42 6y ago
    > clients that speak JS Which also excludes real users, especially if the requirement is "allows JS from google analytics domains".
    - eli 6y ago
      Sure, I'm under no illusion the metrics are perfect but I'm very confident that my non-JS traffic is over 99% automated.

by eli 6y ago

Seirdy 6y ago

I mean, that sounds like more of a feature than a bug. You shouldn't know exactly what people are doing on your site (too invasive), but this approach gives you a rough idea of which requests are being made. I personally find it to still be too revealing in some cases, which is why tools like the nginx IPscrub module[0] should be the upstream default IMO. [0]: https://github.com/masonicboom/ipscrub https://github.com/masonicboom/ipscrub

sofixa 6y ago
> mean, that sounds like more of a feature than a bug. You shouldn't know exactly what people are doing on your site (too invasive), but this approach gives you a rough idea of which requests are being made Why not? Why wouldn't you know what people are doing on your website, within limits? Which section is getting most visits, which is the slowest, how many elements people click on in your lists, etc etc can have immense value in deciding what to do with your site, what needs work, what doesn't work at all, etc.
- Seirdy 6y ago
  I typically see software from the perspective of software freedom. When a user downloads a web page from my server, I can temporarily record that an anonymous user made a request because I'm only logging what the software I'm running on my machine is doing. After that, the user should own that downloaded copy of the page and use it freely without surveillance from me or anyone else. The only restrictions in place are those set by the license (hopefully AGPLv3). Beyond those restrictions, the software doesn't belong to me anymore so I don't collect metrics. It's true that I won't be able to get detailed metrics about what the user is doing, but it's also true that user's copy of my software will only be serving the user directly. Web apps aren't different from other software; telemetry and user studies should be opt-in, transparent, and sent back only with informed consent. Only then can I consider running analytics.

kevin_thibedeau 6y ago

I abandoned GA because they wouldn't do anything to fix the problem with referer spam. Unless they've finally fixed it, a lot of the stats they produce are from non-users.

eli 6y ago
It’s still not awesome if I’m honest. But at least you’re starting from clients that speak JS instead of literally every GET request.
- account42 6y ago
  > clients that speak JS Which also excludes real users, especially if the requirement is "allows JS from google analytics domains".
  - eli 6y ago
    Sure, I'm under no illusion the metrics are perfect but I'm very confident that my non-JS traffic is over 99% automated.