4 ms·

Huh, I have always wondered if for these usecases log-based analytics would suffice. No need to intrude on the client when the request itself already includes l

by c0balt 1y ago

Huh, I have always wondered if for these usecases log-based analytics would suffice. No need to intrude on the client when the request itself already includes lots of data. You get visited page, time of visit and a minimal user identifier (preferred locale/user agent, source IP). That should provide enough data for those questions. If you want to make it slightly more privacy friendly and cheaper only store aggregate data (visited page counts etc). The problem here is probably that logs are not always accessible (managed services like GitHub/GitLab pages).

jsheard 1y ago
The other problem with logs is that it's very difficult to filter out bots masquerading with browser user agents, of which there are a lot nowadays. I've watched the logs on newly registered domains that aren't published anywhere besides the certificate transparency logs and seen the majority of traffic coming from "Chrome". Yeah I'm sure you are.
- diggan 1y ago
  > The other problem with logs is that it's very difficult to filter out all of the bot traffic. It's not very difficult, but isn't not effort-less. Start with something like https://github.com/allinurl/goaccess/blob/master/config/browsers.list https://github.com/allinurl/goaccess/blob/master/config/brow... which captures 99% of the crawlers out there. Then, when you notice there is one particular user-agent/IP/IP-range doing a bunch of requests, add it to list and re-run. Doing filtering based on ASNs that you see are being used for crawling lets you filter most of the AI agents too. We've been dealing with this problem for over 2 decades now, and there are solutions out there that removes almost all of it from your logs.
  - lazide 1y ago
    Sounds about as much fun as manual spam filtering, but with worse tools. :(
    - diggan 1y ago
      Literally takes 5 minutes to setup at most, and most analytics tools ships with a "ignore webcrawlers" option somewhere, like goaccess does for example, taking 0 minutes to use :)
closewith 1y ago
At least in the EU, you still need consent to go down this path, so you end up rebuilding the existing analytics tools to manage that.
- Fire-Dragon-DoL 1y ago
  You don't: the ip, user agent header are necessary for you to serve the request and for you to protect from malicious usage. Both are fine use for gathering data under the current legislation. You can still convert those datasets in analytics.
  - closewith 1y ago
    No, you're wrong. You need consent to use IP address for analytics, even if lawfully collected for another reason under a different legal basis. Lots of precedent on this and it's clear cut.
    - Fire-Dragon-DoL 1y ago
      But you don't need to use the ip for analytics. Aggregate the data and throw it away.
      - closewith 1y ago
        You still need consent for that, as you processed the IP for that purpose and it is explicitly never allowed under other legal bases. Again, this is a completely settled matter in the EU, so you can easily look it up if your don't believe me.
        Fire-Dragon-DoL 1y ago
        Cool, then ignore the ip and just consider the amount of requests to a given endpoint. Counting that should be sufficient

by c0balt 1y ago

jsheard 1y ago

The other problem with logs is that it's very difficult to filter out bots masquerading with browser user agents, of which there are a lot nowadays. I've watched the logs on newly registered domains that aren't published anywhere besides the certificate transparency logs and seen the majority of traffic coming from "Chrome". Yeah I'm sure you are.

diggan 1y ago
> The other problem with logs is that it's very difficult to filter out all of the bot traffic. It's not very difficult, but isn't not effort-less. Start with something like https://github.com/allinurl/goaccess/blob/master/config/browsers.list https://github.com/allinurl/goaccess/blob/master/config/brow... which captures 99% of the crawlers out there. Then, when you notice there is one particular user-agent/IP/IP-range doing a bunch of requests, add it to list and re-run. Doing filtering based on ASNs that you see are being used for crawling lets you filter most of the AI agents too. We've been dealing with this problem for over 2 decades now, and there are solutions out there that removes almost all of it from your logs.
- lazide 1y ago
  Sounds about as much fun as manual spam filtering, but with worse tools. :(
  - diggan 1y ago
    Literally takes 5 minutes to setup at most, and most analytics tools ships with a "ignore webcrawlers" option somewhere, like goaccess does for example, taking 0 minutes to use :)

closewith 1y ago

At least in the EU, you still need consent to go down this path, so you end up rebuilding the existing analytics tools to manage that.

Fire-Dragon-DoL 1y ago
You don't: the ip, user agent header are necessary for you to serve the request and for you to protect from malicious usage. Both are fine use for gathering data under the current legislation. You can still convert those datasets in analytics.
- closewith 1y ago
  No, you're wrong. You need consent to use IP address for analytics, even if lawfully collected for another reason under a different legal basis. Lots of precedent on this and it's clear cut.
  - Fire-Dragon-DoL 1y ago
    But you don't need to use the ip for analytics. Aggregate the data and throw it away.
    - closewith 1y ago
      You still need consent for that, as you processed the IP for that purpose and it is explicitly never allowed under other legal bases. Again, this is a completely settled matter in the EU, so you can easily look it up if your don't believe me.
      - Fire-Dragon-DoL 1y ago
        Cool, then ignore the ip and just consider the amount of requests to a given endpoint. Counting that should be sufficient