4 ms·

That's why I prefer goaccess(1) over any other tool that uses client side Javascript. It's basically a real time website log analyser which gives you enough in

That's why I prefer goaccess(1) over any other tool that uses client side Javascript. It's basically a real time website log analyser which gives you enough information to know whats happening on your website but doesn't require any pesky Javascript etc to do it. Also since it is works by analysing your log files it can never be blocked. (1) https://rt.goaccess.io/?20210826211303 https://rt.goaccess.io/?20210826211303

scrollaway 5y ago
By nature it won't filter out bots, however. One of my clients' website traffic is composed of over 75% bot traffic. Server-side logs are unusable for anything other than site performance.
- dschooh 5y ago
  > By nature it won't filter out bots, however. I have two questions about this: - Since you know they are bots, why couldn't you filter them? - On the other hand, couldn't there be bots that run JavaScript which would be tracked client-side?
  - scrollaway 5y ago
    > Since you know they are bots, why couldn't you filter them? A decent amount can be filtered by UA. This is inconvenient because UA is a very large piece to log and index on, so you need to do UA processing to do anything useful with it and … well by that point it just becomes a chore and I suspect there's good logging services that do this better than you'd spend your time doing yourself. > On the other hand, couldn't there be bots that run JavaScript which would be tracked client-side? They exist but they're more rare by nature, because running JS at bot scale is expensive.
    - dschooh 5y ago
      > well by that point it just becomes a chore and I suspect there's good logging services that do this better than you'd spend your time doing yourself. Which is why the OP suggested using tools like goaccess.
- qualudeheart 5y ago
  Can I evade tracking by disgusing my traffic as bot traffic? Sounds like a startup idea.
  - ignoramous 5y ago
    You're likely going to end up implementing a poor man's Tor.
  - scrollaway 5y ago
    Oh I suspect you'll have a decent amount of sites not serving you ads or analytics if you put "googlebot" or "chrome lighthouse" in your UA.
    - faeyanpiraat 5y ago
      If you use googlebot you’ll likely just get blocked altogether. Using Lighthouse might be an interesting experiment though.
- superasn 5y ago
  Goaccess does have a separate panel for Crawlers/Bots(1) but I think it's based on the UA. There is also an option to ignore IPs/Referrers too which can work very well for such cases. (1) https://goaccess.io/features#:~:text=Spot%20aggressive%20hosts/bots https://goaccess.io/features#:~:text=Spot%20aggressive%20hos...
- Cyberdog 5y ago
  > One of my clients' website traffic is composed of over 75% bot traffic. Server-side logs are unusable for anything other than site performance. I'm unclear how broad you intend that second sentence to be, but there's still a ton of info you can glean just from server-side logs: - Referrer info, and, by extension, popular search terms being used to find your site; - Paths on your site causing 5xx errors (so pages which might be triggering an error in a server-side script) - Paths on your site causing 4xx errors, and associated referrers (might be broken links on your own site; might be stale search engine indexing) - Mobile vs desktop access statistics Finding this data among a bunch of bot-induced noise might be annoying, but if they're good bots and sending proper UA headers specifying their botness, it's easy enough to filter out. Even otherwise there might be typical bot-like behavior you can find and account for such as not sending a referrer header or trying to access known exploitable PHP scripts (in which case you should block that IP address for a few hours or days - there are programs which can do this sort of thing automatically but frustratingly I can't recall the name of one off the top of my head right now). Granted, a lot of this can be spoofed, but I'm pretty sure the number of people sending spoofed referral or UA headers is dwarfed by the number of those (like me) who block Google Analytics and similar cruft entirely.
  - scrollaway 5y ago
    No, you're right, I shouldn't have written something so dismissive. (I do include error tracing as part of "performance" for what it's worth but those have their own system from within the app itself) Frankly I would love to see some serious low-config solutions to analyzing server-side logs. Oh, especially Fastly. Client in question uses Fastly and it blew my mind to find out that there was nothing in place to answer simple questions such as "what are the slowest paths to respond", "which paths are a cache hit most often", "which paths are most hit overall", etc. And being able to look at various dimensions such as browser, bot traffic, country of origin, etc. If you have suggestions…
    - Cyberdog 5y ago
      Any log analyzer will tell you which path is the most hit. For slowest paths, I think a server daemon could theoretically log how long it took to serve the page from request in to last byte out, but I don't know if any of them do that - you might have to set up a custom format for logging, and then from there you'd need to tell your analyzer how to interpret that field. For cache hits, I guess it'd depend on what sort of cache you have in mind, but that might be something you could only effectively log at the application level.
- AlexAndScripts 5y ago
  Can't you have a client side system do nothing but tell the server it's not a bot?
- marvinblum 5y ago
  The middle way is using something like Pirsch's API [0] from your backend. [0] https://pirsch.io/ https://pirsch.io/

scrollaway 5y ago

By nature it won't filter out bots, however. One of my clients' website traffic is composed of over 75% bot traffic. Server-side logs are unusable for anything other than site performance.

dschooh 5y ago
> By nature it won't filter out bots, however. I have two questions about this: - Since you know they are bots, why couldn't you filter them? - On the other hand, couldn't there be bots that run JavaScript which would be tracked client-side?
- scrollaway 5y ago
  > Since you know they are bots, why couldn't you filter them? A decent amount can be filtered by UA. This is inconvenient because UA is a very large piece to log and index on, so you need to do UA processing to do anything useful with it and … well by that point it just becomes a chore and I suspect there's good logging services that do this better than you'd spend your time doing yourself. > On the other hand, couldn't there be bots that run JavaScript which would be tracked client-side? They exist but they're more rare by nature, because running JS at bot scale is expensive.
  - dschooh 5y ago
    > well by that point it just becomes a chore and I suspect there's good logging services that do this better than you'd spend your time doing yourself. Which is why the OP suggested using tools like goaccess.
qualudeheart 5y ago
Can I evade tracking by disgusing my traffic as bot traffic? Sounds like a startup idea.
- ignoramous 5y ago
  You're likely going to end up implementing a poor man's Tor.
- scrollaway 5y ago
  Oh I suspect you'll have a decent amount of sites not serving you ads or analytics if you put "googlebot" or "chrome lighthouse" in your UA.
  - faeyanpiraat 5y ago
    If you use googlebot you’ll likely just get blocked altogether. Using Lighthouse might be an interesting experiment though.
superasn 5y ago
Goaccess does have a separate panel for Crawlers/Bots(1) but I think it's based on the UA. There is also an option to ignore IPs/Referrers too which can work very well for such cases. (1) https://goaccess.io/features#:~:text=Spot%20aggressive%20hosts/bots https://goaccess.io/features#:~:text=Spot%20aggressive%20hos...
Cyberdog 5y ago
> One of my clients' website traffic is composed of over 75% bot traffic. Server-side logs are unusable for anything other than site performance. I'm unclear how broad you intend that second sentence to be, but there's still a ton of info you can glean just from server-side logs: - Referrer info, and, by extension, popular search terms being used to find your site; - Paths on your site causing 5xx errors (so pages which might be triggering an error in a server-side script) - Paths on your site causing 4xx errors, and associated referrers (might be broken links on your own site; might be stale search engine indexing) - Mobile vs desktop access statistics Finding this data among a bunch of bot-induced noise might be annoying, but if they're good bots and sending proper UA headers specifying their botness, it's easy enough to filter out. Even otherwise there might be typical bot-like behavior you can find and account for such as not sending a referrer header or trying to access known exploitable PHP scripts (in which case you should block that IP address for a few hours or days - there are programs which can do this sort of thing automatically but frustratingly I can't recall the name of one off the top of my head right now). Granted, a lot of this can be spoofed, but I'm pretty sure the number of people sending spoofed referral or UA headers is dwarfed by the number of those (like me) who block Google Analytics and similar cruft entirely.
- scrollaway 5y ago
  No, you're right, I shouldn't have written something so dismissive. (I do include error tracing as part of "performance" for what it's worth but those have their own system from within the app itself) Frankly I would love to see some serious low-config solutions to analyzing server-side logs. Oh, especially Fastly. Client in question uses Fastly and it blew my mind to find out that there was nothing in place to answer simple questions such as "what are the slowest paths to respond", "which paths are a cache hit most often", "which paths are most hit overall", etc. And being able to look at various dimensions such as browser, bot traffic, country of origin, etc. If you have suggestions…
  - Cyberdog 5y ago
    Any log analyzer will tell you which path is the most hit. For slowest paths, I think a server daemon could theoretically log how long it took to serve the page from request in to last byte out, but I don't know if any of them do that - you might have to set up a custom format for logging, and then from there you'd need to tell your analyzer how to interpret that field. For cache hits, I guess it'd depend on what sort of cache you have in mind, but that might be something you could only effectively log at the application level.
AlexAndScripts 5y ago
Can't you have a client side system do nothing but tell the server it's not a bot?
marvinblum 5y ago
The middle way is using something like Pirsch's API [0] from your backend. [0] https://pirsch.io/ https://pirsch.io/