3 ms·

GoAccess – Open-source real-time log analyzer and interactive viewer

71 points by gregsadetsky 2mo ago

vbernat 2mo ago
It is now mostly unusable for web analytics. So many scrapers are pretending to be browsers. On my site, there is a factor 10 with what I consider an honest view of human visitors.
- janvdberg 2mo ago
  Yes, this is unfortunately my experience with GoAccess
- joshyi 2mo ago
  If you're using it to serve the marketing team, then yeah. We use it more for server performance and metrics that cookie-cutter JS analytics tools can't really provide. btw, JS-based analytics are no longer as accurate for identifying "real humans" as they used to be. bots have gotten much better at behaving like a person behind a keyboard, so it's increasingly easy for them to make your analytics tool think they're a real user
  - vbernat 2mo ago
    This is not untrue, but I suppose that if you don't prevent bots from accessing your website, they won't try to use the more advanced scrapers running JS and won't trigger your analytics.
- herbst 2mo ago
  When customers ask me for visitor metrics I tend to explain that it doesn't matter anymore before I show them them the highly inflated absolutely unrealistic numbers.
WesolyKubeczek 2mo ago
Unfortunately, each time a new log analyzer appears, it seems to only be analyzing web access logs in combined format.
- tecleandor 2mo ago
  Note that GoAccess is more than 15 years old.
  - joshyi 2mo ago
    and honestly, there's still no other tool that does what GoAccess does quite as well. Most of the alternatives are cookie-cutter, JS-based Google Analytics wannabes. And with bots becoming much smarter and more prevalent when accessing websites, relying solely on JS-based tracking isn't as trustworthy anymore. We use GoAccess to capture what JS-based analytics can't, e.g., server side perf and traffic data. It's not meant to serve the marketing team though.
    - aleks_me2 2mo ago
      [dead]
- tredre3 2mo ago
  GoAccess supports more than combined format. I think they even support stubborn servers who output JSON logs only: https://goaccess.io/man#custom-log https://goaccess.io/man#custom-log
aleks_me2 2mo ago
I've revived AWFFull for static log analysis, just like in the good old webalizer days :-). Link to the report is in my bio, sorry, I can't post links here for now. What I'm seeing matches the observations of the other commenters here exactly: most of the "users" are bots harvesting pages and PDFs. My solution for getting at the real users is GoatCounter.
iJohnDoe 2mo ago
This always looks cool and the real-time aspect always gets attention. However, If I remember correctly, it was kind of a burden to get deployed. Again, if I remember correctly, it was able to make reports for AWS Cloudfront logs, which was actually useful.
- aleks_me2 2mo ago
  The question I always come back to is: how long does it take you to get stats for something like 1 month or 1 quarter with the current "modern" tools? For ad hoc analytics these tools are very nice, but for long-term stats they run into timeouts, because the backend needs a lot of time to collect the data and build the reports for ~1 month or a quarter — or am I wrong about that?
- QuantumNomad_ 2mo ago
  > If I remember correctly, it was kind of a burden to get deployed It probably depends a lot on your setup. I’ve been running GoAccess from a crontab on some of my sites for a couple of years. It was a bit fiddly to get the correct combination of Nginx logs, forwarding real IP from HAProxy, and getting it to show with GeoIP in generated GoAccess HTML reports, but once I got it working satisfactorily it’s been working well since. The only part that is missing in my GoAccess reports is the ASN of the clients. I don’t remember if that’s because ASN mappings are a paid GeoIP database feature or if I simply ignored that part or what, but I do have countries and municipalities correct (I can tell because for example on one of the sites the most frequent visitor is a client machine that I own, and it shows up accordingly prominently with the country and municipality where that client machine is located) and that’s the main thing I wanted to be able to see when setting up GoAccess to use a GeoIP database file. I serve the generated GoAccess HTML reports from Nginx itself with a simple .htaccess file providing username and hashed password and Nginx using that file with HTTP Basic Auth. Since the only person that looks at those reports is myself, and the sites use TLS, I don’t need anything more advanced than HTTP Basic Auth to protect access to the GoAccess HTML reports. And I suppose that in today’s world an LLM would probably be able to do most of the config writing for a setup such as mine for all of the above parts.
  - 2mo ago
evalystai 2mo ago
[flagged]
guessmyname 2mo ago
GoAccess is an awesome tool! If I recall correctly, it was developed as an alternative (Go port) of a popular log analyzer called lnav (https://lnav.org https://lnav.org), which is quite popular too: https://news.ycombinator.com/from?site=lnav.org https://news.ycombinator.com/from?site=lnav.org
- tstack 2mo ago
  > If I recall correctly, it was developed as an alternative (Go port) of a popular log analyzer Hmm, I think you might be mixing things up, goaccess is written in C and probably not written as an alternative to lnav.

71 points by gregsadetsky 2mo ago

It is now mostly unusable for web analytics. So many scrapers are pretending to be browsers. On my site, there is a factor 10 with what I consider an honest view of human visitors.

janvdberg 2mo ago
Yes, this is unfortunately my experience with GoAccess
joshyi 2mo ago
If you're using it to serve the marketing team, then yeah. We use it more for server performance and metrics that cookie-cutter JS analytics tools can't really provide. btw, JS-based analytics are no longer as accurate for identifying "real humans" as they used to be. bots have gotten much better at behaving like a person behind a keyboard, so it's increasingly easy for them to make your analytics tool think they're a real user
- vbernat 2mo ago
  This is not untrue, but I suppose that if you don't prevent bots from accessing your website, they won't try to use the more advanced scrapers running JS and won't trigger your analytics.
herbst 2mo ago
When customers ask me for visitor metrics I tend to explain that it doesn't matter anymore before I show them them the highly inflated absolutely unrealistic numbers.

WesolyKubeczek 2mo ago

Unfortunately, each time a new log analyzer appears, it seems to only be analyzing web access logs in combined format.

tecleandor 2mo ago
Note that GoAccess is more than 15 years old.
- joshyi 2mo ago
  and honestly, there's still no other tool that does what GoAccess does quite as well. Most of the alternatives are cookie-cutter, JS-based Google Analytics wannabes. And with bots becoming much smarter and more prevalent when accessing websites, relying solely on JS-based tracking isn't as trustworthy anymore. We use GoAccess to capture what JS-based analytics can't, e.g., server side perf and traffic data. It's not meant to serve the marketing team though.
  - aleks_me2 2mo ago
    [dead]
tredre3 2mo ago
GoAccess supports more than combined format. I think they even support stubborn servers who output JSON logs only: https://goaccess.io/man#custom-log https://goaccess.io/man#custom-log

aleks_me2 2mo ago

I've revived AWFFull for static log analysis, just like in the good old webalizer days :-). Link to the report is in my bio, sorry, I can't post links here for now. What I'm seeing matches the observations of the other commenters here exactly: most of the "users" are bots harvesting pages and PDFs. My solution for getting at the real users is GoatCounter.

iJohnDoe 2mo ago

This always looks cool and the real-time aspect always gets attention. However, If I remember correctly, it was kind of a burden to get deployed. Again, if I remember correctly, it was able to make reports for AWS Cloudfront logs, which was actually useful.

aleks_me2 2mo ago
The question I always come back to is: how long does it take you to get stats for something like 1 month or 1 quarter with the current "modern" tools? For ad hoc analytics these tools are very nice, but for long-term stats they run into timeouts, because the backend needs a lot of time to collect the data and build the reports for ~1 month or a quarter — or am I wrong about that?
QuantumNomad_ 2mo ago
> If I remember correctly, it was kind of a burden to get deployed It probably depends a lot on your setup. I’ve been running GoAccess from a crontab on some of my sites for a couple of years. It was a bit fiddly to get the correct combination of Nginx logs, forwarding real IP from HAProxy, and getting it to show with GeoIP in generated GoAccess HTML reports, but once I got it working satisfactorily it’s been working well since. The only part that is missing in my GoAccess reports is the ASN of the clients. I don’t remember if that’s because ASN mappings are a paid GeoIP database feature or if I simply ignored that part or what, but I do have countries and municipalities correct (I can tell because for example on one of the sites the most frequent visitor is a client machine that I own, and it shows up accordingly prominently with the country and municipality where that client machine is located) and that’s the main thing I wanted to be able to see when setting up GoAccess to use a GeoIP database file. I serve the generated GoAccess HTML reports from Nginx itself with a simple .htaccess file providing username and hashed password and Nginx using that file with HTTP Basic Auth. Since the only person that looks at those reports is myself, and the sites use TLS, I don’t need anything more advanced than HTTP Basic Auth to protect access to the GoAccess HTML reports. And I suppose that in today’s world an LLM would probably be able to do most of the config writing for a setup such as mine for all of the above parts.
- 2mo ago

evalystai 2mo ago

[flagged]

guessmyname 2mo ago

GoAccess is an awesome tool! If I recall correctly, it was developed as an alternative (Go port) of a popular log analyzer called lnav (https://lnav.org https://lnav.org), which is quite popular too: https://news.ycombinator.com/from?site=lnav.org https://news.ycombinator.com/from?site=lnav.org

tstack 2mo ago
> If I recall correctly, it was developed as an alternative (Go port) of a popular log analyzer Hmm, I think you might be mixing things up, goaccess is written in C and probably not written as an alternative to lnav.