3 ms·

For quite a long time I've been using mostly open-source utilities. And for some time I also wanted to create some kind of utility of my own and to contribute i

For quite a long time I've been using mostly open-source utilities. And for some time I also wanted to create some kind of utility of my own and to contribute into the open source world. One year ago, in April, I had an idea to create my own web analytics service, which would be much more private and provide alternative functionality of Google Analytics. Meet Swetrix - a completely opensource, cookieless, self-hosted and privacy-focused web analytics platform. This service allows you to track many different metrics such as user country, device type, browser, and more. Apply filters to the data, extensively customise your tracking options and your website settings. Feel free to check out our live demo! :) The API is written in NestJS, as databases I use MySQL, Redis and Clickhouse. Frontend (and the browser extensions) are written in React. I learnt a lot about analytics and coding while working on this project!

phillipseamore 4y ago
What I find missing from all the OSS analytics is log parsing agents that I can install on my servers for sites that I don't want to have JS on. The agent would parse the logs periodically (every hour from cron or whatever) and send the parsed/analyzed data to the server handling analytics. This could also augment data for sites that primarily use a JS tracker by adding information on users that have JS disabled (or users on such a bad connection that they leave before the load finishes) as well as bots, uptime checkers etc.
- blaumaus 4y ago
  Actually this sounds reasonable to add, I added this feature to my backlog, it won't be too hard to implement it. By the way, as for now we also support no-JS tracking via 1x1 px GIF.
- sdoering 4y ago
  You might take a look at GoAccess [0]. > GoAccess is an open source real-time web log analyzer and interactive viewer that runs in a terminal in *nix systems or through your browser. [0]: https://goaccess.io/ https://goaccess.io/
ignasv 4y ago
How are MySQL, Redis and Clickhouse used? Which usecase each of them cover?
- blaumaus 4y ago
  MySQL is used for data like user accounts and linked websites. Clickhouse is used for the analytics data (which this DBMS is basically designed for). Redis is used for caching (for example, I store there all the incoming analytics requests, and each minute the cron job takes them all and adds into the Clickhouse DB, as a big chunk), rate limiting, etc.

phillipseamore 4y ago

What I find missing from all the OSS analytics is log parsing agents that I can install on my servers for sites that I don't want to have JS on. The agent would parse the logs periodically (every hour from cron or whatever) and send the parsed/analyzed data to the server handling analytics. This could also augment data for sites that primarily use a JS tracker by adding information on users that have JS disabled (or users on such a bad connection that they leave before the load finishes) as well as bots, uptime checkers etc.

blaumaus 4y ago
Actually this sounds reasonable to add, I added this feature to my backlog, it won't be too hard to implement it. By the way, as for now we also support no-JS tracking via 1x1 px GIF.
sdoering 4y ago
You might take a look at GoAccess [0]. > GoAccess is an open source real-time web log analyzer and interactive viewer that runs in a terminal in *nix systems or through your browser. [0]: https://goaccess.io/ https://goaccess.io/

ignasv 4y ago

How are MySQL, Redis and Clickhouse used? Which usecase each of them cover?

blaumaus 4y ago
MySQL is used for data like user accounts and linked websites. Clickhouse is used for the analytics data (which this DBMS is basically designed for). Redis is used for caching (for example, I store there all the incoming analytics requests, and each minute the cron job takes them all and adds into the Clickhouse DB, as a big chunk), rate limiting, etc.