3 ms·

I find it frustrating to search for stuff for mid-scale problems sometimes. There's a lack of context about what 'metrics' or 'analytics' means, and that leads

I find it frustrating to search for stuff for mid-scale problems sometimes. There's a lack of context about what 'metrics' or 'analytics' means, and that leads to over-engineered solutions. I run 2-3 (primarily database & web) servers. I thought it would be a nice idea to aggregate web accesses and stuff into a central location for analysis. It's probably 50-100MB of logs a day at most. But all the 'industry standard' solutions are for something at much bigger scale. Start with blah, spin that up on a k8s cluster, run this in docker, blah blah. FFS, I don't need that, I just want some nice graphs. So I just shove it all into a postgres database (since we are already familiar with it) and visualize it with Grafana. I'm sure people would complain that it's not the proper way, but I prefer spending my time solving my users problems, not self-inflicted ones.

sanderjd 3y ago
I feel this in my bones. I really find it hard to figure out the right solutions for scaling things that will be "bigger than small" but also "smaller than big" for the foreseeable future. That is, when the "starting point" conventional wisdom ("just use postgres / sqlite for everything; anything else is yagni") isn't quite up to snuff, but the "scale up" solutions are designed to handle far bigger use cases and bring some baggage along with that. I have honestly found Snowflake pretty useful for this, because it seems pretty cheap to me for a small amount of usage, but is also elastic up to this middle-ground of usage. But I suspect that at "truly big" I would / will be interested in different solutions.
mrweasel 3y ago
For the graphs I don't know, but I was happy to learn that Journald can actually ship logs to other Journald installations. Haven't used it yet, but I'm still looking of a place where it would make sense.
skinner927 3y ago
Fairly confident anyone outside of sales is going to tell you the solution that works is the best solution.

sanderjd 3y ago

I feel this in my bones. I really find it hard to figure out the right solutions for scaling things that will be "bigger than small" but also "smaller than big" for the foreseeable future. That is, when the "starting point" conventional wisdom ("just use postgres / sqlite for everything; anything else is yagni") isn't quite up to snuff, but the "scale up" solutions are designed to handle far bigger use cases and bring some baggage along with that. I have honestly found Snowflake pretty useful for this, because it seems pretty cheap to me for a small amount of usage, but is also elastic up to this middle-ground of usage. But I suspect that at "truly big" I would / will be interested in different solutions.

mrweasel 3y ago

For the graphs I don't know, but I was happy to learn that Journald can actually ship logs to other Journald installations. Haven't used it yet, but I'm still looking of a place where it would make sense.

skinner927 3y ago

Fairly confident anyone outside of sales is going to tell you the solution that works is the best solution.