4 ms·
This is interesting. I appreciate the simplicity and DIY aspects. Is it available as a repo somewhere? I recently had to troubleshoot a hanging issue on one of
by imiric 1y ago
This is interesting. I appreciate the simplicity and DIY aspects. Is it available as a repo somewhere?
I recently had to troubleshoot a hanging issue on one of my servers, so I needed something that could ship logs. The modern observability stack is a deep pit of complexity, but OpenTelemetry is a standard, and there are reasonably simple tools in the ecosystem. I knew I didn't want a behemoth like Grafana, and I was aware of SigNoz, though it seems janky. Then I stumbled upon OpenObserve, and it looked promising. Setting it up on a spare mini PC and opentelemetry-collector on the server was pretty straightforward. Getting the collector configuration right took some trial and error, though.
I have to say, I'm quite satisfied with this setup. I ended up installing the collector on other machines, so it's almost like a proper observability system now :)
The graphs are nice. I can expand it to monitor anything else I would need. I haven't setup alerts yet, but it's possible.
I'm not really concerned about monitoring the monitor. It's not a big deal for my use case if it goes down. Metrics and logs will be submitted when it's back up, since they're cached on the servers. Besides, I'm only running OpenObserve on the machine, so there aren't many moving parts.
Anyway, all this is to say that sometimes there's more to be gained from using off-the-shelf tooling instead of rolling your own, even if it involves more complexity. Server monitoring is an old tradition, and there are many robust solutions out there. OTLP isn't that bad, especially at smaller scales, and it opens the door to a large ecosystem. It would be foolish not to take advantage of that.