4 ms·
I've a similar pipeline to yours (for the storage part). I use vector.dev for collecting and aggregating logs, enriching with metadata (cluster, env, AWS tags)
by mr-karan 4y ago
I've a similar pipeline to yours (for the storage part). I use vector.dev for collecting and aggregating logs, enriching with metadata (cluster, env, AWS tags) and then finally storing them in a Clickhouse table.
Do you use any particular UI/Frontend tool for querying these logs?
- Dachande663 4y agoWe already have a back office tool, so there’s just an extra screen that has query builder and outputs. Every few months I’ll tweak and add bits but nice to be able to see a user or entity ID and click on it to view the full resource.
- mr-karan 4y agoGot it, thanks.
- mekster 4y agoI've got that same logging backend and I use Metabase for querying. This is far cleaner and easier to use/learn than Kibana or Greylog. I've also considered Grafana, but it's not good for viewing raw logs.
- mr-karan 4y agoInteresting. In my experience I found Metabase gets clunkier when you've to query for adhoc logs. However it's a great fit when it's just a template and all the end user has to do is to enter some variables to see the logs. I think LogQL + Grafana UI is a pretty rich/interactive experience of viewing logs v/s Metabase.
- mekster 4y agoI'd take regular tabular view for logs like this, https://www.metabase.com/glossary/images/table/example-table.png https://www.metabase.com/glossary/images/table/example-table... than this brain crushing string spaghetti view any day. https://grafana.com/static/img/docs/v64/logs-panel.png https://grafana.com/static/img/docs/v64/logs-panel.png I can easily colorize lines depending on log severity, filter by dates with a drop-down date selector UI, filter by each column value etc all in eye friendly manner. https://pasteboard.co/PQDjUwtFgZTW.png https://pasteboard.co/PQDjUwtFgZTW.png https://pasteboard.co/tfHHA1mgETvL.png https://pasteboard.co/tfHHA1mgETvL.png