5 ms·
I’ve found the hard part is not so much the collection of logs (especially at this scale), but the eventual querying. If you’ve got an unknown set of fields bee
by Dachande663 4y ago
I’ve found the hard part is not so much the collection of logs (especially at this scale), but the eventual querying. If you’ve got an unknown set of fields been logged, queries very quickly devolve into lots of slow table scans or needing materialised views that start hampering your ingest rate.
I settled on a happy/ok midpoint recently whereby I dump logs in a redis queue using filebeat as it’s very simple. Then have a really simple queue consumer that dumps the logs into clickhouse using a schema Uber detailed (split keys and values), so queries can be pretty quick even over arbitrary fields. 30,00 logs an hour and I can normally search for anything in under a second.
- metadat 4y agoWhat are the hardware requirements/ / resources dedicated to pull this off?
- Dachande663 4y agoRuns off a 2GB digital ocean box, which I think is $10 now? It’s probably incredibly boring to describe, but I think that’s why it just tends to work. The whole thing took an afternoon to write (in PHP of all things too).
- simonw 4y agoThis genuinely sounds the opposite of boring to me. I'd love to read a full, detailed description of this, hacky PHP scripts included!
- Dachande663 4y agoI've added some more info https://news.ycombinator.com/item?id=34771486 https://news.ycombinator.com/item?id=34771486
- mr-karan 4y agoI've a similar pipeline to yours (for the storage part). I use vector.dev for collecting and aggregating logs, enriching with metadata (cluster, env, AWS tags) and then finally storing them in a Clickhouse table. Do you use any particular UI/Frontend tool for querying these logs?
- Dachande663 4y agoWe already have a back office tool, so there’s just an extra screen that has query builder and outputs. Every few months I’ll tweak and add bits but nice to be able to see a user or entity ID and click on it to view the full resource.
- mr-karan 4y agoGot it, thanks.
- mekster 4y agoI've got that same logging backend and I use Metabase for querying. This is far cleaner and easier to use/learn than Kibana or Greylog. I've also considered Grafana, but it's not good for viewing raw logs.
- mr-karan 4y agoInteresting. In my experience I found Metabase gets clunkier when you've to query for adhoc logs. However it's a great fit when it's just a template and all the end user has to do is to enter some variables to see the logs. I think LogQL + Grafana UI is a pretty rich/interactive experience of viewing logs v/s Metabase.
- mekster 4y agoI'd take regular tabular view for logs like this, https://www.metabase.com/glossary/images/table/example-table.png https://www.metabase.com/glossary/images/table/example-table... than this brain crushing string spaghetti view any day. https://grafana.com/static/img/docs/v64/logs-panel.png https://grafana.com/static/img/docs/v64/logs-panel.png I can easily colorize lines depending on log severity, filter by dates with a drop-down date selector UI, filter by each column value etc all in eye friendly manner. https://pasteboard.co/PQDjUwtFgZTW.png https://pasteboard.co/PQDjUwtFgZTW.png https://pasteboard.co/tfHHA1mgETvL.png https://pasteboard.co/tfHHA1mgETvL.png
- FrenchTouch42 4y agoCan you share more information about the schema you're mentioning? Thank you!
- ignoramous 4y agoNot OP but they might be referring to Uber moving from ES to ClickHouse to store their schema-flexible, structured logs, mostly to improve ingestion performance: https://archive.is/bFsTF https://archive.is/bFsTF / https://www.uber.com/blog/logging/ https://www.uber.com/blog/logging/ The gist of it is: - Structured logs (json) are stored as kv pairs in parallel arrays, along side metadata (host, timestamp, id, geo, namespace, etc). - Log fields (ie kv pairs) are materialized (indexed) depending on query patterns, and vaccummed up if unused. - Authoring queries and Kibana dashboard support is not trivial but handled with a query translation layer.
- atombender 4y agoWhat do you mean by parallel arrays here? Do you mean something like two arrays [k1, ..., kN] and [v1, ..., vN] in two different columns? Is there a way in Clickhouse to filter such a pair of arrays such that you can do a search akin to vals[indexOfKey("foo")] == "bar"?
- ignoramous 4y agoYep. If you read the blog post I linked to it does talk a tonne about ClickHouse and what it can do (like indexOf, for example).
- atombender 4y agoAh, of course. Thanks!