2 ms·
We (ab)use our logging pipeline for metrics too (can afford to do so because of our rather meagre scale, which also means we don't really need tracing much). W
by ignoramous 4y ago
We (ab)use our logging pipeline for metrics too (can afford to do so because of our rather meagre scale, which also means we don't really need tracing much).
Whenever metrics are needed, we pull (hive-compatible time-partitioned) logs (transformed to parquet or csv) from the blob store into DuckDB and on-demand query from there. No online dashboards, but pretty good for the cost.
https://duckdb.org/docs/extensions/httpfs https://duckdb.org/docs/extensions/httpfs