3 ms·
This thread has a lot of discussion about Wide Events / Structured Logs (same thing) being too big at scale, and you should use metrics instead. Why does it ha
by infogulch 3y ago
This thread has a lot of discussion about Wide Events / Structured Logs (same thing) being too big at scale, and you should use metrics instead.
Why does it have to be an either/or thing? Couldn't you hook up a metrics extractor to the event stream and convert your structured logs to compact metrics in-process before expensive serde/encoding? With this your choice doesn't have to affect the code, just write slogs all the time; if you want structured logs then then output them, but if you only want metrics then switch to the metrics extractor slog handler.
Futher, has nobody tried writing structured logs to parquet files and shipping out 1MB blocks at once? Way less serde/encoding overhead, and column oriented layout compresses like crazy with built-in dictionary and delta encodings.
- pat2man 3y agoThe open telemetry collector does just that. https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/spanmetricsprocessor/README.md https://github.com/open-telemetry/opentelemetry-collector-co...
- pingiun 3y agoseems cool but the top of the page says the thing you suggest is now deprecated
- nijave 3y agoSpan Metrics Processor is replaced with the very similar Span Metrics Connector which is still supported
- paulddraper 3y agoThe Span Metrics Processor being replaced by the Span Metrics Connector is very, very OpenTelemetry.
- lowbloodsugar 3y ago>too big at scale Bigger than meta? o_0
- deleted 3y ago[deleted]
- pavlov 3y agoThe scale isn’t a problem if you also have Meta’s profit margins. But most businesses don’t…
- rmbyrro 3y agoThat's what I don't understand about observability services. They complicate things so much that it's sometimes more expensive to observe than to RUN the infra being observed.
- elbear 3y agoSounds like an unaligned incentives issue. Or maybe after a certain scale observing becomes cheaper than running?
- isburmistrov 3y agoI don't think Meta's margins have something to do with this. Companies smaller in scale than Meta also have less data! And yes, Scuba is in-memory, but it's not the requirement. Check this video out on how Honeycomb implemented their columnar storage: https://vimeo.com/331143124 https://vimeo.com/331143124
- infogulch 3y agoThis package implements the parquet idea for Go's slog package: https://github.com/samber/slog-parquet https://github.com/samber/slog-parquet