4 ms·
Exactly. When they say > Unlike with prometheus, however, with Wide Events approach we don’t need to worry about cardinality This is hinting at the hidden rea
by ricardobeat 3y ago
Exactly. When they say
> Unlike with prometheus, however, with Wide Events approach we don’t need to worry about cardinality
This is hinting at the hidden reason why not everyone does it. You have to 'worry' about cardinality because Prometheus is pre-aggregating data so you can visualize it fast, and optimizing storage. If you want the same speed on a massive PB-scale data lake, with an infinite amount of unstructured data, and in the cloud instead of your own datacenters, it's gonna cost you a lot, and for most companies it is not a sensible expense.
It does work at smaller scale though, we once had an in-house system like this that worked well. Eventually user events were moved to MixPanel, and everything else to Datadog, metrics/logs/traces + a migration to OpenTel. It took months and added 2-digit monthly bills, and in the end debugging or resolving incidents wasn't much improved over having instant access to events and business metrics. Whoever figures out a system that can do "wide events" in a cost-effective way from startup to unicorn scale will absolutely make a killing.
- hosh 3y agoNot everything emits wide events. Maybe you can get the entire application layer like that, but there is also value in logs and metrics emitted from the rest of the infra stack. To be fair, you could probably store and represent everything as wide events and build visualization tools out of that that can combine everything together, even if they are sourced from something else.
- lelandbatey 3y agoWide events seem to be "structured logs with focused schemas" (maybe also published in a special way beyond writing to stdout) but most places I've worked would call that "logging" not "wide events". The reasons we don't use them for everything are as others in the thread say: it's expensive. Metrics (just the numbers, nothing else) can be compressed and aggregated extremely efficiently, hence cheaply. Logs are more expensive due to their arbitrary contents. It's all due to expense really.
- isburmistrov 3y agoColumnar storage stores data very efficiently, too - because it compresses data of a similar nature (columns). Check e.g. ClickHouse on this matter: https://clickhouse.com/docs/en/about-us/distinctive-features https://clickhouse.com/docs/en/about-us/distinctive-features, https://clickhouse.com/blog/working-with-time-series-data-and-functions-ClickHouse https://clickhouse.com/blog/working-with-time-series-data-an... So I wouldn't say that events are "expensive" while metrics are "cheap" - both depend on the actual implementation, and events can be cheap too. And so of course if you have to optimise things, you would need to drop some information you pass to the events, but you would need to do the same for metrics (reduce the number of metrics emitted, reduce the prometheus labels,...).
- flaminHotSpeedo 3y agoThe whole point of wide events is recording an arbitrary set of key value pairs. How do you propose storing that in a columnar datastore?
- phillipcarter 3y agoI can't speak for others, but at Honeycomb that's what we do. There's some details in this blog post that might be interesting: https://www.honeycomb.io/blog/why-observability-requires-distributed-column-store https://www.honeycomb.io/blog/why-observability-requires-dis...
- hagen1778 3y agoStoring telemetry efficiently is only part of what Monitoring is supposed to do. The other part is querying: ad-hoc queries, dashboards, alerting queries executed each 15s or so. For querying to work fast, there has to be an efficient index or multiple indexes depending on the query. Since you referred ClickHouse as efficient columnar storage, please see what makes it different from a time series database - https://altinity.com/wp-content/uploads/2021/11/How-ClickHouse-Inspired-Us-to-Build-a-High-Performance-Time-Series-Database.pdf https://altinity.com/wp-content/uploads/2021/11/How-ClickHou...
- AeroNotix 3y agohttps://victoriametrics.com/ https://victoriametrics.com/ would definitely recommend anyone having performance issues with Prometheus to give VictoriaMetrics a try.
- uaas 3y agoOr, you can use Thanos, the de-facto standard with the biggest OSS community.
- dengolius 3y agoThanos/Mimir community doesn't help to resolve configuration routine or even bigger resource consumption for a huge setup.
- uaas 3y agoBigger resource consumption of what exactly? Leaf Prometheus instances or the Thanos/Mimir stack compared to VictoriaMetrics? Have you seen a large scale migration between the two, with actual numbers?
- valyala 3y agoA few of interesting real-world large-scale migrations are highlighted at https://docs.victoriametrics.com/casestudies/ https://docs.victoriametrics.com/casestudies/
- camel_gopher 3y agoAnd comes with all the downsides of Prometheus as well
- dengolius 3y agoCould you pelase elaborate more on "comes with all the downsides of Prometheus"?
- 3y ago