behemot

3 ms·

1. 1.▲
   by behemot 1y ago

   ClickHouse is pretty good at compressing the wide events, so it's not that dramatic compared to the benefits of having high-cardinality telemetry. check this out: https://clickhouse.com/blog/optimize-clickhouse-cod
2. 2.▲
   by behemot 1y ago

   > Do wide events really have to take up this much space? 100PB is the total volume of the raw, uncompressed data for the full retention period (180 days). compression is what makes it cost-efficient. on this dataset, we see ~15x compress
3. 3.▲
   by behemot 1y ago

   we use k8s + otel filelog receiver. in this case you don't have to connect to the clickhouse instance to collect what it's writing to stdout/stderr, just tail /var/log/pods/*/*/*.log.
4. 4.▲
   by behemot 1y ago

   we keep it for 180 days.
5. 5.▲
   by behemot 1y ago

   > Anecdotally I know robust tracing data can help me find problems in 5-15 minutes that would have taken hours or days with manual probing and scouring logs. exactly. high-cardinality, wide structured events are the way.
6. 6.▲
   by behemot 1y ago

   hey there! I work at ClickHouse. to clarify: the vast majority of this 100PB is structured events. in our case logs are supplementary.