3 ms·

This arch is how the big players do it at scale (ie. datadog, new relic - the second it passes their edge it lands in a kafka cluster). Also otel components lac

This arch is how the big players do it at scale (ie. datadog, new relic - the second it passes their edge it lands in a kafka cluster). Also otel components lack rate limiting(1) meaning its super easy to overload your backend storage (s3). Grafana has some posts how they softened the s3 blow with memcached(2,3). 1. https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/6908 https://github.com/open-telemetry/opentelemetry-collector-co... 2. https://grafana.com/docs/loki/latest/operations/caching/ https://grafana.com/docs/loki/latest/operations/caching/ 3. https://grafana.com/blog/2023/08/23/how-we-scaled-grafana-cloud-logs-memcached-cluster-to-50tb-and-improved-reliability/ https://grafana.com/blog/2023/08/23/how-we-scaled-grafana-cl... I know the post is about telemetry data and my comments on grafana are logs, but the arch bits still apply.

ankitnayan 3y ago
Caching is to improve read performance whereas Kafka is used to handle ingest volume. I couldn't correlate the Grafana articles shared
- blinded 3y ago
  Yep! Should have made that more clear. Brought it up as an example of other parts of the system that require scaling.
wardb 3y ago
Grafana Labs employee here => On the linked articles: I'm not aware of any caching being used in the writing data to S3 part of the pipeline other then some time based/volume based buffering at the ingester microservices before writing the chunks of data to object storage. The linked Loki caching docs/articles are for optimising the read access patterns of S3/object storage, not for writes.
- blinded 3y ago
  Yes. Thanks for the reply!
kilotaras 3y ago
Meta is also using something similar, see 4.1.1 of "Meta’s Next-generation Realtime Monitoring and Analytics Platform" [0] https://www.vldb.org/pvldb/vol15/p3522-mo.pdf https://www.vldb.org/pvldb/vol15/p3522-mo.pdf
- blinded 3y ago
  Super cool!

ankitnayan 3y ago

Caching is to improve read performance whereas Kafka is used to handle ingest volume. I couldn't correlate the Grafana articles shared

blinded 3y ago
Yep! Should have made that more clear. Brought it up as an example of other parts of the system that require scaling.

wardb 3y ago

Grafana Labs employee here => On the linked articles: I'm not aware of any caching being used in the writing data to S3 part of the pipeline other then some time based/volume based buffering at the ingester microservices before writing the chunks of data to object storage. The linked Loki caching docs/articles are for optimising the read access patterns of S3/object storage, not for writes.

blinded 3y ago
Yes. Thanks for the reply!

kilotaras 3y ago

Meta is also using something similar, see 4.1.1 of "Meta’s Next-generation Realtime Monitoring and Analytics Platform" [0] https://www.vldb.org/pvldb/vol15/p3522-mo.pdf https://www.vldb.org/pvldb/vol15/p3522-mo.pdf

blinded 3y ago
Super cool!