4 ms·
Most users do not use Kafka/Zookeeper. The only external service for them is a S3 bucket. They then use the PushAPI. It is perfectly fine if you have only a cou
by fulmicoton 3y ago
Most users do not use Kafka/Zookeeper.
The only external service for them is a S3 bucket. They then use the PushAPI.
It is perfectly fine if you have only a couple of TB a day.
For the crazy large use cases, like the one described in the blog post,
Kafka becomes necessary. At that scale, our users usually already have their data in Kafka or RedPanda and are actually happy to be able to get native integration:
- their data does not need to be copied/replicated in a WAL "again"
- we get exactly-once semantics
Also, in 0.8, we will be adding proper support for distributed ingest.
The feature is actually already implemented and was originally scheduled for 0.7.
but we preferred to test it more before actually shipping it.