3 ms·

can you just buffer some writes in Vector and eliminate Kafka? I setup some Vector to buffer ElasticSearch writes years ago, also for logs, it ran so well with

by est 1y ago

can you just buffer some writes in Vector and eliminate Kafka? I setup some Vector to buffer ElasticSearch writes years ago, also for logs, it ran so well without any problems that I almost fogot about it.

anticodon 1y ago
Or vice versa: make ClickHouse ingest batches directly from Kafka. Messages are already buffered in Kafka, I don't get why Vector is necessary here.
- ch2026 1y ago
  tbh the only thing they needed was a correct schema that didn’t constantly spawn new parts and async_insert enabled.
- est 1y ago
  https://clickhouse.com/docs/knowledgebase/kafka-to-clickhouse-setup https://clickhouse.com/docs/knowledgebase/kafka-to-clickhous... For anyone if curious.