3 ms·
Clickhouse has a built in feature for this called Asynchronous Inserts https://clickhouse.com/docs/en/optimize/asynchronous-inserts https://clickhouse.com/docs/
by chis 2y ago
Clickhouse has a built in feature for this called Asynchronous Inserts https://clickhouse.com/docs/en/optimize/asynchronous-inserts https://clickhouse.com/docs/en/optimize/asynchronous-inserts
It just buffers inserts in a queue and then does them all at once after a second has passed. This document does briefly mention it but it's such a nice feature that saves you from setting up a whole Kafka thing.
- mikeshi42 2y agoYou want to be careful of async inserts as it's in-memory buffered as well and is not replicated afaik. So you can lose a bit of durability when using it (or you make sure your upstream pipeline can retry). It is really nice in a pinch though.
- chis 2y agoMy read on it was that if you keep wait_for_async_insert=True then the client will still wait until there's confirmation of a successful transaction, giving the possibility to retry. It does require one to actually have a retry handler on the client side, but that's a good idea anyways.
- dilyevsky 2y agoIn practice, using async insert with wait means you need batching/retries implemented on the client side or else you may find you have thousands of threads stuck in wait. If you need to replay bc you messed something up you're also SOL. I generally appreciate not having to run kafka but that comes at a cost.
- chis 2y agoVery good point.
- jsumrall 2y agoNote that you will need a thread pool that’s large enough to handle 1 second (or whatever flush timeout you set) of events in your app concurrently. If you have thousands of events per second, plan accordingly.
- iancarroll 2y agoDidn’t spend enough time to find the root cause, but at enough volume this option caused some of our inserts to take several seconds, even with awaiting for success disabled. Just moved to using an in-memory queue in our app and it fixed it.