4 ms·

The way Kafka works, it naturally buffers & coalesces messages even before they get to the brokers, so yes, of course the messages are being coalesced. There i

The way Kafka works, it naturally buffers & coalesces messages even before they get to the brokers, so yes, of course the messages are being coalesced. There is no problem with "just" running a Kafka cluster in each AZ and only replicating data between AZs until it's time to pull it all together. It's just that when presented with a distributed system and AZs, engineers (and in fairness the business requirements) are more than likely to go with a multi-AZ solution. Same goes for regions. So the vast majority of Kafka clusters are multi-AZ but probably shouldn't be, and Kafka gets the bill for that, even though it shouldn't. The Kafka protocol doesn't really preserve order-of-operation within a Kafka partition. It preserves the order of operations within a producer-partition pair (and even then, only if you configure it a certain way). The standard implementation does this by preserving the order-of-broker-receipt-of-messages from producers, but from an external system's vantage point, it really only means that (if configured the right way) messages with any given key, from any given producer, will be preserved in the order they are received.

cduzz 3y ago
The architect mentioned a 100ms (or 10 times per second) buffer / flush rate; presumably there's some windowing settings so I can flush at 10,000 records or after 1/10th of a second, and can chose 10 records or 1/100th of a second if I don't mind the bill. Kafka's nice because there are a lot of knobs to adjust how you prioritize availability or durability or latency. Kafka's tedious because there are a lot of knobs... I was curious as to the nature of the knobs on this product. I probably should have also asked "and by S3 do you mean S3 or any S3 like object store?" probably that's answered elsewhere.
- cbsmith 3y ago
  The impression I have is that they have deliberately removed a lot of the knobs, so I'd speculate that you can't tweak the buffer/flush rate. However, that's just my speculation.
- ryanworl 3y ago
  [WarpStream CTO here] WarpStream flushes after 4MiB of data or a configurable amount of time. Flushes can also happen concurrently. In general, we'd prefer to not introduce many knobs. We're running a realistic throughput testing workload in our staging and production environments 24/7, so we've configured most of the knobs already to reasonable defaults. We just added support for other S3-compatible storage systems today: https://docs.warpstream.com/warpstream/reference/use-the-agents-with-an-s3-compatible-object-store-minio-oracle-cloud-etc https://docs.warpstream.com/warpstream/reference/use-the-age...

cduzz 3y ago

The architect mentioned a 100ms (or 10 times per second) buffer / flush rate; presumably there's some windowing settings so I can flush at 10,000 records or after 1/10th of a second, and can chose 10 records or 1/100th of a second if I don't mind the bill. Kafka's nice because there are a lot of knobs to adjust how you prioritize availability or durability or latency. Kafka's tedious because there are a lot of knobs... I was curious as to the nature of the knobs on this product. I probably should have also asked "and by S3 do you mean S3 or any S3 like object store?" probably that's answered elsewhere.

cbsmith 3y ago
The impression I have is that they have deliberately removed a lot of the knobs, so I'd speculate that you can't tweak the buffer/flush rate. However, that's just my speculation.
ryanworl 3y ago
[WarpStream CTO here] WarpStream flushes after 4MiB of data or a configurable amount of time. Flushes can also happen concurrently. In general, we'd prefer to not introduce many knobs. We're running a realistic throughput testing workload in our staging and production environments 24/7, so we've configured most of the knobs already to reasonable defaults. We just added support for other S3-compatible storage systems today: https://docs.warpstream.com/warpstream/reference/use-the-agents-with-an-s3-compatible-object-store-minio-oracle-cloud-etc https://docs.warpstream.com/warpstream/reference/use-the-age...