4 ms·

Couple questions: 1. aren't you going to get murderous S3 API call bills if you're pushing each message directly into S3? How're you buffering / queuing / coal

by cduzz 3y ago

Couple questions: 1. aren't you going to get murderous S3 API call bills if you're pushing each message directly into S3? How're you buffering / queuing / coalescing messages durably without local storage? 2. what's the problem with "just" running a kafka cluster in each AZ and not replicating data between AZs until it's time to ETL the data to wherever? AZ1 clients push to AZ1 clusters; AZ2 clients push to AZ2 clusters, etc. 3. What's done to preserve order-of-operation within a kafka partition?

cbsmith 3y ago
The way Kafka works, it naturally buffers & coalesces messages even before they get to the brokers, so yes, of course the messages are being coalesced. There is no problem with "just" running a Kafka cluster in each AZ and only replicating data between AZs until it's time to pull it all together. It's just that when presented with a distributed system and AZs, engineers (and in fairness the business requirements) are more than likely to go with a multi-AZ solution. Same goes for regions. So the vast majority of Kafka clusters are multi-AZ but probably shouldn't be, and Kafka gets the bill for that, even though it shouldn't. The Kafka protocol doesn't really preserve order-of-operation within a Kafka partition. It preserves the order of operations within a producer-partition pair (and even then, only if you configure it a certain way). The standard implementation does this by preserving the order-of-broker-receipt-of-messages from producers, but from an external system's vantage point, it really only means that (if configured the right way) messages with any given key, from any given producer, will be preserved in the order they are received.
- cduzz 3y ago
  The architect mentioned a 100ms (or 10 times per second) buffer / flush rate; presumably there's some windowing settings so I can flush at 10,000 records or after 1/10th of a second, and can chose 10 records or 1/100th of a second if I don't mind the bill. Kafka's nice because there are a lot of knobs to adjust how you prioritize availability or durability or latency. Kafka's tedious because there are a lot of knobs... I was curious as to the nature of the knobs on this product. I probably should have also asked "and by S3 do you mean S3 or any S3 like object store?" probably that's answered elsewhere.
  - cbsmith 3y ago
    The impression I have is that they have deliberately removed a lot of the knobs, so I'd speculate that you can't tweak the buffer/flush rate. However, that's just my speculation.
  - ryanworl 3y ago
    [WarpStream CTO here] WarpStream flushes after 4MiB of data or a configurable amount of time. Flushes can also happen concurrently. In general, we'd prefer to not introduce many knobs. We're running a realistic throughput testing workload in our staging and production environments 24/7, so we've configured most of the knobs already to reasonable defaults. We just added support for other S3-compatible storage systems today: https://docs.warpstream.com/warpstream/reference/use-the-agents-with-an-s3-compatible-object-store-minio-oracle-cloud-etc https://docs.warpstream.com/warpstream/reference/use-the-age...
ilikehurdles 3y ago
Re #1 I don’t think Amazon charges for data transfers from within AWS, assuming Kafka is hosted there.
- vasco 3y ago
  S3 requests are paid regardless of source (tldr GETs are cheaper than PUTs). You're confused with data transfer within a VPC.
- finnh 3y ago
  There are per-API-call charges. It's just not per-byte.
  - richieartoul 3y ago
    (WarpStream founder) Exactly, and we designed WarpStream's storage system to minimize the number of object storage API calls it has to make while still maintaining relatively low latency (P99 of ~1s end-to-end)
morelisp 3y ago
I'm also especially interested in 3) - from the arch overview it sounds like all agents are actively writing and actively compacting, how do they coordinate which topic-partitions to compact? Is the Cloud Metadata Store essentially responsible for handing out the offsets?
- richieartoul 3y ago
  (WarpStream founder here) Yes exactly. The cloud metadata store assigns offsets, does compaction planning/scheduling, handles service discovery, etc
  - morelisp 3y ago
    I see - this also explains the tiny consumer group / topic-partition limit of the free plan, then...
    - ryanworl 3y ago
      [WarpStream co-founder here] What do you think would be a good limit for the free plan? This isn't actually an architectural constraint for us. We just didn't want to promise unlimited usage forever so we picked a somewhat arbitrary number to start with.
      - morelisp 3y ago
        I'm probably the wrong person to ask, sorry! My interest is purely academic, both for data governance and technical reasons I'll probably never be able to use this. I also don't mean to imply they're unfair, free is free - just that it explains why things which are pretty cheap in Kafka proper trigger billing (vs. of course things that are expensive in Kafka now being really cheap). I do think a lot of people using Kafka compaction topics as their source-of-truth are defaulting to 256-~1000 partitions, if you want to entice them I'd probably offer at least 1024. (But I suspect your target market is not using these - rather they're salivating over the possibility of the automatic format transcoding you mentioned in another comment...)
ryanworl 3y ago
[WarpStream co-founder and CTO here] 1. Each WarpStream Agent flushes a file to S3 with all the data for every topic-partition it has received requests for in the last ~100ms or so. This means the S3 PUT operations costs scales with the number of Agents you run and the flushing interval, not the number of topic-partitions. We do not acknowledge Produce requests until data has been durably persisted in S3 and our cloud control plane. 2. We think people shouldn't have to choose between reliability and costs. WarpStream gives you the reliability and availability of running in three AZs but with the cost of one. 3. We have a custom metadata database running in our cloud control plane which handles ordering.
- poorman 3y ago
  It sounds like there's a sweet spot here. If you are not ACKing Produce requests for 100ms then there's a huge amount latency. If the user want's to reduce that latency from 100ms to say 1ms then their S3 GET requests cost just went up by 100x.
  - ryanworl 3y ago
    [WarpStream co-founder here] We've done lots of customer research here and, combined with the experience my co-founder and I have, we can confidently say most Kafka users (especially high-throughput users) would happily make a trade off of increased end-to-end latency in exchange for a massive cost reduction and the operational simplicity provided by WarpStream.
    - cgio 3y ago
      I don’t want to disagree with the research here, but what is not evident from the article is that this is not a magical solution that improves upon Kafka hands down, but rather a solution that addresses trade offs someone might be willing to entertain. I think on the query side things may be quite suboptimal in this setup if I understand it correctly. Correct me if I am wrong but if two agents write on a single topic, I would need to read two files to consume. Also I remember infamous stories about the cost of deleting data from S3, how do you tackle that if you have that many individual files? With these trade offs how does the solution compare to using Aurora?

by cduzz 3y ago

cbsmith 3y ago

The way Kafka works, it naturally buffers & coalesces messages even before they get to the brokers, so yes, of course the messages are being coalesced. There is no problem with "just" running a Kafka cluster in each AZ and only replicating data between AZs until it's time to pull it all together. It's just that when presented with a distributed system and AZs, engineers (and in fairness the business requirements) are more than likely to go with a multi-AZ solution. Same goes for regions. So the vast majority of Kafka clusters are multi-AZ but probably shouldn't be, and Kafka gets the bill for that, even though it shouldn't. The Kafka protocol doesn't really preserve order-of-operation within a Kafka partition. It preserves the order of operations within a producer-partition pair (and even then, only if you configure it a certain way). The standard implementation does this by preserving the order-of-broker-receipt-of-messages from producers, but from an external system's vantage point, it really only means that (if configured the right way) messages with any given key, from any given producer, will be preserved in the order they are received.

cduzz 3y ago
The architect mentioned a 100ms (or 10 times per second) buffer / flush rate; presumably there's some windowing settings so I can flush at 10,000 records or after 1/10th of a second, and can chose 10 records or 1/100th of a second if I don't mind the bill. Kafka's nice because there are a lot of knobs to adjust how you prioritize availability or durability or latency. Kafka's tedious because there are a lot of knobs... I was curious as to the nature of the knobs on this product. I probably should have also asked "and by S3 do you mean S3 or any S3 like object store?" probably that's answered elsewhere.
- cbsmith 3y ago
  The impression I have is that they have deliberately removed a lot of the knobs, so I'd speculate that you can't tweak the buffer/flush rate. However, that's just my speculation.
- ryanworl 3y ago
  [WarpStream CTO here] WarpStream flushes after 4MiB of data or a configurable amount of time. Flushes can also happen concurrently. In general, we'd prefer to not introduce many knobs. We're running a realistic throughput testing workload in our staging and production environments 24/7, so we've configured most of the knobs already to reasonable defaults. We just added support for other S3-compatible storage systems today: https://docs.warpstream.com/warpstream/reference/use-the-agents-with-an-s3-compatible-object-store-minio-oracle-cloud-etc https://docs.warpstream.com/warpstream/reference/use-the-age...

ilikehurdles 3y ago

Re #1 I don’t think Amazon charges for data transfers from within AWS, assuming Kafka is hosted there.

vasco 3y ago
S3 requests are paid regardless of source (tldr GETs are cheaper than PUTs). You're confused with data transfer within a VPC.
finnh 3y ago
There are per-API-call charges. It's just not per-byte.
- richieartoul 3y ago
  (WarpStream founder) Exactly, and we designed WarpStream's storage system to minimize the number of object storage API calls it has to make while still maintaining relatively low latency (P99 of ~1s end-to-end)

morelisp 3y ago

I'm also especially interested in 3) - from the arch overview it sounds like all agents are actively writing and actively compacting, how do they coordinate which topic-partitions to compact? Is the Cloud Metadata Store essentially responsible for handing out the offsets?

richieartoul 3y ago
(WarpStream founder here) Yes exactly. The cloud metadata store assigns offsets, does compaction planning/scheduling, handles service discovery, etc
- morelisp 3y ago
  I see - this also explains the tiny consumer group / topic-partition limit of the free plan, then...
  - ryanworl 3y ago
    [WarpStream co-founder here] What do you think would be a good limit for the free plan? This isn't actually an architectural constraint for us. We just didn't want to promise unlimited usage forever so we picked a somewhat arbitrary number to start with.
    - morelisp 3y ago
      I'm probably the wrong person to ask, sorry! My interest is purely academic, both for data governance and technical reasons I'll probably never be able to use this. I also don't mean to imply they're unfair, free is free - just that it explains why things which are pretty cheap in Kafka proper trigger billing (vs. of course things that are expensive in Kafka now being really cheap). I do think a lot of people using Kafka compaction topics as their source-of-truth are defaulting to 256-~1000 partitions, if you want to entice them I'd probably offer at least 1024. (But I suspect your target market is not using these - rather they're salivating over the possibility of the automatic format transcoding you mentioned in another comment...)

ryanworl 3y ago

[WarpStream co-founder and CTO here] 1. Each WarpStream Agent flushes a file to S3 with all the data for every topic-partition it has received requests for in the last ~100ms or so. This means the S3 PUT operations costs scales with the number of Agents you run and the flushing interval, not the number of topic-partitions. We do not acknowledge Produce requests until data has been durably persisted in S3 and our cloud control plane. 2. We think people shouldn't have to choose between reliability and costs. WarpStream gives you the reliability and availability of running in three AZs but with the cost of one. 3. We have a custom metadata database running in our cloud control plane which handles ordering.

poorman 3y ago
It sounds like there's a sweet spot here. If you are not ACKing Produce requests for 100ms then there's a huge amount latency. If the user want's to reduce that latency from 100ms to say 1ms then their S3 GET requests cost just went up by 100x.
- ryanworl 3y ago
  [WarpStream co-founder here] We've done lots of customer research here and, combined with the experience my co-founder and I have, we can confidently say most Kafka users (especially high-throughput users) would happily make a trade off of increased end-to-end latency in exchange for a massive cost reduction and the operational simplicity provided by WarpStream.
  - cgio 3y ago
    I don’t want to disagree with the research here, but what is not evident from the article is that this is not a magical solution that improves upon Kafka hands down, but rather a solution that addresses trade offs someone might be willing to entertain. I think on the query side things may be quite suboptimal in this setup if I understand it correctly. Correct me if I am wrong but if two agents write on a single topic, I would need to read two files to consume. Also I remember infamous stories about the cost of deleting data from S3, how do you tackle that if you have that many individual files? With these trade offs how does the solution compare to using Aurora?