4 ms·

(WarpStream founder here) Yes exactly. The cloud metadata store assigns offsets, does compaction planning/scheduling, handles service discovery, etc

• morelisp 3y ago
  I see - this also explains the tiny consumer group / topic-partition limit of the free plan, then...
  • ryanworl 3y ago
    [WarpStream co-founder here] What do you think would be a good limit for the free plan? This isn't actually an architectural constraint for us. We just didn't want to promise unlimited usage forever so we picked a somewhat arbitrary number to start with.
    • morelisp 3y ago
      I'm probably the wrong person to ask, sorry! My interest is purely academic, both for data governance and technical reasons I'll probably never be able to use this. I also don't mean to imply they're unfair, free is free - just that it explains why things which are pretty cheap in Kafka proper trigger billing (vs. of course things that are expensive in Kafka now being really cheap). I do think a lot of people using Kafka compaction topics as their source-of-truth are defaulting to 256-~1000 partitions, if you want to entice them I'd probably offer at least 1024. (But I suspect your target market is not using these - rather they're salivating over the possibility of the automatic format transcoding you mentioned in another comment...)