4 ms·

The blog post mentions that partitions are too low-level an abstraction to program against. Does that mean WarpStream doesn't use partitions? Do you provide an

• richieartoul 3y ago
  (WarpStream founder) No WarpStream has partitions internally and provides the same ordering guarantees Kafka does at the partition level. We're just saying that we think for most streaming applications this is not a great programming model, and we think there is an opportunity to do something better (but we haven't done that yet).
  • necubi 3y ago
    Do you still provide low-level control over partition subscriptions and offset management? Any plan to support Kafka transactions? That's all required to build exactly-once systems on top of Kafka (like the stateful stream processing engine I work on) even if it's not the easiest interface for normal application-level development.
    • ryanworl 3y ago
      [WarpStream CTO here] WarpStream is Kafka protocol compatible, so we do support topic-partitions and consumer groups. We do not expose support for transactions or idempotent producing today, but the internals of the system support that and we will probably work on the idempotent producer sometime in the next month, with transactions coming shortly after, depending on demand from the Developer Preview users.