11 ms·

What If We Could Rebuild Kafka from Scratch?

• oulipo 2y ago
  There are a few interesting projects to replace Kafka: Redpanda / Pulsar / AutoMQ have some of you some experience with those and able to give pros/cons?
  • SkipperCat 2y ago
    I've used Redpanda and I like it. Its Kafka compliant so all the tools that work with Apache Kafka also work with Redpanda. Plus, no JVM and no zookeeper. Redpanda most importantly is faster that Apache Kafka. We were able to get a lot more throughput. Its also stable, especially compared to dealing with anything that requires a JVM.
• hardwaresofton 2y ago
  See also: Warpstream, which was so good it got acquired by Confluent. Feels like there is another squeeze in that idea if someone “just” took all their docs and replicated the feature set. But maybe that’s what S2 is already aiming at. Wonder how long warpstream docs, marketing materials and useful blogs will stay up.
  • dangoodmanUT 2y ago
    i wouldn't say it was so good it got acquired by them, rather confluent had no s3-backed play and it was easier for them to acquire warpstream than to add it to kafka directly warpstream has latency issues, which downstream turn into cost issues
    • hardwaresofton 2y ago
      That's a good point -- I assumed there were other choices, but now that I look, warpstream may have been the only already-kafka-compatible option. That said, they were at least "good enough" to make "buy" more appealing than "build"
    • jjk7 2y ago
      Confluent has "Freight" which is their integrated s3-backed play.
• ActorNightly 2y ago
  [flagged]
  • butterlettuce 2y ago
    Step 2, use Python for everything
    • mirekrusin 2y ago
      Good choice, leaves space to rewrite in rust later, right?
      • Spivak 2y ago
        The fact that this is so common I think yearns for a language that is basically python and rust smashed together where a project would have some code in the python side and some code in the rust side intermixed fluidly like how you can drop to asm in C. Don't even really try to make the two halves of the language similar. An embedded interpreter and JIT in rust basically but jostled around a bit to make it more cohesive and the data interop more fluid— PyO3 but backwards.
        • alpaca128 2y ago
          You can already do that with a macro: https://crates.io/crates/inline-python https://crates.io/crates/inline-python But I'm doubtful that it's going to make things simpler if one can't even decide on a language.
          • Spivak 2y ago
            It's not really a "decision" thing, more that the pattern of write most of the code in python, rewrite the performance critical bits in rust as a cpython extension would be nice if it were made first class.
        • mirekrusin 2y ago
          Maybe something will eventually crystalize out of mojo?
• Ozzie_osman 2y ago
  I feel like everyone's journey with Kafka ends up being pretty similar. Initially, you think "oh, an append-only log that can scale, brilliant and simple" then you try it out and realize it is far, far, from being simple.
  • carlmr 2y ago
    I'm wondering how much of that is bad developer UX and defaults, and how much of that is inherent complexity in the problem space. Like the article outlines, partitions are not that useful for most people. Instead of removing them, how about having them behind a feature flag, i.e. not on by default. That would ease 99% of users problems. The next point in the article which to me resonates is the lack of proper schema support. That's just bad UX again, not inherent complexity of the problem space. On testing side, why do I need to spin up a Kafka testcontainer, why is there no in-memory kafka server that I can use for simple testing purposes.
    • ahoka 2y ago
      I think it's just horrible software built on great ideas sold on a false premise (this is a generic message queue and if you don't use this you cannot "scale").
      • deleted 2y ago
        [deleted]
      • carlmr 2y ago
        I kind of agree on the horrible software bit, but what do you use instead? And can you convince your company to use that, too?
        • buster 2y ago
          I find that many such systems really just need a scalable messaging system. Use RabbitMQ, Nats, Pub/Sub, ... There are plenty. Confluent has rather good marketing and when you need messaging but can also gain a persistent, super scalable data store and more, why not use that instead? The obvious answer is: Because there is no one-size-fits-all-solution with no drawbacks.
• Spivak 2y ago
  Once you start asking to query the log by keys, multi-tenancy trees of topics, synchronous commits-ish, and schemas aren't we just in normal db territory where the kafka log becomes the query log. I think you need to go backwards and be like what is the feature a rdbms/nosql db can't do and go from there. Because the wishlist is looking like CQRS with the front queue being durable but events removed once persisted in the backing db where the clients query events from the db. The backing db in this wishlist would be something in the vein of Aurora to achieve the storage compute split.
• peanut-walrus 2y ago
  Object storage for Kafka? Wouldn't this 10x the latency and cost? I feel like Kafka is a victim of it's own success, it's excellent for what it was designed, but since the design is simple and elegant, people have been using it for all sorts of things for which it was not designed. And well, of course it's not perfect for these use cases.
  • ako 2y ago
    Warpstream already does Kafka with object storage.
    • rad_gruchalski 2y ago
      WarpStream has been acquired by Confluent.
  • mrweasel 2y ago
    > people have been using it for all sorts of things for which it was not designed Kafka is misused for some weird stuff. I've seen it used as a user database, which makes absolutely no sense. I've also seen it used a "key/value" store, which I can't imagine being efficient as you'd have to scan the entire log. Part of it seems to stem from "We need somewhere to store X. We already have Kafka, and requesting a database or key/value store is just a bit to much work, so let's stuff it into Kafka". I had a client ask for a Kafka cluster, when queried about what they'd need it for we got "We don't know yet". Well that's going to make it a bit hard to dimension and tune it correctly. Everyone else used Kafka, so they wanted to use it too.
  • biorach 2y ago
    > the design is simple and elegant Kafka is simple and elegant?
    • EdwardDiego 2y ago
      The design is.
    • peanut-walrus 2y ago
• vim-guru 2y ago
  https://nats.io https://nats.io is easier to use than Kafka and already solves several of the points in this post I believe, like removing partitions, supporting key-based streams, and having flexible topic hierarchies.
  • fasteo 2y ago
    And JetStream[1] adds persistence to make it more comparable to Kafka [1] https://docs.nats.io/nats-concepts/jetstream https://docs.nats.io/nats-concepts/jetstream
  • wvh 2y ago
    I came here to say just that. Nats solves a lot of those challenges, like different ways to query and preserve messages, hierarchical data, decent authn/authz options for multi-tenancy, much lighter and easier to set up, etc. It has more of a messaging and k/v store feel than the log Kafka is, so while there's some overlap, I don't think they fit the exact same use cases. Nats is fast, but I haven't seen any benchmarks for specifically the bulk write-once append log situation Kafka is usually used for. Still, if a hypothetical new Kafka would incorporate some of Nats' features, that would be a good thing.
  • boomskats 2y ago
    Also remember that NATS was donated to the CNCF a while back, and as a result people built a huge ecosystem around it. Easy to forget.
    • rmoff 2y ago
      Relevant: https://www.cncf.io/blog/2025/04/24/protecting-nats-and-the-integrity-of-open-source-cncfs-commitment-to-the-community/ https://www.cncf.io/blog/2025/04/24/protecting-nats-and-the-...
      • cmckn 2y ago
        Yikes, what a dastardly move from Synadia. This will only have a chilling effect on the project, which has struggled to reach critical mass.
• nitwit005 2y ago
  > When producing a record to a topic and then using that record for materializing some derived data view on some downstream data store, there’s no way for the producer to know when it will be able to "see" that downstream update. For certain use cases it would be helpful to be able to guarantee that derived data views have been updated when a produce request gets acknowledged, allowing Kafka to act as a log for a true database with strong read-your-own-writes semantics. Just don't use Kafka. Write to the downstream datastore directly. Then you know your data is committed and you have a database to query.
  • menzoic 2y ago
    Writing directly to the datastore ignores the need for queuing the writes. How do you solve for that need?
    • immibis 2y ago
      Why do you need to queue the writes?
      • smitty1e 2y ago
        A proficient coder can write a program to accomplish a task in the singular. In the plural, accomplishing that task in a performant way at enterprise scale seems to involve turning every function call into an asynchronous, queued service of some sort. Which then begets additional deployment and monitoring services. A queued problem requires a cute solution, bringing acute pain.
      • vergessenmir 2y ago
        some writes might fail, you may need to retry, the data store may be temporarily available etc. There may be many things that go wrong and how you handle this depends on your data guarantees and consistency requirements. If you're not queuing what are you doing when a write fails, throwing away the data?
        • bushbaba 2y ago
• YetAnotherNick 2y ago
  I wish there is a global file system with node local disks, which has rule driven affinity to nodes for data. We have two extremes, one like EFS or S3 express which doesn't have any affinity to the processing system, and other what Kafka etc is doing where they have tightly integrated logic for this which makes systems more complicated.
  • XorNot 2y ago
    I might be misunderstanding but isn't this like, literally what GlusterFS used to do? Like distinctly recall running it at home as a global unioning filesystem where content had to fully fit on the specific device it was targeted at (rather then being striped or whatever).
• imcritic 2y ago
  Since we are dreaming - add ETL there as well!
• vermon 2y ago
  Interesting, if partitioning is not a useful concept of Kafka, what are some of the better alternatives for controlling consumer concurrency?
  • galeaspablo 2y ago
    It is useful, but it is not generally applicable. Given an arbitrary causality graph between n messages, it would be ideal if you could consume your messages in topological order. And that you could do so in O(n log n). No queuing system in the world does arbitrary causality graphs without O(n^2) costs. I dream of the day where this changes. And because of this, we’ve adapted our message causality topologies to cope with the consuming mechanisms of Kafka et al To make this less abstract, imagine you have two bank accounts, each with a stream. MoneyOut in Bob’s account should come BEFORE MoneyIn when he transfers to Alice’s account, despite each bank account having different partition keys.
    • dpflan 2y ago
      Can you elaborate on how you have “adapted…message causality topologies to cope with consuming mechanisms” in relation to the example of a bank account? The causality topology being what here, couldn’t one day MoneyIn should come before else there can be now true MoneyOut?
      • galeaspablo 2y ago
        Right on, great question. Some examples: Example Option 1 You give up on the guarantees across partition keys (bank accounts), and you accept that balances will not reflect a causally consistent state of the past. E.g., Bob deposits 100, Bob sends 50 to Alice. Balances: Bob 0 Alice 50 # the source system was never in this state Bob 100 Alice 50 # the source system was never in this state Bob 50 Alice 50 # eventually consistent final state Example Option 2 You give up on parallelism, and consume in total order (i.e., one single partition / unit of parallelism - e.g., in Kafka set a partitioner that always hashes to the same value). Example Option 3 In the consumer you "wait" whenever you get a message that violates causal order. E.g., Bob deposits 100 Bob sends 50 to Alice (Bob-MoneyOut 50 -> Alice-MoneyIn 50). If we attempt to consume Alice-MoneyIn before Bob-MoneyOut, we exponentially back off from the partition containing Alice-MoneyIn. (Option 3 is terrible because of O(n^2) processing times in the worst case and the possibility for deadlocks (two partitions are waiting for one another))
• Mistletoe 2y ago
  I know it’s not what the article is about but I really wish we could rebuild Franz Kafka and hear what he thought about the tech dystopia we are in. >I cannot make you understand. I cannot make anyone understand what is happening inside me. I cannot even explain it to myself. -Franz Kafka, The Metamorphosis
  • tannhaeuser 2y ago
    They named it like that for a reason ;)
  • gherkinnn 2y ago
    Both der Proceß and das Schloß apply remarkably well to our times. No extra steps necessary.
• jackbauer24 2y ago
  [dead]
• elvircrn 2y ago
  Surprised there's no mention of Redpanda here.
  • axegon_ 2y ago
    Having used both Kafka and Redpanda on several occasions, I'd pick Redpanda any day of the week without a second thought. Easier to setup, easier to maintain, a lot less finicky ans uses a fraction of the resources.
    • enether 2y ago
      In what way is it materially easier to maintain and less finicky? I read a lot about this but I haven’t seen a concise bullet point list of why, which leads me to naturally distrust such claims. Ditto for the resources - Kafka is usually bottlenecked on disk/network, and whether it’s c++ or not doesn’t solve that
      • SkipperCat 2y ago
        I've been using Redpanda for a few years now and here's what I've noticed. Its a compiled binary - no JVM to manage. Java apps have always been a headache for me. Plus, no zookeeper - one less thing to break. The biggest benefit I've seen is the performance. Redpanda just outperforms Apache Kakfa on similar hardware. Its also Kafka compliant in every way I've noticed, so all my favorite tools that interact with Kafka work the same with Redpanda. Redpanda, like Kafka, writes to disk, so you'll always be limited by your hardware no matter what you use (but NVMe's are fast and affordable). YMMV, but its been a good experience for me.
    • dangoodmanUT 2y ago
      same
  • EdwardDiego 2y ago
    Redpanda isn't FOSS. Apache Kafka is.
• fintler 2y ago
  Keep an eye out for Northguard. It's the name of LinkedIn's rewrite of Kafka that was announced at a stream processing meetup about a week ago.
  • selkin 2y ago
    It solves some issues, and creates some, since Northguard isn’t compatible with the current Kafka ecosystem. As such, you can no longer use existing software that is built on Kafka as-is. It may not be a grave concern for LinkedIn, but it could be for others that currently benefit from using the existing Kafka ecosystem.
    • fintler 2y ago
      Yeah, it's definitely a significant shift. The Xinfra component helps with Kafka compatibility, but that still has quite a bit of complexity to it. Also, it's written in C++, so that requires a different mindset to operate.
      • selkin 2y ago
        I understood Xinfra to not use the Kafka protocol as its API. As such, even with Xinfra deployed, you have to rewrite all the software that connects to Kafka, regardless of programming language.
• ghuntley 2y ago
  I'm going to get downvoted for this, but you can literally rebuild Kafka via AI right now in record time using the steps detailed at https://ghuntley.com/z80 https://ghuntley.com/z80. I'm currently building a full workload scheduler/orchestrator. I'm sick of Kubernetes. The world needs better -> https://x.com/GeoffreyHuntley/status/1915677858867105862 https://x.com/GeoffreyHuntley/status/1915677858867105862
  • EdwardDiego 2y ago
    > but you can literally rebuild Kafka via AI right now in record time Go on then. Post the repo when you have :)
• lewdwig 2y ago
  Ah the siren call of the ground-up rewrite. I didn’t know how deep the assumption of hard disks underpinning everything is baked into its design. But don’t public cloud providers already all have cloud-native event sourcing? If that’s what you need, just use that instead of Kafka.
• frklem 2y ago
  "Faced with such a marked defensive negative attitude on the part of a biased culture, men who have knowledge of technical objects and appreciate their significance try to justify their judgment by giving to the technical object the only status that today has any stability apart from that granted to aesthetic objects, the status of something sacred. This, of course, gives rise to an intemperate technicism that is nothing other than idolatry of the machine and, through such idolatry, by way of identification, it leads to a technocratic yearning for unconditional power. The desire for power confirms the machine as a way to supremacy and makes of it the modern philtre (love-potion)." Gilbert Simondon, On the mode of existence of technical objects. This is exactly what I interpret from these kind of articles: engineering just for the cause of engineering. I am not saying we should not investigate on how to improve our engineered artifacts, or that we should not improve them. But I see a generalized lack of reflection on why we should do it, and I think it is related to a detachment from the domains we create software for. The article suggests uses of the technology that come from so different ways of using it, that it looses coherence as a technical item.
  • gunnarmorling 2y ago
    For each of the items discussed I explicitly mention why they would be desirable to have. How is this engineering for the sake of engineering?
    • frklem 2y ago
      True, for each of the points discussed, there is an explicit mention on why it is desirable. But those are technical solutions, to technical problems. There is nothing wrong with that. The issue is, that the whole article is about technicalities because of technicalities, hence the 'engineering for the cause of engineering' (which is different from '.. for the sake of...'). It is at this point that the 'idea of rebuilding Kafka' becomes a pure technical one, detached from the intention of having something like Kafka. Other commenters in the thread also pointed out to the fact of Kafka not having a clear intention. I agree that a lot of software nowadays suffer from the same problem.
      • deleted 2y ago
• supermatt 2y ago
  > "Do away with partitions" > "Key-level streams (... of events)" When you are leaning on the storage backend for physical partitioning (as per the cloud example, where they would literally partition based on keys), doesnt this effectively just boil down to renaming partitions to keys, and keys to events?
  • gunnarmorling 2y ago
    That's one way to look at this, yes. The difference being that keys actually have a meaning to clients (as providers of ordering and also a failure domain), whereas partitions in their current form don't.
    • supermatt 2y ago
      > partitions in their current form don't That depends on how you are using partitions. A partition per topic is effectively going to give you exactly that. What you call a key is then just a topic, and hierarchy (including multitenancy and other forms of namespacing) can be implemented via topic naming convention. This isn't even a novel way to use kafka - its quite a common approach in practice. Obviously this then comes at the cost of throughput - which is exactly why systems that use these approaches are often much slower than partitioned topics in kafka. Even in your object store example there needs to be synchronization across the storage partitions, and that overhead will give you substantially reduced throughput - as you are effectively using a distributed lock for each write to the complete quorum.
• olavgg 2y ago
  How many of the Apache Kafka issues are adressed by switching to Apache Pulsar? I skipped learning Kafka, and jumped right into Pulsar. It works great for our use case. No complaints. But I wonder why so few use it?
  • EdwardDiego 2y ago
    Some, but then Pulsar brings its own issues.
  • enether 2y ago
    There’s inherently a lot of path-dependent network effects in open source software. Just because something is 10-30% better in certain cases almost never warrants its adoption, if on the other side you get much less human expertise, documentation/resources and battle tested testimonies. This, imo, is the story of most Kafka competitors
  • jsumrall 2y ago
    I've been down this path, and if my experience is more common, then it really boils down to the classic "Nobody gets fired for buying IBM", and here IBM -> Confluent. StreamNative seems like an excellent team, and I hope they succeed. But as another comment has written, something (puslar) being better (than kafka) has to either be adopted from the start, or be a big enough improvement to change— and as difficult and feature-poor that Kafka is, it still gets the job done. I can rant longer about this topic but Pulsar _should_ be more popular, but unfortunately Confluent has dominated here and rent-seeking this field into the ground.
• mgaunard 2y ago
  I can't count the number of bad message queues and buses I've seen in my career. While it would be useful to just blame Kafka for being bad technology it seems many other people get it wrong, too.
  • mgaunard 2y ago
    s/useful/easy/ (can't edit)
• 0x445442 2y ago
  How about logging the logs so I can shell into the server to search the messages.
  • EdwardDiego 2y ago
    What messages? They're just blobs of bytes.
  • jjk7 2y ago
    You can do this with Confluent
• tyingq 2y ago
  He mentions Automq right in the opener. And if I follow the link, they pitch it in a way that sounds very "too good to be true". Anyone here have some real world experience with it?
  • wanshao 2y ago
    [dead]
• galeaspablo 2y ago
  Agreed. The head of line problem is worth solving for certain use cases. But today, all streaming systems (or workarounds) with per message key acknowledgements incur O(n^2) costs in either computation, bandwidth, or storage per n messages. This applies to Pulsar for example, which is often used for this feature. Now, now, this degenerate time/space complexity might not show up every day, but when it does, you’re toast, and you have to wait it out. My colleagues and I have studied this problem in depth for years, and our conclusion is that a fundamental architectural change is needed to support scalable per message key acknowledgements. Furthermore, the architecture will fundamentally require a sorted index, meaning that any such a queuing / streaming system will process n messages in O (n log n). We’ve wanted to blog about this for a while, but never found the time. I hope this comment helps out if you’re thinking of relying on per message key acknowledgments; you should expect sporadic outages / delays.
  • cogman10 2y ago
    > streaming system will process n messages in O (n log n) I'm guessing this is mostly around how backed up the stream is. n isn't the total number of messages but rather the current number of unacked messages. Would a radix structure work better here? If you throw something like a UUID7 on the messages and store them in a radix structure you should be able to get O(n) performance here correct? Or am I not understanding the problem well.
    • bjornsing 2y ago
      I think the problem is that if you want quick access to all messages with a particular key then you have to maintain some kind of index over all persisted messages. So n would be total number of persisted messages as I read it, which can be quite large. But even storing them in the first place is O(n), so O(n log n) might not be so bad.
      • galeaspablo 2y ago
        That's correct. And keep in mind that you might have new consumers starting from the beginning come into play, so you have to permanently retain the indexes. And yes, O(n log n ) is not bad at all. Sorted database indexes (whether SQL, NoSQL, or AcmeVendorSQL, etc.) already take O(n log n) to insert n elements into data storage or to read n elements from data storage.
• rvz 2y ago
  Every time another startup falls for the Java + Kafka arguments, it keeps the AWS consultants happier. Fast forward into 2025, there are many performant, efficient and less complex alternatives to Kafka that save you money, instead of burning millions in operational costs "to scale". Unless you are at a hundred million dollar revenue company, choosing Kafka in 2025 is doesn't make sense anymore.
  • mrkeen 2y ago
    When I pitched Kafka to my backend team in 2018, I got pushback from the ops team who wanted me to use something AWS-native instead, i.e. Kinesis. A big part of why I doubled-down on Kafka is because it's vendor neutral and I can just run the actual thing locally.
  • EdwardDiego 2y ago
    Wut. Kafka shouldn't be used for low dataflow systems, true, but you can scale a long way with a simple 3 node cluster.
  • enether 2y ago
    Can you share three? Without concrete suggestions, this is just a disparaging narrative (that a lot of vendors use)
• bionhoward 2y ago
  step 1: don’t use the JVM
  • nthingtohide 2y ago
    do you think Grafana made the right choice by using Golang ?
  • openplatypus 2y ago
    Don't use best platform engineered and perfected over decades? Why?
  • Sohcahtoa82 2y ago
    It's not the 90s anymore. The JVM is very performant.
    • pas 2y ago
      Yes, but if you can use compile-time (arena or otherwise) allocation then you can go even faster, no?
• dangoodmanUT 2y ago
  I think this is missing a key point about partitions: Write visibility ordering The problem with guaranteed order is that you have to have some agreed upon counter/clock for ordering, otherwise a slow write from one producer to S3 could result in consumers already passing that offset before it was written, thus the write is effectively lost unless the consumers wind-back. Having partitions means we can assign a dedicated writer for that partition that guarantees that writes are in order. With s3-direct writing, you lose that benefit, even with a timestamp/offset oracle. You'd need some metadata system that can do serializable isolation to guarantee that segments are written (visible to consumers) in the order of their timestamps. Doing that transactional system directly against S3 would be super slow (and you still need either bounded-error clocks, or a timestamp oracle service).
  • BeeOnRope 2y ago
    Writes to the log don't need to be in the order of the producers timestamp, they just need to be in some (and respect ack to produce causality, etc).
• iwontberude 2y ago
  We could (and will repeatedly) rebuild Kafka from scratch. Solved the question and I didn’t even need to read the article.
• bjornsing 2y ago
  > Key-centric access: instead of partition-based access, efficient access and replay of all the messages with one and the same key would be desirable. I’ve been working on a datastore that’s perfect for this [1], but I’m getting very little traction. Does anyone have any ideas why that is? Is my marketing just bad, or is this feature just not very useful after all? 1. https://www.haystackdb.dev/ https://www.haystackdb.dev/
  • galeaspablo 2y ago
    Some input from previously working on a superset of this problem. And being in a similar position. Mature projects have too much bureacracy, and even spending time talking to you = opportunity cost. So making a case for why you're going to solve a problem for them is tough. New projects (whether at big companies or small companies) have 20 other things to worry about, so the problem isn't big enough. I wrote about this in our blog if you're curious: https://ambar.cloud/blog/a-new-path-for-ambar https://ambar.cloud/blog/a-new-path-for-ambar
    • bjornsing 2y ago
      Thanks. Interesting read, and an interesting product / service. Have been thinking about the same approach myself…
  • notfed 2y ago
    The website seems very vapid. Extraordinary claims require extraordinary evidence. Personally I see a lack of evidence here (that this vendor-locked product is better than existing freeware) and I'm going to move on.
    • bjornsing 2y ago
      Thanks for the honest assessment!
  • tinix 2y ago
    what can you do that redis can't? I'm also skeptical of the graph on your front page that claims S3 cost as much as DynamoDB. that alone makes it look like total nonsense. as someone else said, extraordinary claims require extraordinary evidence.
• eluusive 2y ago
  This is basically NATS.io
• derefr 2y ago
  > You either want to have global ordering of all messages on a given topic, or (more commonly) ordering of all messages with the same key. In contrast, defined ordering of otherwise unrelated messages whose key happens to yield the same partition after hashing isn’t that valuable, so there’s not much point in exposing partitions as a concept to users. The user gets global ordering when 1. you-the-MQ assign both messages and partitions stable + unique + order + user-exposed identifiers; 2. the user constructs a "globally-collatable ID" from the (perPartitionMsgSequenceNumber, partitionID) tuple; 3. the user does a client-side streaming merge-sort of messages received by the partitions, sorting by this collation ID. (Where even in an ACK-on-receive design, messages don't actually get ACKed until they exit the client-side per-partition sort buffer and enter the linearized stream.) The definition of "exposed to users" is a bit interesting here, as you might think you could do this merge-sort on the backend, just exposing a pre-linearized stream to the client. But one of the key points/benefits of Kafka-like systems, under high throughput load (which is their domain of comparative advantage, and so should be assumed to be the deployed use-case), is that you can parallelize consumption cheaply, by just assigning your consumer-workers partitions of the topic to consume. And this still works under global ordering, under some provisos: • your workload can be structured as a map/reduce, and you don't need global ordering for the map step, only the reduce step; • it's not impractical for you to materialize+embed the original intended input collation-ordering into the transform workers' output (because otherwise it will be lost in all but very specific situations.) Plenty of systems fit these constraints, and happily rely on doing this kind of post-linearized map/reduce parallelized Kafka partition consumption. And if you "hide" this on an API level, this parallelization becomes impossible. Note, however, that "on an API level" bit. This is only a problem insofar as your system design is protocol-centered, with the expectation of "cheap, easy" third-party client implementations. If your MQ is not just a backend, but also a fat client SDK library — then you can put the partition-collation into the fat client, and it will still end up being "transparent" to the user. (Save for the user possibly wondering why the client library opens O(K) TCP connections to the broker to consume certain topics under certain configurations.) See also: why Google's Colossus has a fat client SDK library.
• debadyutirc 2y ago
  This is a question we asked 6 years ago. What if we wrote it in Rust. And leveraged and WASM. We have been at it for the past 6 years. https://github.com/infinyon/fluvio https://github.com/infinyon/fluvio For the past 2 years we have also been building Flink using Rust and WASM. https://github.com/infinyon/stateful-dataflow-examples/ https://github.com/infinyon/stateful-dataflow-examples/
  • gregfurman 2y ago
    Fluvio looks awesome! Any chance you’re going to be reviving support for the Kafka wire protocol? https://github.com/infinyon/fluvio/issues/4259 https://github.com/infinyon/fluvio/issues/4259
    • deleted 2y ago
      [deleted]
    • debadyutirc 2y ago
      We have had folks over the years asking us about the Kafka wire compatibility. We had a project 3 years ago which we archived. I think we have a case for reviving it in the near future.
  • Micoloth 2y ago
    Interesting! How would you say your project compares to Arroyo?
    • debadyutirc 2y ago
      Fluvio is streaming transport. And we built Stateful DataFlow on top of that for Stream Processing. Arroyo is SQL first stream processing. Fluvio is streaming transport which can send data to Arroyo and there is an integration. Stateful DataFlow and Arroyo are similar in the stream processing pattern and the use of Apache Arrow. The interfaces are different. Fluvio and Stateful DataFlow support for SQL is the same dialect as columnar SQL supported by Polars. The Fluvio and Stateful DataFlow paradigm is more intricate more expressive and the platform is broader and deeper.
• gitroom 2y ago
  honestly, kafka always felt like way more moving parts than my brain wants to track, but at the same time, its kinda impressive how the ecosystem just keeps growing - you think the reason people stick to it is hype, laziness, or just not enough real pain yet to switch?
• selkin 2y ago
  This is a useful Gedankenexperiment, but I think the replies suggesting that the conclusion is that we should replace Kafka with something new are quiet about what seems obvious to me: Kafka's biggest strength is the wide and useful ecosystem built on top of it. It is also a weaknesses, as we have to keep some (but not of all) the design decisions we wouldn't have made had we started from scratch today. Or we could drop backwards compatibility, at the cost of having to recreate the ecosystem we already have.
• tezza 2y ago
  Now that really would be Kafka-esque
  • rhet0rica 2y ago
    One morning, when Hacker News woke from troubled dreams, it found itself transformed in its bed into a horrible bikeshed.
    • pas 2y ago
      no no, into a hideous design document. /s ... seriously
• redditor98654 2y ago
  I agree on the head of the line blocking problem and that not everyone needs the per partition ordering. For that I have started to use SQS FIFO with the message grouping key being the logical key for the event/resource. This gives me ordering within the key and not extra ordering across keys. So I don’t have the head of line blocking problem. If I need multiple independent consumers, I just instead publish to SNS FIFO and let my consumers create their own SQS fifo queues that are subscribed to the topic. The ordering is maintained across SNS and SQS. I also get native DLQ support for poison pills and an SQS consumer is dead simple to operate vs a Kafka consumer. It does not solve all of the mentioned problems like being able to see what the keys are in the queue or lookup by a given key but as a messaging solution that offers ordering for a key, this is hard to beat.
  • renatomefi 2y ago
    How do you handle DLQ for ordered keys? I assume if you drop a message then you lose semantics
    • redditor98654 2y ago
      Correct, you lose that ordering but then the only other option is to retry indefinitely. You have to think through what your error handling looks like as is the case with any application.
• smittywerben 2y ago
  I don't understand how everyone hates Kafka I use it as a typed write-ahead JSON log with library support for most languages. Yes the systems I've built with this were overengineered but it worked and was reliable. I just bought a larger disk instead of using whatever remains of the great battle of the zookeeper. I just assumed the fact it has any integration support with standard RDBMs must be a byproduct of being Java as purely an accident.