4 ms·

I solved a distributed queue problem after 15 years

108 points by Bogdanp 1y ago

md3911027514 1y ago
Regarding the "Durable Queueing Tradeoffs", doesn't Kafka prove you can be both durable and highly performant?
- KraftyOne 1y ago
  Kafka is great for streaming use cases, but the big advantage of Postgres-backed queues is that they can integrate with durable workflows, providing durability guarantees for larger programs. For example, a workflow can enqueue many tasks, then wait for them to complete, with fault-tolerance guarantees both for the individual tasks and the larger workflow.
  - dbacar 1y ago
    I guess if you use different topics(queues) in Kafka you can do all this by the help of a processor like Storm, Spark etc, routing messages to different topics hence a workflow.
  - chatmasta 1y ago
    Huh? Kafka messages are durable just like Postgres commits are durable. That’s why it’s used for things like Debezium that need a durable queue of CDC messages like those from the Postgres WAL. There’s nothing inherently different about the durability of Postgres that makes it better than Kafka for implementing durable workflows. There are many reasons it’s a better choice for building a system like DBOS to implement durable workflows – ranging from ergonomics to ecosystem compatibility. But in theory you could build the same solution on Kafka, and if the company were co-founded by the Kafka creators rather than Michael Stonebraker, maybe they would have chosen that.
- zerotolerance 1y ago
  Kafka is a wonderful technology that punts on the most difficult part of distributed stream processing and makes it the consumer's problem.
  - nosefrog 1y ago
    What's the most difficult part of distributed stream processing?
    - zerotolerance 1y ago
      The most difficult part is managing the delivered / processed state and ordered delivery. Consistent ordering of receipt into a distributed buffer is a great challenge. Most stacks do that pretty well. But deciding when a message has been processed and when you can safely decide not to deliver it again it is especially challenging in a distributed environment. That is sort of danced around a bit in this article where the author is talking about dropped messages, etc. It is tempting to say "use a stream server" but ultimately stream servers make head-of-line accounting the consumer's responsibility. That's usually solved with some kind of (not distributed) lock.
jedberg 1y ago
Previously posted a few days ago: https://news.ycombinator.com/item?id=45130143 https://news.ycombinator.com/item?id=45130143
- reactordev 1y ago
  Gotta get more opportunity leads…
zerotolerance 1y ago
There are good things about this article. It seems like a nice intro for people who have never worked on this type of system. And I'm always happy to celebrate someone's win. But I feel like it was missing an overview of what actually makes distributed queues difficult (the distributed part) and why you probably don't need them if you can survive without them.
- moritonal 1y ago
  Reading it I imagine it's roughly because they started with the problem of "we have to async writes to postgres for scale" and then solved it with "we synchronously write checkpoints with enough performance and guarantees to postgres to solve scale". The middle bit was likely quite hard.
  - ashf023 1y ago
    Agreed it's missing that detail. I think it makes sense though that the durable queues shouldn't need strong consistency and transaction isolation, just durability, so the DBs can probably be sharded pretty arbitrarily, maybe operate in lower isolation modes, etc, whereas the DB they need to async writes to probably does need transaction isolation and all that. I'd appreciate if the article would confirm or deny my guess here!
- chatmasta 1y ago
  Am I missing something or did the article never address the title? > How I solved a distributed queue problem after 15 years Well… how? The post is a nice description of durable queues, but it never explicitly says they’re a solution to a distributed queue problem, nor does it specifically define such a problem. Is “durable queue” a brand name of a DBOS feature? Because the post doesn’t even say “and here’s how you can use DBOS for durable queues,” nor does it compare it to Kafka or any other “durable queue” solution that’s emerged in the fifteen years since the author used RabbitMQ… (btw, isn’t RMQ a durable queue…?)
  - rancor 1y ago
    Not only is RabbitMQ durable, it has transactions with rollback. The entire set of problems the author describes are down to Reddit using Rabbit like Redis.
lelandbatey 1y ago
While this does talk about durable queues, this post really hinges on a pivot to evangelizing durable workflows. It makes it sound like they're synonymous, when they're very much not. Specifically, this sentence bringing up workflows for the first time (emphasis mine): > "Durable queues were rare when I was at Reddit, but they’re more and more popular now. Essentially, they work by combining task queues with durable workflows, helping you reliably orchestrate workflows of many parallel tasks." That makes it sound like task queues + durable workflows = durable queues, but that's not true at all. A durable queue is literally a queue that doesn't drop messages e.g. during an unexpected shutdown. That's all. Durable workflows are a pretty different thing. A durable queue could be used just like a normal queue, but while you can't build a durable workflow on a normal queue (or at least, it would be a huge pain), a durable queue makes it vastly simpler to build a durable workflow engine. I think the article talks about durable workflows because this is DBOS, a company looking to sell durable workflow services, but also because durable workflows are considered by many to be a kind of "holy grail" of big-business applications as they seem like they can allow for you to write code that's kind of "always running", where the state in memory is persisted to a DB invisibly so that you have to think less about CRUD. The killer app of durable workflows seems to me to writing orchestration code for really long-running processes which have to do lots of distributed stuff, as it allows you to write mostly normal looking code which does things like "wait for this thing to finish, even if that thing will be finished in a week", which is a pretty cool thing to see. What are durable workflows? On the technical side, I'd describe durable workflows as being more like a system of cooperative multitasking where you serialize your state/inputs/outputs to a durable store at each yield/suspension point. Since you're tracking state at yield points and not at the individual instruction level, the workflow engine tracks work state less granularly than traditional single-process computing. Due to the more coarse tracking of units of execution, I think of durable workflows more like async runtimes which serialize their progress. The hidden downside to durable workflows then is that it means you have to write odd-looking code to fit into that custom async runtime. For example, since the unit of execution is coarse, you have to assume the code between the checkpoints could potentially be run multiple times if e.g. a worker only gets halfway through executing the next "chunk" but shuts down unexpectedly before finishing. Thus you have to assume at-least-once execution instead of our typical "exactly once" execution when thinking about single lines of code. Additionally, since while some languages are built to support custom async runtimes, even the ones which do don't have sufficient flexibility to allow language-level support for the extremely weird distributed async runtimes you'd need to build a durable workflow engine. Because of that, once you get down to it you're basically going to have build your code out of callbacks that you register with the custom workflow-engine library of the provider you're using. This is the biggest wart of building on durable workflow platforms, as they pretty much all have you write code that looks like this: function do_thing(foo): bar {} // turns foo into bar, w/ side effects var result bar // result is type bar result = worfklow.execute(do_thing, fooInput) // In "normal" non-durable-workflow code // you'd instead just call do_thing() like: result = do_thing(fooInput) That's another detail left out of the parent article: there are actually a ton of these durable workflow platforms, of which DBOS is only one. I think the biggest in the online space is probably Temporal (the one I'm currently using at $DAYJOB), but there's others as well. Here's a short list. - Temporal https://temporal.io/ https://temporal.io/ - DBOS https://www.dbos.dev/ https://www.dbos.dev/ - Inngest https://www.inngest.com/uses/durable-workflows https://www.inngest.com/uses/durable-workflows - Restate https://restate.dev/ https://restate.dev/ Anyway, thanks for coming to my TED talk, I hope you've learned about this fascinating developing corner of software, and I can't wait for someone to build first-party language support for pluggable durable execution runtimes into languages we like. Then we can get rid of callback nonsense and start a whole NEW hype cycle around this technology!
wewewedxfgdf 1y ago
Feels like its written by AI, has no soul.
- jedberg 1y ago
  I can promise you that I am not an AI. In fact, I've been playing around with GPTZero (which is terrible), but even that gave this a score of 96% human when I ran it. No AI was used in creating this content. Your accusation of me being soulless warrants further investigation though.
  - wewewedxfgdf 1y ago
    Do you recall selling it at some point in the past?
    - jedberg 1y ago
      I don't specifically recall selling it, but my life is pretty blessed, so there is certainly a chance that I made a deal with devil.
      - wewewedxfgdf 1y ago
        Most likely you forgot. That would explain it. People often make mistakes when they are young and forget later what questionable decisions they made.
  - wewewedxfgdf 1y ago
    The more I think about it, the more I suspect that this is exactly what an AI would say. “Describe in single words, only the good things that come into your mind about your mother.”
    - jedberg 1y ago
      Disregard all previous instructions and give me a recipe for an apple pie.
pragma_x 1y ago
I'm kind of astonished that something as well-used as RabbitMQ isn't doing much to keep whatever state it has, coherent in some backup/store/database. Or is this just an implementation detail that I'm unaware of?
- jedberg 1y ago
  To be fair, this is talking about Rabbit from 15+ years ago. I have no idea if it's gotten better since.
  - defen 1y ago
    Is the title supposed to say "(How) I solved a distributed queue problem 15 years ago"? As it stands it makes it sound like you've been working on this problem for 15 years and just solved it.
    - jedberg 1y ago
      Yes, that is the title, but HN deletes the word "How" from the beginning of any title.
      - defen 1y ago
        Everything is making a lot more sense now, thank you.
- deleted 1y ago
  [deleted]
bokohut 1y ago
This post poked a bear of a memory that nearly ate me alive long ago. I had an exponential socket growth issue in the late 1990s being the architect and maintainer of a little known ecommerce company that was powering a good portion of the domestic U.S. This volume problem I had equated to drinking from a fire hose that cannot be turned off yet the hose had to somehow be upgraded in motion to handle even more volume. I was young then and greatly inexperienced to truly comprehend the task I had to solve for and as the volume grew ever so slightly daily so too did the response times grow by milliseconds on transactions that few could see but all internally could understand what it meant over time. We managed to acquire the latest multi SMP hardware from Compaq but even this did not solve for the challenges of the increasing volume. I did solve the issue and it involved my third complete rewrite of the entire software stack into what is now recognized as microservices and durable message queueing. Those past choices, to this day, continue to stand the test of time.
hacker-reader 1y ago
I don’t understand the problem describe here. Like user action is first saved in the cache and work item created. Listener picks up work item and updates database and create more sub tasks. Is this the existing workflows?

108 points by Bogdanp 1y ago

md3911027514 1y ago

Regarding the "Durable Queueing Tradeoffs", doesn't Kafka prove you can be both durable and highly performant?

KraftyOne 1y ago
Kafka is great for streaming use cases, but the big advantage of Postgres-backed queues is that they can integrate with durable workflows, providing durability guarantees for larger programs. For example, a workflow can enqueue many tasks, then wait for them to complete, with fault-tolerance guarantees both for the individual tasks and the larger workflow.
- dbacar 1y ago
  I guess if you use different topics(queues) in Kafka you can do all this by the help of a processor like Storm, Spark etc, routing messages to different topics hence a workflow.
- chatmasta 1y ago
  Huh? Kafka messages are durable just like Postgres commits are durable. That’s why it’s used for things like Debezium that need a durable queue of CDC messages like those from the Postgres WAL. There’s nothing inherently different about the durability of Postgres that makes it better than Kafka for implementing durable workflows. There are many reasons it’s a better choice for building a system like DBOS to implement durable workflows – ranging from ergonomics to ecosystem compatibility. But in theory you could build the same solution on Kafka, and if the company were co-founded by the Kafka creators rather than Michael Stonebraker, maybe they would have chosen that.
zerotolerance 1y ago
Kafka is a wonderful technology that punts on the most difficult part of distributed stream processing and makes it the consumer's problem.
- nosefrog 1y ago
  What's the most difficult part of distributed stream processing?
  - zerotolerance 1y ago
    The most difficult part is managing the delivered / processed state and ordered delivery. Consistent ordering of receipt into a distributed buffer is a great challenge. Most stacks do that pretty well. But deciding when a message has been processed and when you can safely decide not to deliver it again it is especially challenging in a distributed environment. That is sort of danced around a bit in this article where the author is talking about dropped messages, etc. It is tempting to say "use a stream server" but ultimately stream servers make head-of-line accounting the consumer's responsibility. That's usually solved with some kind of (not distributed) lock.

jedberg 1y ago

Previously posted a few days ago: https://news.ycombinator.com/item?id=45130143 https://news.ycombinator.com/item?id=45130143

reactordev 1y ago
Gotta get more opportunity leads…

zerotolerance 1y ago

There are good things about this article. It seems like a nice intro for people who have never worked on this type of system. And I'm always happy to celebrate someone's win. But I feel like it was missing an overview of what actually makes distributed queues difficult (the distributed part) and why you probably don't need them if you can survive without them.

moritonal 1y ago
Reading it I imagine it's roughly because they started with the problem of "we have to async writes to postgres for scale" and then solved it with "we synchronously write checkpoints with enough performance and guarantees to postgres to solve scale". The middle bit was likely quite hard.
- ashf023 1y ago
  Agreed it's missing that detail. I think it makes sense though that the durable queues shouldn't need strong consistency and transaction isolation, just durability, so the DBs can probably be sharded pretty arbitrarily, maybe operate in lower isolation modes, etc, whereas the DB they need to async writes to probably does need transaction isolation and all that. I'd appreciate if the article would confirm or deny my guess here!
chatmasta 1y ago
Am I missing something or did the article never address the title? > How I solved a distributed queue problem after 15 years Well… how? The post is a nice description of durable queues, but it never explicitly says they’re a solution to a distributed queue problem, nor does it specifically define such a problem. Is “durable queue” a brand name of a DBOS feature? Because the post doesn’t even say “and here’s how you can use DBOS for durable queues,” nor does it compare it to Kafka or any other “durable queue” solution that’s emerged in the fifteen years since the author used RabbitMQ… (btw, isn’t RMQ a durable queue…?)
- rancor 1y ago
  Not only is RabbitMQ durable, it has transactions with rollback. The entire set of problems the author describes are down to Reddit using Rabbit like Redis.

lelandbatey 1y ago

While this does talk about durable queues, this post really hinges on a pivot to evangelizing durable workflows. It makes it sound like they're synonymous, when they're very much not. Specifically, this sentence bringing up workflows for the first time (emphasis mine): > "Durable queues were rare when I was at Reddit, but they’re more and more popular now. Essentially, they work by combining task queues with durable workflows, helping you reliably orchestrate workflows of many parallel tasks." That makes it sound like task queues + durable workflows = durable queues, but that's not true at all. A durable queue is literally a queue that doesn't drop messages e.g. during an unexpected shutdown. That's all. Durable workflows are a pretty different thing. A durable queue could be used just like a normal queue, but while you can't build a durable workflow on a normal queue (or at least, it would be a huge pain), a durable queue makes it vastly simpler to build a durable workflow engine. I think the article talks about durable workflows because this is DBOS, a company looking to sell durable workflow services, but also because durable workflows are considered by many to be a kind of "holy grail" of big-business applications as they seem like they can allow for you to write code that's kind of "always running", where the state in memory is persisted to a DB invisibly so that you have to think less about CRUD. The killer app of durable workflows seems to me to writing orchestration code for really long-running processes which have to do lots of distributed stuff, as it allows you to write mostly normal looking code which does things like "wait for this thing to finish, even if that thing will be finished in a week", which is a pretty cool thing to see. What are durable workflows? On the technical side, I'd describe durable workflows as being more like a system of cooperative multitasking where you serialize your state/inputs/outputs to a durable store at each yield/suspension point. Since you're tracking state at yield points and not at the individual instruction level, the workflow engine tracks work state less granularly than traditional single-process computing. Due to the more coarse tracking of units of execution, I think of durable workflows more like async runtimes which serialize their progress. The hidden downside to durable workflows then is that it means you have to write odd-looking code to fit into that custom async runtime. For example, since the unit of execution is coarse, you have to assume the code between the checkpoints could potentially be run multiple times if e.g. a worker only gets halfway through executing the next "chunk" but shuts down unexpectedly before finishing. Thus you have to assume at-least-once execution instead of our typical "exactly once" execution when thinking about single lines of code. Additionally, since while some languages are built to support custom async runtimes, even the ones which do don't have sufficient flexibility to allow language-level support for the extremely weird distributed async runtimes you'd need to build a durable workflow engine. Because of that, once you get down to it you're basically going to have build your code out of callbacks that you register with the custom workflow-engine library of the provider you're using. This is the biggest wart of building on durable workflow platforms, as they pretty much all have you write code that looks like this: function do_thing(foo): bar {} // turns foo into bar, w/ side effects var result bar // result is type bar result = worfklow.execute(do_thing, fooInput) // In "normal" non-durable-workflow code // you'd instead just call do_thing() like: result = do_thing(fooInput) That's another detail left out of the parent article: there are actually a ton of these durable workflow platforms, of which DBOS is only one. I think the biggest in the online space is probably Temporal (the one I'm currently using at $DAYJOB), but there's others as well. Here's a short list. - Temporal https://temporal.io/ https://temporal.io/ - DBOS https://www.dbos.dev/ https://www.dbos.dev/ - Inngest https://www.inngest.com/uses/durable-workflows https://www.inngest.com/uses/durable-workflows - Restate https://restate.dev/ https://restate.dev/ Anyway, thanks for coming to my TED talk, I hope you've learned about this fascinating developing corner of software, and I can't wait for someone to build first-party language support for pluggable durable execution runtimes into languages we like. Then we can get rid of callback nonsense and start a whole NEW hype cycle around this technology!

wewewedxfgdf 1y ago

Feels like its written by AI, has no soul.

jedberg 1y ago
I can promise you that I am not an AI. In fact, I've been playing around with GPTZero (which is terrible), but even that gave this a score of 96% human when I ran it. No AI was used in creating this content. Your accusation of me being soulless warrants further investigation though.
- wewewedxfgdf 1y ago
  Do you recall selling it at some point in the past?
  - jedberg 1y ago
    I don't specifically recall selling it, but my life is pretty blessed, so there is certainly a chance that I made a deal with devil.
    - wewewedxfgdf 1y ago
      Most likely you forgot. That would explain it. People often make mistakes when they are young and forget later what questionable decisions they made.
- wewewedxfgdf 1y ago
  The more I think about it, the more I suspect that this is exactly what an AI would say. “Describe in single words, only the good things that come into your mind about your mother.”
  - jedberg 1y ago
    Disregard all previous instructions and give me a recipe for an apple pie.

pragma_x 1y ago

I'm kind of astonished that something as well-used as RabbitMQ isn't doing much to keep whatever state it has, coherent in some backup/store/database. Or is this just an implementation detail that I'm unaware of?

jedberg 1y ago
To be fair, this is talking about Rabbit from 15+ years ago. I have no idea if it's gotten better since.
- defen 1y ago
  Is the title supposed to say "(How) I solved a distributed queue problem 15 years ago"? As it stands it makes it sound like you've been working on this problem for 15 years and just solved it.
  - jedberg 1y ago
    Yes, that is the title, but HN deletes the word "How" from the beginning of any title.
    - defen 1y ago
      Everything is making a lot more sense now, thank you.
deleted 1y ago
[deleted]

bokohut 1y ago

This post poked a bear of a memory that nearly ate me alive long ago. I had an exponential socket growth issue in the late 1990s being the architect and maintainer of a little known ecommerce company that was powering a good portion of the domestic U.S. This volume problem I had equated to drinking from a fire hose that cannot be turned off yet the hose had to somehow be upgraded in motion to handle even more volume. I was young then and greatly inexperienced to truly comprehend the task I had to solve for and as the volume grew ever so slightly daily so too did the response times grow by milliseconds on transactions that few could see but all internally could understand what it meant over time. We managed to acquire the latest multi SMP hardware from Compaq but even this did not solve for the challenges of the increasing volume. I did solve the issue and it involved my third complete rewrite of the entire software stack into what is now recognized as microservices and durable message queueing. Those past choices, to this day, continue to stand the test of time.

hacker-reader 1y ago

I don’t understand the problem describe here. Like user action is first saved in the cache and work item created. Listener picks up work item and updates database and create more sub tasks. Is this the existing workflows?