4 ms·

Hot take: I don't think that people should even use queues any more. It's like using raw HTTP2, or managing threads. Use a workflow engine which abstracts qu

by tonyhb 3y ago

Hot take: I don't think that people should even use queues any more. It's like using raw HTTP2, or managing threads. Use a workflow engine which abstracts queues for you so you can "just write code" without managing jobs/schedules/state/retries. https://www.inngest.com/blog/how-durable-workflow-engines-work https://www.inngest.com/blog/how-durable-workflow-engines-wo... (disclaimer: I'm biased as I'm an author of a workflow engine)

xwowsersx 3y ago
In the example in your post, the queue is abstracted away in some sense, but you aren't just "writing code". There's step.run(), step.sleep(), step.sleepUntil() sprinkled throughout. I'd say in something like Celery, which is explicitly about running jobs asynchronously backed by a message broker/queue, you really do just write code, but then call the function with .s().apply_async() and so on. Now, I'm not saying you're wrong that we ought to abstract queues away, just pointing out that the abstraction leaks in your workflow example. Happy to hear where/how I'm wrong on this.
- tonyhb 3y ago
  It leaks in that the entire function becomes declarative, and state is colocated to a single function. You don't have multiple jobs for each attempt, passing indexes into each job queue. You don't have to worry about enqueueing each "step" as a separate job. The code becomes "wait until this time", vs "enqueue this function with this state to run at this time via this message broker, which may enqueue other jobs in ways you can't see". There's no real silver bullet that will let you run code in the future without specifying "when" that code should run — but workflow engines are by far the more productive of the two.
- mfateev 3y ago
  Check out temporal.io that fully abstract this. Disclaimer, I'm one of the founders.
  - xwowsersx 3y ago
    hah, I was thinking about temporal as I was writing this. I have played with temporal pretty extensively.

by tonyhb 3y ago

xwowsersx 3y ago

In the example in your post, the queue is abstracted away in some sense, but you aren't just "writing code". There's step.run(), step.sleep(), step.sleepUntil() sprinkled throughout. I'd say in something like Celery, which is explicitly about running jobs asynchronously backed by a message broker/queue, you really do just write code, but then call the function with .s().apply_async() and so on. Now, I'm not saying you're wrong that we ought to abstract queues away, just pointing out that the abstraction leaks in your workflow example. Happy to hear where/how I'm wrong on this.

tonyhb 3y ago
It leaks in that the entire function becomes declarative, and state is colocated to a single function. You don't have multiple jobs for each attempt, passing indexes into each job queue. You don't have to worry about enqueueing each "step" as a separate job. The code becomes "wait until this time", vs "enqueue this function with this state to run at this time via this message broker, which may enqueue other jobs in ways you can't see". There's no real silver bullet that will let you run code in the future without specifying "when" that code should run — but workflow engines are by far the more productive of the two.
mfateev 3y ago
Check out temporal.io that fully abstract this. Disclaimer, I'm one of the founders.
- xwowsersx 3y ago
  hah, I was thinking about temporal as I was writing this. I have played with temporal pretty extensively.