9 ms·

Disque 1.0 RC1 is out

275 points by antirez 11y ago

yowmamasita 11y ago
Q: Can I use this on top of any non-ACID-compliant database to avoid race conditions?
- joneholland 11y ago
  No.
  - vhost- 11y ago
    Why?
    - joneholland 11y ago
      It's a AP system. Race conditions come with the territory.
    - ykl 11y ago
      See the CAP theorem [0]. Disque is designed to guarantee availability and partition tolerance, which according to CAP means Disque cannot guarantee atomic consistency (the C in CAP is not quite the same as either the A and C in ACID [1]) between nodes. No atomic consistency means no guarantee of a single action to single response correspondence, which means race conditions are possible. [0] https://en.wikipedia.org/wiki/CAP_theorem https://en.wikipedia.org/wiki/CAP_theorem [1] https://en.wikipedia.org/wiki/Consistency_(database_systems) https://en.wikipedia.org/wiki/Consistency_(database_systems)
lobster_johnson 11y ago
Disque is definitely exciting, and looks like it can replace RabbitMQ, which has serious flaws in its clustering design. I'm looking forward to trying it out. However, if some constructive criticism is permitted, I have to say that, having written distributed applications for many years, I have come to dislike the "classical" push/pop queue data model: * Acking is a bad idea. It requires the broker to manage a lot of state, including locking and timeouts. * Re-queuing invalidates total ordering. * On the performance side, parallel distributed queue consumption (which also breaks total ordering) is directly at odds with this model. * Queues as opaque objects — you can only inspect by popping the top message, and you cannot access older, dequeued messages. Fortunately, Disque allows you to read the entire queue without mutating it, but it doesn't look like you can read old messages. * Complicated queue topologies (fanouts, dead letter queues, etc.) become a logical necessity of the strict FIFO structure. (These topologies need to be declared every time the client starts up, and introduces the possibility of schema conflicts.) * Logical de-duping is probably not possible. Apache Kafka gets the data model right. It wisely acknowledges that queues are linear and should stay that way: In Kafka, queues are strictly append-only logs where every consumer has a cursor to the last position it read. In this model, many of the classical concerns melt away: Acks/nacks are unnecessary (consumers simply "commit" their position); total ordering is always preserved (since the queue cannot be reordered) and parallelism is made explicit (through named partitions); de-duping is trivial, and complicated topologies are largely eliminated (AMQP-type "exchanges" that fan out to separate queues are unnecessary because multiple readers can all consume the same queue without changing it, as their position is independent of the queue); and you get to choose either at-most-once or at-least-once delivery consistency by how carefully you manage your offset. Since logs are strictly linear, you are also given the choice of how much history to keep — all of it, if you want — which opens up some interesting use cases that are not possible with classical brokers. Kafka isn't perfect. It's a huge pain if you're not in Java land. There are no modern, mature "high-level" client implementations for Go, Ruby or Node.js. Its reliance on the JVM and on ZooKeeper makes it fairly heavyweight, both on the server and on the consumer side (the new API for broker-stored offsets simplifies things, but non-Java clients are far behind). I would really love to have a lighter-weight, language-agnostic Kafka-like implementation without the Java baggage. Last point: The fact that Disque calls its messages "jobs" makes me a little disappointed that it is, in fact, not a job management system. I'd love a solid, distributed job manager.
- Jasper_ 11y ago
  Kafka has a different data model which works for some scenarios, but not others. We attempted to use Kafka as part of a job management system, where long-running jobs were scheduled and workers consumed partitions, but what we found was that since consumers work on partitions, a long-running task could block an entire partition's worth of work, with no way to migrate it to another partition. Kafka works really well when the bottleneck is the broker to begin with -- if you have a lot of small, lightweight messages being passed to other systems, and the broker is having trouble keeping up. Analytics and logs are great examples where Kafka's data model works really well. But for a smaller number of larger messages that takes seconds to minutes to process (we were using it to schedule data downloads and video processing), workers could be sitting idle while there was still plenty more work to do. We moved to Amazon SQS for now, but I imagine we'll move to something like Disque in the future.
  - lobster_johnson 11y ago
    I would argue that job management is unrelated to messaging, at least according my loose definition of a task that: * Has a well-defined lifetime — unstarted, running, paused, completed successfully, or failed; * Is executed from some kind of parameterized "job specification" that describes its inputs and desired behavior; * Has state data (e.g., completion % progress, log output, metrics, transactional continutation state for pausing/retrying); * Can be scheduled on multiple nodes; * Can be created and scheduled multiple times; * Can be scheduled by priority; * Can have its resources (RAM, CPU, I/O) constrained. The thing is, queues are terrible at this. People continue to abuse messaging systems as a scheduler, and it always sucks. For example, consider acking: For a long-running tasks that might take hours, do you really want to hold up the broker's internal lock for all that time? What if you restart the broker — the task is still running, now you decoupled its running state from the queue. Similarly, queues are terrible at finding out what's waiting, or what has happened. By using the queue as job state, you are coupling scheduling with execution, which is just wrong. A queue (one that supports priorities, mind you) is, however, excellent for management commands: "Start job X", "pause job X" and so on. But you will want to maintain execution state separately from the queue. A relational database is quite good at this, though one could use a NoSQL databases, too. First, create a row representing the job: {id="job1", state="new", type="import_stuff", params="..."}. Then push a queue message {commmand="start_job", job_id="job1"}. Run a queue consumer or ten on each node. Each consumer acks the message, finds the job in the table and starts it. One could run it as a child process, or in a Docker container. One could of course use an orchestration service (like Mesos/Marathon) to run it, and monitor the service for container events and update the job table appropriately. I would want to take it further and allow the job to update itself through API, so that it could publish metrics and log messages, as well as maintain whatever data is useful snapshotting. For example, a long-running task to import data into a database would benefit from maintaining a cursor so that you could pause/kill the job and later resume it at the point where it was.
vdfs 11y ago
Is there a product-ready alternative to Disque?
- bastawhiz 11y ago
  There are plenty, but few/none that solve the problem Disque solves. You could easily use Kafka or RabbitMQ, but Disque aims to provide simplicity rather than be full-featured. @antirez described it as solving the case where you just need a queue with very basic guarantees and good performance without all the bells and whistles that a more complex system has. Disque is (spiritually, not technically) a lot like Mongo was five years ago compared to something much heavier like Postgres. In that sense, there are few (if any) product-ready alternatives that solve the same problem. It should also be noted that you could probably successfully use Redis for much of what Disque does (Disque is based on Redis). Disque adds nice features, though, and removes most of the features of Redis that are outside its scope.
- BradRuderman 11y ago
  It seems like Disque is a "simplified" Kafka, or a more vertically purposed Redis. It seems that it differs from RabbitMQ significantly because rabbit requires a queue to push the messages in, while Disque allows jobs to be pushed independently of consumers being setup.
  - lobster_johnson 11y ago
    Kafka is arguably simpler, at least in terms of data model, than Disque. Disque needs to handle acking and queue mutation and things like lock timeouts and dead lettering, all of which complicates its queue structure and API. Kafka queues (partitions, to be precise) are append-only, which simplifies a lot of things.
- AznHisoka 11y ago
  I've been using Sidekiq for the past 2 years, and deem it as production ready. It does force you to use Redis and Ruby though.
  - 11y ago
willcodeforfoo 11y ago
Neat, love reading antirez's posts and the README for this is amazing.
Dowwie 11y ago
This is exciting. Disque is intended to replace redis in celery (python). I'm looking forward to using it!
kevan 11y ago
> However I’m not living into the illusion that I got everything right in the first release, so it will take months (or years?) of iteration to really reach the operational simplicity I’m targeting. It's always refreshing to hear such good programmers acknowledging how hard building complex systems is.
- BogusIKnow 11y ago
  And this from antirez who writes such excellent programms, Redis runs in several companies I've been, without ever crashing, making any trouble - most people I've met forget they have Redis running because it just works and works and works. Can't praise that piece of code high enough.
  - Mahn 11y ago
    Indeed, we have had redis running in production for four years now and I can't remember a single crash happening. If something goes wrong in our stack redis is the last thing we check. Antirez should write a http web server :-)
    - BogusIKnow 11y ago
      +100 'Antirez should write a http web server'
- nikolay 11y ago
  ... and this is a real professional talk. I'm tired of newbies being baselessly overconfident...
bipin_nag 11y ago
Looking forward to Aphyr's Call Me Maybe for Disque.
- deanclatworthy 11y ago
  This was my first thought also. aphyr seems to be the go to for finding flaws in everyone's distributed systems.
  - jbaiter 11y ago
    If you look at the last paragraph, it seems aphyr has been involved in the development: > I was not alone during the past months, while hacking with Disque and trying to figure out how to shape it, I received the help of: [...] Kyle Kingsbury, [...] Redis Labs and Pivotal, and probably more people I’m not remembering right now. Thank you for your help.
- qopp 11y ago
  Someone else could also do the analysis, Jepsen is open source
  - antirez 11y ago
    Good hint. I tried to run the initial tests Kyle (Aphyr) wrote for Disque in order to debug a problem, and after running Jepsen a few times you realize how cool it is, and you want to run your own tests. However Jepsen is a non trivial system and requires some Clojure skill, so I postponed this activity for later... resorting to simpler systems to test Disque for now. But soon or later I want to spend a few weeks at learning Jepsen and enough Clojure in order to do my tests. However note that the value of having Kyle testing Disque is not just his ability to use Jepsen well (of course) but also the design of the tests so in order to stress the right things to find bugs. And finally the whole analysis that is performed together with the test. So regardless of the fact I'll learn Jepsen or not I hope Disque will get some official testing with Jepsen. However I understand Kyle is a single person and there is a huge queue of systems to be tested.
leeoniya 11y ago
good question in the blog's comments: How is this compared to 0mq or nanomsg?
- lobster_johnson 11y ago
  ZeroMQ is not a messaging broker, it's a socket abstraction. (I interpret the zero in ZeroMQ to mean "no".) So they are not comparable at all. You can build a broker with ZeroMQ, which people have done [1], in the same sense that you can build a broker on plain TCP sockets. I believe the same applies to Nanomsg. [1] https://github.com/zeromq/malamute https://github.com/zeromq/malamute, https://github.com/sintaxi/waterfront https://github.com/sintaxi/waterfront, https://github.com/zeromq/zbroker https://github.com/zeromq/zbroker
JSno 11y ago
How does this (Disque) compare to Kafka? What are the differences and same?
deleted 11y ago
[deleted]
navaneethkn 11y ago
This looks interesting! Currently, we have a huge background processing system built using Sidekiq backed by Redis. The biggest problem we are facing today is to run machines with lot of in-memory storage because jobs are queued to Redis and Redis works in-memory. These machines costs a lot. I think the same issue will happen even here because this is also in-memory store and persistence works like how Redis handles it.
ksec 11y ago
Idle observation, The blog shows what simple, text, and minimal graphics / pics ( In this case there are Zero ) can load incredibly fast...

275 points by antirez 11y ago

yowmamasita 11y ago

Q: Can I use this on top of any non-ACID-compliant database to avoid race conditions?

joneholland 11y ago
No.
- vhost- 11y ago
  Why?
  - joneholland 11y ago
    It's a AP system. Race conditions come with the territory.
  - ykl 11y ago
    See the CAP theorem [0]. Disque is designed to guarantee availability and partition tolerance, which according to CAP means Disque cannot guarantee atomic consistency (the C in CAP is not quite the same as either the A and C in ACID [1]) between nodes. No atomic consistency means no guarantee of a single action to single response correspondence, which means race conditions are possible. [0] https://en.wikipedia.org/wiki/CAP_theorem https://en.wikipedia.org/wiki/CAP_theorem [1] https://en.wikipedia.org/wiki/Consistency_(database_systems) https://en.wikipedia.org/wiki/Consistency_(database_systems)

lobster_johnson 11y ago

Disque is definitely exciting, and looks like it can replace RabbitMQ, which has serious flaws in its clustering design. I'm looking forward to trying it out. However, if some constructive criticism is permitted, I have to say that, having written distributed applications for many years, I have come to dislike the "classical" push/pop queue data model: * Acking is a bad idea. It requires the broker to manage a lot of state, including locking and timeouts. * Re-queuing invalidates total ordering. * On the performance side, parallel distributed queue consumption (which also breaks total ordering) is directly at odds with this model. * Queues as opaque objects — you can only inspect by popping the top message, and you cannot access older, dequeued messages. Fortunately, Disque allows you to read the entire queue without mutating it, but it doesn't look like you can read old messages. * Complicated queue topologies (fanouts, dead letter queues, etc.) become a logical necessity of the strict FIFO structure. (These topologies need to be declared every time the client starts up, and introduces the possibility of schema conflicts.) * Logical de-duping is probably not possible. Apache Kafka gets the data model right. It wisely acknowledges that queues are linear and should stay that way: In Kafka, queues are strictly append-only logs where every consumer has a cursor to the last position it read. In this model, many of the classical concerns melt away: Acks/nacks are unnecessary (consumers simply "commit" their position); total ordering is always preserved (since the queue cannot be reordered) and parallelism is made explicit (through named partitions); de-duping is trivial, and complicated topologies are largely eliminated (AMQP-type "exchanges" that fan out to separate queues are unnecessary because multiple readers can all consume the same queue without changing it, as their position is independent of the queue); and you get to choose either at-most-once or at-least-once delivery consistency by how carefully you manage your offset. Since logs are strictly linear, you are also given the choice of how much history to keep — all of it, if you want — which opens up some interesting use cases that are not possible with classical brokers. Kafka isn't perfect. It's a huge pain if you're not in Java land. There are no modern, mature "high-level" client implementations for Go, Ruby or Node.js. Its reliance on the JVM and on ZooKeeper makes it fairly heavyweight, both on the server and on the consumer side (the new API for broker-stored offsets simplifies things, but non-Java clients are far behind). I would really love to have a lighter-weight, language-agnostic Kafka-like implementation without the Java baggage. Last point: The fact that Disque calls its messages "jobs" makes me a little disappointed that it is, in fact, not a job management system. I'd love a solid, distributed job manager.

Jasper_ 11y ago
Kafka has a different data model which works for some scenarios, but not others. We attempted to use Kafka as part of a job management system, where long-running jobs were scheduled and workers consumed partitions, but what we found was that since consumers work on partitions, a long-running task could block an entire partition's worth of work, with no way to migrate it to another partition. Kafka works really well when the bottleneck is the broker to begin with -- if you have a lot of small, lightweight messages being passed to other systems, and the broker is having trouble keeping up. Analytics and logs are great examples where Kafka's data model works really well. But for a smaller number of larger messages that takes seconds to minutes to process (we were using it to schedule data downloads and video processing), workers could be sitting idle while there was still plenty more work to do. We moved to Amazon SQS for now, but I imagine we'll move to something like Disque in the future.
- lobster_johnson 11y ago
  I would argue that job management is unrelated to messaging, at least according my loose definition of a task that: * Has a well-defined lifetime — unstarted, running, paused, completed successfully, or failed; * Is executed from some kind of parameterized "job specification" that describes its inputs and desired behavior; * Has state data (e.g., completion % progress, log output, metrics, transactional continutation state for pausing/retrying); * Can be scheduled on multiple nodes; * Can be created and scheduled multiple times; * Can be scheduled by priority; * Can have its resources (RAM, CPU, I/O) constrained. The thing is, queues are terrible at this. People continue to abuse messaging systems as a scheduler, and it always sucks. For example, consider acking: For a long-running tasks that might take hours, do you really want to hold up the broker's internal lock for all that time? What if you restart the broker — the task is still running, now you decoupled its running state from the queue. Similarly, queues are terrible at finding out what's waiting, or what has happened. By using the queue as job state, you are coupling scheduling with execution, which is just wrong. A queue (one that supports priorities, mind you) is, however, excellent for management commands: "Start job X", "pause job X" and so on. But you will want to maintain execution state separately from the queue. A relational database is quite good at this, though one could use a NoSQL databases, too. First, create a row representing the job: {id="job1", state="new", type="import_stuff", params="..."}. Then push a queue message {commmand="start_job", job_id="job1"}. Run a queue consumer or ten on each node. Each consumer acks the message, finds the job in the table and starts it. One could run it as a child process, or in a Docker container. One could of course use an orchestration service (like Mesos/Marathon) to run it, and monitor the service for container events and update the job table appropriately. I would want to take it further and allow the job to update itself through API, so that it could publish metrics and log messages, as well as maintain whatever data is useful snapshotting. For example, a long-running task to import data into a database would benefit from maintaining a cursor so that you could pause/kill the job and later resume it at the point where it was.

vdfs 11y ago

Is there a product-ready alternative to Disque?

bastawhiz 11y ago
There are plenty, but few/none that solve the problem Disque solves. You could easily use Kafka or RabbitMQ, but Disque aims to provide simplicity rather than be full-featured. @antirez described it as solving the case where you just need a queue with very basic guarantees and good performance without all the bells and whistles that a more complex system has. Disque is (spiritually, not technically) a lot like Mongo was five years ago compared to something much heavier like Postgres. In that sense, there are few (if any) product-ready alternatives that solve the same problem. It should also be noted that you could probably successfully use Redis for much of what Disque does (Disque is based on Redis). Disque adds nice features, though, and removes most of the features of Redis that are outside its scope.
BradRuderman 11y ago
It seems like Disque is a "simplified" Kafka, or a more vertically purposed Redis. It seems that it differs from RabbitMQ significantly because rabbit requires a queue to push the messages in, while Disque allows jobs to be pushed independently of consumers being setup.
- lobster_johnson 11y ago
  Kafka is arguably simpler, at least in terms of data model, than Disque. Disque needs to handle acking and queue mutation and things like lock timeouts and dead lettering, all of which complicates its queue structure and API. Kafka queues (partitions, to be precise) are append-only, which simplifies a lot of things.
AznHisoka 11y ago
I've been using Sidekiq for the past 2 years, and deem it as production ready. It does force you to use Redis and Ruby though.
- 11y ago

willcodeforfoo 11y ago

Neat, love reading antirez's posts and the README for this is amazing.

Dowwie 11y ago

This is exciting. Disque is intended to replace redis in celery (python). I'm looking forward to using it!

kevan 11y ago

> However I’m not living into the illusion that I got everything right in the first release, so it will take months (or years?) of iteration to really reach the operational simplicity I’m targeting. It's always refreshing to hear such good programmers acknowledging how hard building complex systems is.

BogusIKnow 11y ago
And this from antirez who writes such excellent programms, Redis runs in several companies I've been, without ever crashing, making any trouble - most people I've met forget they have Redis running because it just works and works and works. Can't praise that piece of code high enough.
- Mahn 11y ago
  Indeed, we have had redis running in production for four years now and I can't remember a single crash happening. If something goes wrong in our stack redis is the last thing we check. Antirez should write a http web server :-)
  - BogusIKnow 11y ago
    +100 'Antirez should write a http web server'
nikolay 11y ago
... and this is a real professional talk. I'm tired of newbies being baselessly overconfident...

bipin_nag 11y ago

Looking forward to Aphyr's Call Me Maybe for Disque.

deanclatworthy 11y ago
This was my first thought also. aphyr seems to be the go to for finding flaws in everyone's distributed systems.
- jbaiter 11y ago
  If you look at the last paragraph, it seems aphyr has been involved in the development: > I was not alone during the past months, while hacking with Disque and trying to figure out how to shape it, I received the help of: [...] Kyle Kingsbury, [...] Redis Labs and Pivotal, and probably more people I’m not remembering right now. Thank you for your help.
qopp 11y ago
Someone else could also do the analysis, Jepsen is open source
- antirez 11y ago
  Good hint. I tried to run the initial tests Kyle (Aphyr) wrote for Disque in order to debug a problem, and after running Jepsen a few times you realize how cool it is, and you want to run your own tests. However Jepsen is a non trivial system and requires some Clojure skill, so I postponed this activity for later... resorting to simpler systems to test Disque for now. But soon or later I want to spend a few weeks at learning Jepsen and enough Clojure in order to do my tests. However note that the value of having Kyle testing Disque is not just his ability to use Jepsen well (of course) but also the design of the tests so in order to stress the right things to find bugs. And finally the whole analysis that is performed together with the test. So regardless of the fact I'll learn Jepsen or not I hope Disque will get some official testing with Jepsen. However I understand Kyle is a single person and there is a huge queue of systems to be tested.

leeoniya 11y ago

good question in the blog's comments: How is this compared to 0mq or nanomsg?

lobster_johnson 11y ago
ZeroMQ is not a messaging broker, it's a socket abstraction. (I interpret the zero in ZeroMQ to mean "no".) So they are not comparable at all. You can build a broker with ZeroMQ, which people have done [1], in the same sense that you can build a broker on plain TCP sockets. I believe the same applies to Nanomsg. [1] https://github.com/zeromq/malamute https://github.com/zeromq/malamute, https://github.com/sintaxi/waterfront https://github.com/sintaxi/waterfront, https://github.com/zeromq/zbroker https://github.com/zeromq/zbroker

JSno 11y ago

How does this (Disque) compare to Kafka? What are the differences and same?

deleted 11y ago

[deleted]

navaneethkn 11y ago

This looks interesting! Currently, we have a huge background processing system built using Sidekiq backed by Redis. The biggest problem we are facing today is to run machines with lot of in-memory storage because jobs are queued to Redis and Redis works in-memory. These machines costs a lot. I think the same issue will happen even here because this is also in-memory store and persistence works like how Redis handles it.

ksec 11y ago

Idle observation, The blog shows what simple, text, and minimal graphics / pics ( In this case there are Zero ) can load incredibly fast...