4 ms·

Outstanding comment. I'm working at a distributed queue and those are exactly the set of questions I ask myself when building different parts of the system, but

by antirez 12y ago

Outstanding comment. I'm working at a distributed queue and those are exactly the set of questions I ask myself when building different parts of the system, but your exposition is something I'm going to cut&paste and read every time I'm going to work to my project. Btw, we both work at Pivotal :-) Just a question: "Is queueing partially/totally consistent across a group of servers or divided up for maximal throughput". I'm not sure I understand this fully, could you please extend that a bit with some more context? Thanks.

eternalban 12y ago
Total consistency requires global consensus and that puts a definitive break on performance.
- antirez 12y ago
  Yep, but are we talking about consistency in the order of messages? That was not clear to me.
  - jacques_chester 12y ago
    I principally was thinking of order, yes. When we application developers hear "queue" we typically assume it's going to have a FIFO discipline.
jacques_chester 12y ago
> Btw, we both work at Pivotal :-) I'm at Labs in NYC on secondment to Cloud Foundry. You should come see us and do a tech talk! > "Is queueing partially/totally consistent across a group of servers or divided up for maximal throughput" I didn't do a good job of explaining this. Basically, assuming the most popular queueing discipline -- FIFO -- you can either set up brokers to be highly available, or to scale approximately linearly, but not both. This is because HA + FIFO + at-least-once queueing requires servers to coordinate the state of a queue and to either write to disk or replicate messages. It's really, really hard and it can ruin your day when there's a partition. If you relax all the guarantees that make life easier for application developers, you can just send any old message to any old server. No server needs to coordinate with any other and so scaling is closer to linear. Amazon's SQS is a really good case study in what queues with looser guarantees can achieve.
- antirez 12y ago
  I love NYC! That's tempting :-) Thanks a lot for further specifying the cross-consistency guarantee. Basically the system I'm developing is a lot more like Amazon's SQS, it gives guarantees about message durability / replication, but only provides weak ordering, for three reasons: 1) What you said, scalability. 2) Availability since the system is available into the minority partition as well. 3) My queue is fundamentally designed with the idea that messages must be acknowledged by consumers, or they are reissued after a job-specific retry time (that can be set to 0 if you really want at-most-once delivery, but it's a rare requirement). When you have auto-re-queue strict ordering is basically useless since it is violated every time a consumer does not acknowledge the message fast enough. However not strict ordering does not mean random, so it tries to approximate a FIFO using each node's wall clock timestamps. This means that while messages may be delivered in random order, usually what is queued first is served first, which is no guarantee at all from the POV of the developer, but is a more general guarantee about the fact that if there are N users waiting into a web application for some thing to get processed, the first in queue will likely be served before the last. p.s. sorry for errors, writing with my daughter pulling my arms :-)

by antirez 12y ago

eternalban 12y ago

Total consistency requires global consensus and that puts a definitive break on performance.

antirez 12y ago
Yep, but are we talking about consistency in the order of messages? That was not clear to me.
- jacques_chester 12y ago
  I principally was thinking of order, yes. When we application developers hear "queue" we typically assume it's going to have a FIFO discipline.

jacques_chester 12y ago

> Btw, we both work at Pivotal :-) I'm at Labs in NYC on secondment to Cloud Foundry. You should come see us and do a tech talk! > "Is queueing partially/totally consistent across a group of servers or divided up for maximal throughput" I didn't do a good job of explaining this. Basically, assuming the most popular queueing discipline -- FIFO -- you can either set up brokers to be highly available, or to scale approximately linearly, but not both. This is because HA + FIFO + at-least-once queueing requires servers to coordinate the state of a queue and to either write to disk or replicate messages. It's really, really hard and it can ruin your day when there's a partition. If you relax all the guarantees that make life easier for application developers, you can just send any old message to any old server. No server needs to coordinate with any other and so scaling is closer to linear. Amazon's SQS is a really good case study in what queues with looser guarantees can achieve.

antirez 12y ago
I love NYC! That's tempting :-) Thanks a lot for further specifying the cross-consistency guarantee. Basically the system I'm developing is a lot more like Amazon's SQS, it gives guarantees about message durability / replication, but only provides weak ordering, for three reasons: 1) What you said, scalability. 2) Availability since the system is available into the minority partition as well. 3) My queue is fundamentally designed with the idea that messages must be acknowledged by consumers, or they are reissued after a job-specific retry time (that can be set to 0 if you really want at-most-once delivery, but it's a rare requirement). When you have auto-re-queue strict ordering is basically useless since it is violated every time a consumer does not acknowledge the message fast enough. However not strict ordering does not mean random, so it tries to approximate a FIFO using each node's wall clock timestamps. This means that while messages may be delivered in random order, usually what is queued first is served first, which is no guarantee at all from the POV of the developer, but is a more general guarantee about the fact that if there are N users waiting into a web application for some thing to get processed, the first in queue will likely be served before the last. p.s. sorry for errors, writing with my daughter pulling my arms :-)