6 ms·

Queues and databases

197 points by edmccard 12y ago

heterogenic 12y ago
Aaaand, that's one more op system I can look forward to replacing with Redis. Totally off-topic, but I have such a fangirl crush on Salvatore Sanfilippo... Every time he posts I get butterflies.
- stevekemp 12y ago
  He's a wonderful coder, and a great writer. The two together are sadly a rare pairing.
- dbenhur 12y ago
  Are you sure that's a good thing? antirez seems to have a hard time reasoning correctly about distributed systems: http://aphyr.com/posts/283-call-me-maybe-redis http://aphyr.com/posts/283-call-me-maybe-redis http://aphyr.com/posts/307-call-me-maybe-redis-redux http://aphyr.com/posts/307-call-me-maybe-redis-redux
  - antirez 12y ago
    In the first post Redis + asynchronous replication + (an alpha version of) Sentinel were found to lose writes during partitions. This is an obvious result of asynchronous replication + failover, since this is, technically, an eventually consistent system where the merge function is picking a single timeline when there are divergences. Redis was improved since then by adding heuristics to practically lose less writes, but basically this was an expected result that nobody can change. In the second blog post I proposed, just for an argument of a different discussion, a strongly consistent system where basically you have God coordinating the failover, and every write replicated N times. This is something that's not even worth to test, since it is obvious that a system like that is linearizable, however it was still tested, and since it was obviously flawless per definition, the flaw was found, very surprising, in something that violates any CP system, that is, serving reads directly without any agreement. Later Aphyr in this article (http://aphyr.com/posts/316-call-me-maybe-etcd-and-consul http://aphyr.com/posts/316-call-me-maybe-etcd-and-consul) pointed to reading from a possible stale leader as a flaw, while in its implementation in the previous article he setup the things in order to explicitly read from nodes directly. In both the instances I can't see how this translates to me not understanding stuff. That said, distributed systems are all about details, and while I'll try hard to do my best, I can still end with something buggy. I try to improve my knowledge every day and I understand I'm not an expert in the field and that I need to learn, test my designs, make them publish in the hope that other people will analyze them, and so forth. However I believe I'm not the only one that needs to perform this learning process, especially if you want to point at random articles on the web as evidence of my failure, you also need to have a solid understanding of the basic principles.
    - heterogenic 12y ago
      swoon
sirrice 12y ago
Relevant paper: Queues are databases, Jim Gray http://research.microsoft.com/pubs/69641/tr-95-56.pdf http://research.microsoft.com/pubs/69641/tr-95-56.pdf
- emmelaich 12y ago
  Good paper. Likewise, I would say that databases are queues. In fact everything in computing is a queue; disk i/o, cpu tasks, people at their keyboards.
brryant 12y ago
I have no doubt that whenever Salvatore releases a distributed, Redis queue feature it will be amazing. But if you're on AWS, SQS is a solid at-least-once delivery queue service that is hard to beat. (at least when you don't want to manage your own redis machines)
- netcraft 12y ago
  What is your fallback though if amazon loses your confidence? AWS infrastructures seem very nice, but aren't you hitching yourself to their wagon in a way that doesnt give you an easy or cheap exit strategy if things go south? I keep hoping one day that they will open source the software that runs everything so that worst case scenario you arent SOL if you want to migrate away. Some might say that would lose customers for them, but it would probably gain people like me.
  - boomzilla 12y ago
    Restrict yourself to a set of simple, generic queue API, .e.g. put, getNext, getWithID. Abstract those APIs and implement with them with SQS first. It's fairly easy to switch to a another implementation (Kafka, activemq, etc.) later. Even in production, you can easily switch off the producers, drain the queue, move to the new queue, then restart.
  - brryant 12y ago
    If amazon loses my confidence I think I'd start shopping at Walmart before I switch infrastructure platforms... I see AWS as a century old business, just in the same way PG&E and utility companies of the 20th century have developed a stronghold in energy distribution. But if I had to host one myself, http://nsq.io/ http://nsq.io/ is the way to go.
  - mdaniel 12y ago
    If you have a lot of tooling targeting AWS, but want some reassurance for the tools (i.e. not speaking to the data, merely the APIs) then Eucalyptus (https://www.eucalyptus.com/ https://www.eucalyptus.com/) which is Open Source (https://github.com/eucalyptus https://github.com/eucalyptus) under the GPLv3 might be a way out of that lock-in. I haven't yet taken the time to provision a local Eucalyptus cluster to test the actual versus the marketing, but it certainly seems like they have a lot of momentum which I wouldn't expect if it didn't do as advertised.
ryanjshaw 12y ago
Something that is often missing in discussions about queues is a discussion about consistency across queues. This really confuses me and makes me wonder if I'm missing something that everybody else understands. Let's say I'm building a payment system that needs to report settled cash flows to a separate accounting system. In other words, when Alice clicks "Approve" on a payment that Bob has created, the payment system -- in a single database transaction -- records two accounting entries, updates account balances, finalises the payment status, creates audit entries, and must now send the details of the transaction to the company's accounting system (and presumably an external payment network, which is much the same problem, with just more components to fail along the way). If you just put that message onto a brokered queue that feeds the accounting system, there is a risk, however small, that the queue loses the message. In some systems, losing a message may not be an issue; in our particular system, losing an accounting transaction is a serious problem so we must deal with the possibility. I've seen 3 patterns for dealing with this in practice (I'd be very interested if there are others, I can think of a 4th but have not seen it used): 1. a reconciliation process is put in place between the two systems, whereby the payment system delivers a reconciliation file (e.g. daily) and a matching system pairs transactions with a reconciliation file produced by the accounting system; mismatches ("breaks") lead to further investigation 2. the receiver sends an ACK back to the sender; if the sender does not receive an ACK back timeously it may elect to resend, or send an alert to operators, who investigate further 3. distributed transactions: I'm not going to discuss these because they reduce to a single system in analysis of a simple system, and a nightmare in a complex system (Also you can implement #1 and #2 simultaneously to cover each other; often the #1 process conceptually does more work since it ensures double-entry accounting principles are followed in e.g. financial systems) #1 is nice and simple, but the downside is it happens as a batch process typically at end of day; in a payments system this can be a serious problem because e.g. in an inter-bank transfer application it impairs effective cash management on nostro accounts at other banks by attracting finance charges on negative balances overnight. You can speed it up by reducing the batch size and increasing the frequency, and when you go far enough you end up with #2. #2 requires the sender to maintain state about what was put onto the queue and whether it was acknowledged or not. The interesting thing about this is that this state is in effect a copy of the queue's state. Even better, the state here is consistent with the rest of the state in your database. You don't need any ACK protocol or recon so long as you access the database-backed queue transactionally. So: why introduce a 3rd component, a standalone (typically brokered) queue, between process 1 and 2, when you already have everything you need in the shared database? In my mind it only makes sense when there are sound reasons, e.g. for performance reasons, or there is no shared database because the applications belong to different business units, or to decouple applications that are in the same business unit but conceptually independent. Note that the principle idea in OP's article -- creating a time/processing barrier -- is support by database-backed queues. While the email delivery queue example in OP's post is a fair example of something you want to decouple from the rest of the application because it's a very generic service, what if you're extending your payment system to produce PDF receipts that are specialised to this system and incorporate data from the system database? This is a high-CPU requirement that you definitely don't need or want on your critical path but would you really deploy a queue component and a separate database to backup, or waste time decoupling schemas for such an intrinsically tightly coupled feature? What surprises me is how often on sites like HN and StackOverflow the default answer to somebody who wants to know how to offload computation is "use a [brokered] queue". I emphasize "brokered" because the database is very good at implementing queues, but I rarely see a discussion about the reliability of brokered queues and the implications for consistency vs. just using a database-backed queue in your application DB -- often you can't just put things on a queue without making the rest of your application aware of this fact, yet the solution is frequently presented without this caveat, or even an acknowledgement of the alternative. What am I missing?
sacheendra 12y ago
This looks like something that is very useful for low latency applications which don't do a lot of processing. Have you looked at Hazelcast(http://hazelcast.org/ http://hazelcast.org/).
JonnieCache 12y ago
Here's what I'm building my new queue with: https://github.com/jondot/sneakers https://github.com/jondot/sneakers It's a ruby library that implements a Resque-like interface on top of RabbitMQ. Crucially, it doesn't have any opinion on the message contents. It doesn't even deserialize them for you. Most of what it does is manage a unicorn-like process/thread pool plus the associated worker objects, and consumes messages in a thread-safe way. That and the usual configuration boilerplate. Also check out this branch which implements a max-retries strategy using dead letter exchanges, and is kept up to date with the upstream: https://github.com/Yesware/sneakers https://github.com/Yesware/sneakers I imagine it will be merged soon. The thread/process management is done by the serverengine gem: https://github.com/frsyuki/serverengine https://github.com/frsyuki/serverengine
- lobster_johnson 12y ago
  Sneakers is nice, but keep in mind that RabbitMQ has a huge problem with partitions -- which can occur even on flawless networks, but will certainly occur frequently on VPS systems like AWS and Digital Ocean. See [1] for an excellent analysis. Unfortunately, once you have a partition there is no way to merge the isolated nodes back into the cluster; since queues "live" on the node they were initially created on, this could be extremely disruptive. The only way to recover is to discard queue data. I love how RabbitMQ works, but it's terrible in a virtualization environment. Unfortunately there aren't any worthwhile competitors. Kafka comes closest, but it lacks support for topic fanouts, which means every producer has to know about every client, which isn't really acceptable. [1] http://aphyr.com/posts/315-call-me-maybe-rabbitmq http://aphyr.com/posts/315-call-me-maybe-rabbitmq
  - JonnieCache 12y ago
    Fortunately all our jobs are idempotent, and I hope to keep it that way. Postgres keeps track of what's currently "in-flight" so to speak, so my plan for failures is just to re-queue stuff. The good thing about sneakers is we're not locked into any particular routing scheme or message structure so it'll be easy to move away from once we eventually experience problems. So, community, am I being naive? Will the idempotency thing not save me? (Also, lobster_johnson when you say terrible in a virtualisation environment, are you referring to AWS-ish shared VPSs or to a containers-on-colocated-boxes solution, which is what we favor?)
baq 12y ago
see http://aphyr.com/posts/283-call-me-maybe-redis http://aphyr.com/posts/283-call-me-maybe-redis for a discussion of distributed behaviour of redis.
- antirez 12y ago
  Redis + asynchronous replication + failover is a given distributed system (eventual consistency + merge function being picking a single timeline, so it has very limited write safety during certain partitions). When you use N Redis instances as a building block for a distributed system, you don't incur in the limitations of Redis asynchronous replication + failover model. As written in the article the queue system I'm designing uses N independent Redis instances.

197 points by edmccard 12y ago

Aaaand, that's one more op system I can look forward to replacing with Redis. Totally off-topic, but I have such a fangirl crush on Salvatore Sanfilippo... Every time he posts I get butterflies.

stevekemp 12y ago
He's a wonderful coder, and a great writer. The two together are sadly a rare pairing.
dbenhur 12y ago
Are you sure that's a good thing? antirez seems to have a hard time reasoning correctly about distributed systems: http://aphyr.com/posts/283-call-me-maybe-redis http://aphyr.com/posts/283-call-me-maybe-redis http://aphyr.com/posts/307-call-me-maybe-redis-redux http://aphyr.com/posts/307-call-me-maybe-redis-redux
- antirez 12y ago
  In the first post Redis + asynchronous replication + (an alpha version of) Sentinel were found to lose writes during partitions. This is an obvious result of asynchronous replication + failover, since this is, technically, an eventually consistent system where the merge function is picking a single timeline when there are divergences. Redis was improved since then by adding heuristics to practically lose less writes, but basically this was an expected result that nobody can change. In the second blog post I proposed, just for an argument of a different discussion, a strongly consistent system where basically you have God coordinating the failover, and every write replicated N times. This is something that's not even worth to test, since it is obvious that a system like that is linearizable, however it was still tested, and since it was obviously flawless per definition, the flaw was found, very surprising, in something that violates any CP system, that is, serving reads directly without any agreement. Later Aphyr in this article (http://aphyr.com/posts/316-call-me-maybe-etcd-and-consul http://aphyr.com/posts/316-call-me-maybe-etcd-and-consul) pointed to reading from a possible stale leader as a flaw, while in its implementation in the previous article he setup the things in order to explicitly read from nodes directly. In both the instances I can't see how this translates to me not understanding stuff. That said, distributed systems are all about details, and while I'll try hard to do my best, I can still end with something buggy. I try to improve my knowledge every day and I understand I'm not an expert in the field and that I need to learn, test my designs, make them publish in the hope that other people will analyze them, and so forth. However I believe I'm not the only one that needs to perform this learning process, especially if you want to point at random articles on the web as evidence of my failure, you also need to have a solid understanding of the basic principles.
  - heterogenic 12y ago
    swoon

sirrice 12y ago

Relevant paper: Queues are databases, Jim Gray http://research.microsoft.com/pubs/69641/tr-95-56.pdf http://research.microsoft.com/pubs/69641/tr-95-56.pdf

emmelaich 12y ago
Good paper. Likewise, I would say that databases are queues. In fact everything in computing is a queue; disk i/o, cpu tasks, people at their keyboards.

brryant 12y ago

I have no doubt that whenever Salvatore releases a distributed, Redis queue feature it will be amazing. But if you're on AWS, SQS is a solid at-least-once delivery queue service that is hard to beat. (at least when you don't want to manage your own redis machines)

netcraft 12y ago
What is your fallback though if amazon loses your confidence? AWS infrastructures seem very nice, but aren't you hitching yourself to their wagon in a way that doesnt give you an easy or cheap exit strategy if things go south? I keep hoping one day that they will open source the software that runs everything so that worst case scenario you arent SOL if you want to migrate away. Some might say that would lose customers for them, but it would probably gain people like me.
- boomzilla 12y ago
  Restrict yourself to a set of simple, generic queue API, .e.g. put, getNext, getWithID. Abstract those APIs and implement with them with SQS first. It's fairly easy to switch to a another implementation (Kafka, activemq, etc.) later. Even in production, you can easily switch off the producers, drain the queue, move to the new queue, then restart.
- brryant 12y ago
  If amazon loses my confidence I think I'd start shopping at Walmart before I switch infrastructure platforms... I see AWS as a century old business, just in the same way PG&E and utility companies of the 20th century have developed a stronghold in energy distribution. But if I had to host one myself, http://nsq.io/ http://nsq.io/ is the way to go.
- mdaniel 12y ago
  If you have a lot of tooling targeting AWS, but want some reassurance for the tools (i.e. not speaking to the data, merely the APIs) then Eucalyptus (https://www.eucalyptus.com/ https://www.eucalyptus.com/) which is Open Source (https://github.com/eucalyptus https://github.com/eucalyptus) under the GPLv3 might be a way out of that lock-in. I haven't yet taken the time to provision a local Eucalyptus cluster to test the actual versus the marketing, but it certainly seems like they have a lot of momentum which I wouldn't expect if it didn't do as advertised.

ryanjshaw 12y ago

Something that is often missing in discussions about queues is a discussion about consistency across queues. This really confuses me and makes me wonder if I'm missing something that everybody else understands. Let's say I'm building a payment system that needs to report settled cash flows to a separate accounting system. In other words, when Alice clicks "Approve" on a payment that Bob has created, the payment system -- in a single database transaction -- records two accounting entries, updates account balances, finalises the payment status, creates audit entries, and must now send the details of the transaction to the company's accounting system (and presumably an external payment network, which is much the same problem, with just more components to fail along the way). If you just put that message onto a brokered queue that feeds the accounting system, there is a risk, however small, that the queue loses the message. In some systems, losing a message may not be an issue; in our particular system, losing an accounting transaction is a serious problem so we must deal with the possibility. I've seen 3 patterns for dealing with this in practice (I'd be very interested if there are others, I can think of a 4th but have not seen it used): 1. a reconciliation process is put in place between the two systems, whereby the payment system delivers a reconciliation file (e.g. daily) and a matching system pairs transactions with a reconciliation file produced by the accounting system; mismatches ("breaks") lead to further investigation 2. the receiver sends an ACK back to the sender; if the sender does not receive an ACK back timeously it may elect to resend, or send an alert to operators, who investigate further 3. distributed transactions: I'm not going to discuss these because they reduce to a single system in analysis of a simple system, and a nightmare in a complex system (Also you can implement #1 and #2 simultaneously to cover each other; often the #1 process conceptually does more work since it ensures double-entry accounting principles are followed in e.g. financial systems) #1 is nice and simple, but the downside is it happens as a batch process typically at end of day; in a payments system this can be a serious problem because e.g. in an inter-bank transfer application it impairs effective cash management on nostro accounts at other banks by attracting finance charges on negative balances overnight. You can speed it up by reducing the batch size and increasing the frequency, and when you go far enough you end up with #2. #2 requires the sender to maintain state about what was put onto the queue and whether it was acknowledged or not. The interesting thing about this is that this state is in effect a copy of the queue's state. Even better, the state here is consistent with the rest of the state in your database. You don't need any ACK protocol or recon so long as you access the database-backed queue transactionally. So: why introduce a 3rd component, a standalone (typically brokered) queue, between process 1 and 2, when you already have everything you need in the shared database? In my mind it only makes sense when there are sound reasons, e.g. for performance reasons, or there is no shared database because the applications belong to different business units, or to decouple applications that are in the same business unit but conceptually independent. Note that the principle idea in OP's article -- creating a time/processing barrier -- is support by database-backed queues. While the email delivery queue example in OP's post is a fair example of something you want to decouple from the rest of the application because it's a very generic service, what if you're extending your payment system to produce PDF receipts that are specialised to this system and incorporate data from the system database? This is a high-CPU requirement that you definitely don't need or want on your critical path but would you really deploy a queue component and a separate database to backup, or waste time decoupling schemas for such an intrinsically tightly coupled feature? What surprises me is how often on sites like HN and StackOverflow the default answer to somebody who wants to know how to offload computation is "use a [brokered] queue". I emphasize "brokered" because the database is very good at implementing queues, but I rarely see a discussion about the reliability of brokered queues and the implications for consistency vs. just using a database-backed queue in your application DB -- often you can't just put things on a queue without making the rest of your application aware of this fact, yet the solution is frequently presented without this caveat, or even an acknowledgement of the alternative. What am I missing?

sacheendra 12y ago

This looks like something that is very useful for low latency applications which don't do a lot of processing. Have you looked at Hazelcast(http://hazelcast.org/ http://hazelcast.org/).

JonnieCache 12y ago

Here's what I'm building my new queue with: https://github.com/jondot/sneakers https://github.com/jondot/sneakers It's a ruby library that implements a Resque-like interface on top of RabbitMQ. Crucially, it doesn't have any opinion on the message contents. It doesn't even deserialize them for you. Most of what it does is manage a unicorn-like process/thread pool plus the associated worker objects, and consumes messages in a thread-safe way. That and the usual configuration boilerplate. Also check out this branch which implements a max-retries strategy using dead letter exchanges, and is kept up to date with the upstream: https://github.com/Yesware/sneakers https://github.com/Yesware/sneakers I imagine it will be merged soon. The thread/process management is done by the serverengine gem: https://github.com/frsyuki/serverengine https://github.com/frsyuki/serverengine

lobster_johnson 12y ago
Sneakers is nice, but keep in mind that RabbitMQ has a huge problem with partitions -- which can occur even on flawless networks, but will certainly occur frequently on VPS systems like AWS and Digital Ocean. See [1] for an excellent analysis. Unfortunately, once you have a partition there is no way to merge the isolated nodes back into the cluster; since queues "live" on the node they were initially created on, this could be extremely disruptive. The only way to recover is to discard queue data. I love how RabbitMQ works, but it's terrible in a virtualization environment. Unfortunately there aren't any worthwhile competitors. Kafka comes closest, but it lacks support for topic fanouts, which means every producer has to know about every client, which isn't really acceptable. [1] http://aphyr.com/posts/315-call-me-maybe-rabbitmq http://aphyr.com/posts/315-call-me-maybe-rabbitmq
- JonnieCache 12y ago
  Fortunately all our jobs are idempotent, and I hope to keep it that way. Postgres keeps track of what's currently "in-flight" so to speak, so my plan for failures is just to re-queue stuff. The good thing about sneakers is we're not locked into any particular routing scheme or message structure so it'll be easy to move away from once we eventually experience problems. So, community, am I being naive? Will the idempotency thing not save me? (Also, lobster_johnson when you say terrible in a virtualisation environment, are you referring to AWS-ish shared VPSs or to a containers-on-colocated-boxes solution, which is what we favor?)

baq 12y ago

see http://aphyr.com/posts/283-call-me-maybe-redis http://aphyr.com/posts/283-call-me-maybe-redis for a discussion of distributed behaviour of redis.

antirez 12y ago
Redis + asynchronous replication + failover is a given distributed system (eventual consistency + merge function being picking a single timeline, so it has very limited write safety during certain partitions). When you use N Redis instances as a building block for a distributed system, you don't incur in the limitations of Redis asynchronous replication + failover model. As written in the article the queue system I'm designing uses N independent Redis instances.