3 ms·

I have no doubt that whenever Salvatore releases a distributed, Redis queue feature it will be amazing. But if you're on AWS, SQS is a solid at-least-once deliv

by brryant 12y ago

I have no doubt that whenever Salvatore releases a distributed, Redis queue feature it will be amazing. But if you're on AWS, SQS is a solid at-least-once delivery queue service that is hard to beat. (at least when you don't want to manage your own redis machines)

netcraft 12y ago
What is your fallback though if amazon loses your confidence? AWS infrastructures seem very nice, but aren't you hitching yourself to their wagon in a way that doesnt give you an easy or cheap exit strategy if things go south? I keep hoping one day that they will open source the software that runs everything so that worst case scenario you arent SOL if you want to migrate away. Some might say that would lose customers for them, but it would probably gain people like me.
- boomzilla 12y ago
  Restrict yourself to a set of simple, generic queue API, .e.g. put, getNext, getWithID. Abstract those APIs and implement with them with SQS first. It's fairly easy to switch to a another implementation (Kafka, activemq, etc.) later. Even in production, you can easily switch off the producers, drain the queue, move to the new queue, then restart.
- brryant 12y ago
  If amazon loses my confidence I think I'd start shopping at Walmart before I switch infrastructure platforms... I see AWS as a century old business, just in the same way PG&E and utility companies of the 20th century have developed a stronghold in energy distribution. But if I had to host one myself, http://nsq.io/ http://nsq.io/ is the way to go.
- mdaniel 12y ago
  If you have a lot of tooling targeting AWS, but want some reassurance for the tools (i.e. not speaking to the data, merely the APIs) then Eucalyptus (https://www.eucalyptus.com/ https://www.eucalyptus.com/) which is Open Source (https://github.com/eucalyptus https://github.com/eucalyptus) under the GPLv3 might be a way out of that lock-in. I haven't yet taken the time to provision a local Eucalyptus cluster to test the actual versus the marketing, but it certainly seems like they have a lot of momentum which I wouldn't expect if it didn't do as advertised.
rubiquity 12y ago
SQS is certainly simple but I've had race condition issues with it in the past.
- brryant 12y ago
  Are you deleting messages after you receive them? I haven't run into any race conditions but I suppose if I did, I'd use redis's HyperLogLog or something similar to see if I've already processed the message.
  - iampims 12y ago
    HyperLogLog is for cardinality estimation, not set membership. A bloom filter would be more appropriate I believe.

by brryant 12y ago

netcraft 12y ago

What is your fallback though if amazon loses your confidence? AWS infrastructures seem very nice, but aren't you hitching yourself to their wagon in a way that doesnt give you an easy or cheap exit strategy if things go south? I keep hoping one day that they will open source the software that runs everything so that worst case scenario you arent SOL if you want to migrate away. Some might say that would lose customers for them, but it would probably gain people like me.

boomzilla 12y ago
Restrict yourself to a set of simple, generic queue API, .e.g. put, getNext, getWithID. Abstract those APIs and implement with them with SQS first. It's fairly easy to switch to a another implementation (Kafka, activemq, etc.) later. Even in production, you can easily switch off the producers, drain the queue, move to the new queue, then restart.
brryant 12y ago
If amazon loses my confidence I think I'd start shopping at Walmart before I switch infrastructure platforms... I see AWS as a century old business, just in the same way PG&E and utility companies of the 20th century have developed a stronghold in energy distribution. But if I had to host one myself, http://nsq.io/ http://nsq.io/ is the way to go.
mdaniel 12y ago
If you have a lot of tooling targeting AWS, but want some reassurance for the tools (i.e. not speaking to the data, merely the APIs) then Eucalyptus (https://www.eucalyptus.com/ https://www.eucalyptus.com/) which is Open Source (https://github.com/eucalyptus https://github.com/eucalyptus) under the GPLv3 might be a way out of that lock-in. I haven't yet taken the time to provision a local Eucalyptus cluster to test the actual versus the marketing, but it certainly seems like they have a lot of momentum which I wouldn't expect if it didn't do as advertised.

rubiquity 12y ago

SQS is certainly simple but I've had race condition issues with it in the past.

brryant 12y ago
Are you deleting messages after you receive them? I haven't run into any race conditions but I suppose if I did, I'd use redis's HyperLogLog or something similar to see if I've already processed the message.
- iampims 12y ago
  HyperLogLog is for cardinality estimation, not set membership. A bloom filter would be more appropriate I believe.