4 ms·

Rump: Hot-sync two Redis databases using dumps

34 points by jdorfman 10y ago

joelhaasnoot 10y ago
This is one of those cases where I wonder what the benefits are of Amazon vs running your own instance and just using the commands the way they were intended (by Redis in this case). The Amazon version of Elasticsearch has similar disadvantages with some blocked commands (understandably for security reasons).
- AznHisoka 10y ago
  As someone who runs our own instances, these limitations seem very insane. what if the redis server goes down? That means there's no failover whatsoever? It also seems like such a waste of open source work just to overcome a proprietary, third party limitation.
  - mperham 10y ago
    I believe elasticache has its own internal HA. Your client must disconnect and reconnect if the Redis suddenly becomes read-only (due to a failover). https://github.com/mperham/sidekiq/blob/master/lib/sidekiq.rb#L97 https://github.com/mperham/sidekiq/blob/master/lib/sidekiq.r...
    - AznHisoka 10y ago
      so its manual failover and you cant use redis sentinel?
      - mperham 10y ago
        If the master fails, your app gets a READONLY error due to the failover, you disconnect and reconnect the connection. Their network magic ensures you get the new master.
        toomuchtodo 10y ago
        > Their network magic ensures you get the new master. DNS with a very short TTL.
koolba 10y ago
So this gets a list of keys from the server A via the SCAN command, gets their value via DUMP, and sequentially runs RESTORE on server B to load the value? If so then there's zero transactional consistency between the keys themselves. If each key is independent then that's not really an issue, but if you have multiple keys that make sense only in context of each other, it'd be possible to get values for each that corresponding to different moments in time (each reflecting when the DUMP command was invoked for that particular key). I get that this is meant to work around limitations in AWS's Redis offering (namely the lack of BGSAVE, direct access to RDB, or SLAVEOF) but I'd suggesting highlighting that inconsistency at the top of the docs.
shiftoutbox 10y ago
I am sorry; your catchie name is already being used by another project . Please go make a fancy coffee beverage and try again . https://wiki.netbsd.org/rumpkernel/ https://wiki.netbsd.org/rumpkernel/
gommm 10y ago
Elasticache disabling SLAVEOF is the main reason why I don't use it in production.

34 points by jdorfman 10y ago

This is one of those cases where I wonder what the benefits are of Amazon vs running your own instance and just using the commands the way they were intended (by Redis in this case). The Amazon version of Elasticsearch has similar disadvantages with some blocked commands (understandably for security reasons).

AznHisoka 10y ago
As someone who runs our own instances, these limitations seem very insane. what if the redis server goes down? That means there's no failover whatsoever? It also seems like such a waste of open source work just to overcome a proprietary, third party limitation.
- mperham 10y ago
  I believe elasticache has its own internal HA. Your client must disconnect and reconnect if the Redis suddenly becomes read-only (due to a failover). https://github.com/mperham/sidekiq/blob/master/lib/sidekiq.rb#L97 https://github.com/mperham/sidekiq/blob/master/lib/sidekiq.r...
  - AznHisoka 10y ago
    so its manual failover and you cant use redis sentinel?
    - mperham 10y ago
      If the master fails, your app gets a READONLY error due to the failover, you disconnect and reconnect the connection. Their network magic ensures you get the new master.
      - toomuchtodo 10y ago
        > Their network magic ensures you get the new master. DNS with a very short TTL.

koolba 10y ago

So this gets a list of keys from the server A via the SCAN command, gets their value via DUMP, and sequentially runs RESTORE on server B to load the value? If so then there's zero transactional consistency between the keys themselves. If each key is independent then that's not really an issue, but if you have multiple keys that make sense only in context of each other, it'd be possible to get values for each that corresponding to different moments in time (each reflecting when the DUMP command was invoked for that particular key). I get that this is meant to work around limitations in AWS's Redis offering (namely the lack of BGSAVE, direct access to RDB, or SLAVEOF) but I'd suggesting highlighting that inconsistency at the top of the docs.

shiftoutbox 10y ago

I am sorry; your catchie name is already being used by another project . Please go make a fancy coffee beverage and try again . https://wiki.netbsd.org/rumpkernel/ https://wiki.netbsd.org/rumpkernel/

gommm 10y ago

Elasticache disabling SLAVEOF is the main reason why I don't use it in production.