4 ms·

We had a saying at my old job: if something’s broken it’s never Redis. Redis is such a tank in my experience. We set it up. Secured it. And then forgot about it

We had a saying at my old job: if something’s broken it’s never Redis. Redis is such a tank in my experience. We set it up. Secured it. And then forgot about it.

junon 6y ago
Absolutely. Of course I plan for what happens if Redis were to fail, but I don't remember a single time it ever did. It really, truly is a tank. Antirez, if you're reading this (I know you'll eventually find your way here), thanks for making my job a little easier over the years <3
rshaw1 6y ago
I thought the same up until yesterday! Yesterday the replica randomly disconnected from the master and could no longer reconnect, the resyncs were failing because a replication buffer on the master was being hit (https://redislabs.com/blog/top-redis-headaches-for-devops-replication-buffer/ https://redislabs.com/blog/top-redis-headaches-for-devops-re...). Once that buffer was increased the replica was able to sync the snapshot from the master but for some reason was taking a very long time to load the snapshot, during this time the sentinels thought the replica was healthy again and started allowing the application to read from them, of course Redis responded with an error "dataset loading". We were running Redis 6.0.8 and upgrading the replica to 6.2.1 allowed the replica to sync and become healthy in seconds. I'm still not sure why the sentinels thought the replica was healthy as issuing commands to it always returned an error.
- gigatexal 6y ago
  Whoa! You should blog about this for posterity for the next hopeless soul who has to figure out why the thing that never dies died.
- deleted 6y ago
  [deleted]
lefrancaiz 6y ago
>The sharding project bought us some time regarding database capacity, but as we soon found out, there was a huge single point of failure in our infrastructure. All those shards were still using a single Redis. At one point, the outage of that Redis took down all of Shopify, causing a major disruption we later called “Redismageddon”. This taught us an important lesson to avoid any resources that are shared across all of Shopify. It seems that it does happen sometimes, however. https://shopify.engineering/e-commerce-at-scale-inside-shopifys-tech-stack https://shopify.engineering/e-commerce-at-scale-inside-shopi...
spullara 6y ago
At Twitter we hit 15s memory allocation pause times due to fragmentation. We had to switch the memory allocator to fix it.
- hrpnk 6y ago
  Which memory allocator did you end up using?
  - spullara 6y ago
    I think we settled on jemalloc but my memory may be failing me.
    - o-__-o 6y ago
      I see what you did there
    - lenn0x 6y ago
      You are correct that's what it was

We had a saying at my old job: if something’s broken it’s never Redis. Redis is such a tank in my experience. We set it up. Secured it. And then forgot about it.

junon 6y ago

Absolutely. Of course I plan for what happens if Redis were to fail, but I don't remember a single time it ever did. It really, truly is a tank. Antirez, if you're reading this (I know you'll eventually find your way here), thanks for making my job a little easier over the years <3

rshaw1 6y ago

I thought the same up until yesterday! Yesterday the replica randomly disconnected from the master and could no longer reconnect, the resyncs were failing because a replication buffer on the master was being hit (https://redislabs.com/blog/top-redis-headaches-for-devops-replication-buffer/ https://redislabs.com/blog/top-redis-headaches-for-devops-re...). Once that buffer was increased the replica was able to sync the snapshot from the master but for some reason was taking a very long time to load the snapshot, during this time the sentinels thought the replica was healthy again and started allowing the application to read from them, of course Redis responded with an error "dataset loading". We were running Redis 6.0.8 and upgrading the replica to 6.2.1 allowed the replica to sync and become healthy in seconds. I'm still not sure why the sentinels thought the replica was healthy as issuing commands to it always returned an error.

gigatexal 6y ago
Whoa! You should blog about this for posterity for the next hopeless soul who has to figure out why the thing that never dies died.
deleted 6y ago
[deleted]

lefrancaiz 6y ago

>The sharding project bought us some time regarding database capacity, but as we soon found out, there was a huge single point of failure in our infrastructure. All those shards were still using a single Redis. At one point, the outage of that Redis took down all of Shopify, causing a major disruption we later called “Redismageddon”. This taught us an important lesson to avoid any resources that are shared across all of Shopify. It seems that it does happen sometimes, however. https://shopify.engineering/e-commerce-at-scale-inside-shopifys-tech-stack https://shopify.engineering/e-commerce-at-scale-inside-shopi...

spullara 6y ago

At Twitter we hit 15s memory allocation pause times due to fragmentation. We had to switch the memory allocator to fix it.

hrpnk 6y ago
Which memory allocator did you end up using?
- spullara 6y ago
  I think we settled on jemalloc but my memory may be failing me.
  - o-__-o 6y ago
    I see what you did there
  - lenn0x 6y ago
    You are correct that's what it was