6 ms·

How Gas app cut its Redis Engine CPU Utilization by 80%

20 points by aloukissas 4y ago

bravetraveler 4y ago
Ah, yes - the usual fix: return true; things do indeed get faster when there's less to do. They patched out a gross inefficiency that they created
- mdeeks 4y ago
  I think the entire point of this post is just to share a war story and help others understand some Redis problems and debugging steps. It isn't some deeper optimization guide or anything. This exact problem is something I have caused and encountered multiple times in my career now. On my first day at Instacart the entire site was down in an extended outage due to a hot key issue exactly like this. I think stories like this are worth sharing with the community even if it seems trivial to yourself.
  - bravetraveler 4y ago
    That's fair, there is value (to someone) in showing how the sausage is made. That said, the diagnostics exhibited here are the exact things I'd do, as a person who visits Redis once maybe every two years. They're not exactly groundbreaking, and I was hoping to find revelations here; not basics. Looking at what the cluster as a whole is doing, and a specific node, and trying to move it. Amazing.
- zimpenfish 4y ago
  > They patched out a gross inefficiency that they created You'd hope that code reviews would catch this kind of thing, especially on a small team.
  - boruto 4y ago
    I would also understand not everything is always perfect, things happen and people learn as they make mistakes
  - aloukissas 4y ago
    This is team of 2 engineers that's building + shipping faster than most teams I've seen while having insane number DAUs. Moving fast (and patching problems even faster) is the name of the game here and these guys delivered.
dementiapatent 4y ago
Fear of this is why I prefer a sorted set with ZREMRANGEBYSCORE to remove expired items
nickphx 4y ago
Redis provides an app called redis insight, works great for debugging such issues and it mostly supports elastic cache. https://github.com/redisinsight/redisinsight https://github.com/redisinsight/redisinsight

20 points by aloukissas 4y ago

Ah, yes - the usual fix: return true; things do indeed get faster when there's less to do. They patched out a gross inefficiency that they created

mdeeks 4y ago
I think the entire point of this post is just to share a war story and help others understand some Redis problems and debugging steps. It isn't some deeper optimization guide or anything. This exact problem is something I have caused and encountered multiple times in my career now. On my first day at Instacart the entire site was down in an extended outage due to a hot key issue exactly like this. I think stories like this are worth sharing with the community even if it seems trivial to yourself.
- bravetraveler 4y ago
  That's fair, there is value (to someone) in showing how the sausage is made. That said, the diagnostics exhibited here are the exact things I'd do, as a person who visits Redis once maybe every two years. They're not exactly groundbreaking, and I was hoping to find revelations here; not basics. Looking at what the cluster as a whole is doing, and a specific node, and trying to move it. Amazing.
zimpenfish 4y ago
> They patched out a gross inefficiency that they created You'd hope that code reviews would catch this kind of thing, especially on a small team.
- boruto 4y ago
  I would also understand not everything is always perfect, things happen and people learn as they make mistakes
- aloukissas 4y ago
  This is team of 2 engineers that's building + shipping faster than most teams I've seen while having insane number DAUs. Moving fast (and patching problems even faster) is the name of the game here and these guys delivered.

dementiapatent 4y ago

Fear of this is why I prefer a sorted set with ZREMRANGEBYSCORE to remove expired items

nickphx 4y ago

Redis provides an app called redis insight, works great for debugging such issues and it mostly supports elastic cache. https://github.com/redisinsight/redisinsight https://github.com/redisinsight/redisinsight