9 ms·
Ways to shoot yourself in the foot with Redis
- _8j50 3y agoDon't expose your redis to the internet (please!). Don't whitelist large swathes of your cloud/hosting provider's subnets either. Of course redis isn't special, mongo, elastic, docker, k8s,etc... even if it is a testing server and you will never put important data on it.
- amenghra 3y agoThis. Configure private vlans and/or Wireguard or whatever VPN software you prefer.
- nforgerit 3y agoAnd what about mTLS?
- GauntletWizard 3y agoMTLS doesn't affect this advice at all. You should, where possible, use MTLS because it's good security. You shouldn't leave your redis server open to the internet anyway, to cut down on logspam. With MTLS, a good security posture is to log every connection establishment, with basic metadata about the certificate involved - it's SAN and public key hash are the best bet. For troubleshooting, do that logging before the authentication decision. But anyone can make their own certificate, so keeping network controls keeps that list free of clutter.
- _8j50 3y agomtls solves the common redis abuses I know of, but still, don't presume a vuln in redis may not be discovered in the future. But if you are going out of your way to configure mtls the I am guessing you have a good reason to?
- amenghra 3y agoSome companies build their trust model on top of mTLS and that's fine. TLS handles the authentication before anything hits redis. I can see people debating the pros and cons of say Wireguard vs mTLS vs ipsec vs other protocols. Such debate might be useful if you are starting from scratch. However, if you have existing infrastructure using either of these, you would need a compelling reason to switch.
- welder 3y agoChange the default `stop-writes-on-bgsave-error` to "no" or you're asking for trouble... a ticking time bomb.
- chrisbolt 3y agoIsn’t it another ticking time bomb to accept writes that will be lost if the server is shut down?
- GauntletWizard 3y agoExpecting that any key in redis will be there next time you read it is a ticking timebomb. Redis is not a database. It's a cache. Unless you're using AOF mode with fsync always, you can lose writes. If you're doing that, you should be using a real database instead.
- chrisbolt 3y agoThe first line of redis.io: > The open source, in-memory data store
- GauntletWizard 3y agoI don't mean to knock redis. I love redis. I implore you to benchmark Redis in FSYNC_ALWAYS vs Postgres. I encourage you to benchmark it as well with FSYNC_EVERYSEC and understand the tradeoffs that makes - Postgres is still very competitive with EVERYSEC in most workloads, and with a lot less tradeoffs in data reliability.
- yawaramin 3y agoYeah, in-memory. That should tell you it's not a persistent data store.
- deleted 3y ago[deleted]
- welder 3y ago
- kgeist 3y agoAnother one: don't use distributed locks using Redis (Redlock) as if they were just another mutex. Someone on the team decided to use Redlock to guard a section of code which accessed a third-party API. The code was racy when accessed from several concurrently running app instances, so access to it had to be serialized. A property of distributed locking is that it has timeouts (based on Redis' TTL if I remember correctly) - other instances will assume the lock is released after N seconds, to make sure an app instance which died does not leave the lock in the acquired state forever. So one day responses from the third party API started taking more time than Redlock's timeout. Other app instances were assuming the lock was released and basically started accessing the API simultaneously without any synchronization. Data corruption ensued.
- GauntletWizard 3y agoYou should do two things to combat this- one is to carefully monitor third party API timings and lock acquisition timings. Knowing when you approach your distributed locking timeouts (and alerting if they time out more than occasionally) is key to... Well, using distributed locks at all. There are distributed locking systems that require active unlocking without timeout, but they break pretty easily if your process crashes and require manual intervention. The second is to use a redis client that has its own thread - your application blocking on a third party API response shouldn't prevent you from updating/reacquiring the lock. You want a short timeout on the lock for liveness but a longer maximum lock acquire time so that if it takes several periods to complete a task you still can. The third is to not use APIs without idempotency. :)
- remote_phone 3y agoThat doesn’t make sense, they can’t assume the lock is freed after the timeout. They have to retry to get the lock again, because another process might have taken the lock. Also, redis is single threaded so access to redis is by definition serialized.
- codegladiator 3y agoThe lock is explicitly release by the redis server itself after the ttl. It's not that the Client will assume that the lock is released.
- ljm 3y agoI've found that your mileage will vary when using Redis in clustered mode because the even if there is an official Redis driver in your language of choice that supports it, this might not be exposed by any libraries that depend on it. In those cases you'll just be connecting to a single specific instance in the cluster but will mistakenly believe that isn't the case. I've noticed this particularly with Ruby where the official gem has cluster and sentinel support, but many other gems that depend on Redis expose their own abstraction for configuring it and it isn't compatible with the official package. Of course, I think that running Redis in clustered mode is actually just another way to shoot yourself in the foot, especially if a standalone instance isn't causing you any trouble, as you can easily run into problems with resharding or poorly distributing the keyspace. Maybe just try out Sentinal for HA and failover support if you want some resilience.
- jrockway 3y agoIt seems like you can run Envoy as a sidecar next to each application instance to allow non-cluster-aware libraries to use the cluster: https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/other_protocols/redis.html#redis-cluster-support https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overv...
- koolba 3y ago> I wrote a basic session cache using GET, which fell back to a database query and SET to populate the cache in the event of a miss. Crucially, it held onto the Redis connection for the duration of that fallback condition and allowed errors from SET to fail the entire operation. Increased traffic, combined with a slow query in Postgres, caused this arrangement to effectively DOS our Redis connection pool for minutes at a time. This has nothing to do with the redis server. This is bad application code monopolizing a single connection waiting for an unrelated operation. A stateless request / response to interact with redis for the individual operations does not hold any such locks.
- philbo 3y ago> This has nothing to do with the redis server. This is bad application code monopolizing a single connection waiting for an unrelated operation. Well, yes. That is why the preceding sentence, which you didn't quote, said "poorly-implemented application logic". So thanks for agreeing with my post, I guess. The point, in case you missed it, was to advertise ways I'd fucked up and hopefully help others not to fuck up the same way in future. It was never my intention to say Redis was the problem and I'm sorry if it made you think that.
- zgluck 3y agoAnd the default limit is 10k connections. https://redis.io/docs/reference/clients/#maximum-concurrent-connected-clients https://redis.io/docs/reference/clients/#maximum-concurrent-...
- adventured 3y agoI have been using Redis for a long time and one of the things I love about it, is how difficult it is to shoot yourself in the foot with it. From the first use, after briefly reading some basic tips on what not to do, it was ridiculously simple to just get to work with it. I've never once run into a security or performance issue with it.
- jontonsoup 3y agoHas anyone seen max (p100) client latencies of 300 to 400ms but totally normal p99? We see this across almost all our redis clusters on elasticache and have no idea why. CPU usage is tiny. Slowlog shows nothing.
- nicwolff 3y agoAre you evicting or deleting large sets (or lists or sorted sets)? We use a Django ORM caching library that adds each resultset's cache key to a set of keys to invalidate when that table is updated – at which point it issues `DEL <set key>` and if that set has grown to hundreds of thousands – or millions! – of keys the main Redis process will block completely for as long as it takes to loop through and evict them.
- jontonsoup 3y agonope!
- secondcoming 3y agoIs it doing backups?
- jontonsoup 3y agoMy understanding is elasticache does not let you turn them off.
- secondcoming 3y agoThat would be surprising, have you tried with CONFIG SET xyz ?
- jontonsoup 3y agoRunning from the client side says that the config command doesn’t exist. Not sure how to run from the server side on elasticache.
- scrame 3y agoI had a jr dev connect and typed 'flushall' because he thought it would refresh the dataset to disk. thankfully it was on a staging env, I think he's at google now.
- spacephysics 3y agoOne time during my internship years ago I took down a production server because of a command I ran on it that I didn’t fully understand. Since then I treat any prod server terminal like I’m entering launch codes for a middle system. Anything outside of ls or cd I’m very careful, read the command a couple times before executing, etc.
- returningfory2 3y agoIn my opinion you weren't at fault here. Production systems should be designed so that one person can't inadvertently destroy things.
- RyanHamilton 3y agoThis is the way.
- ljm 3y agoIn almost every place I've worked at, the most difficult thing has been getting people out of ad-hoc JFDI style development and debugging, where everything in production is fair game, and into a process where you avoid touching production as much as humanly possible. Takes a lot of effort to stop people opening up a shell in prod or grabbing a prod DB dump or even just connecting to the prod datastore directly from their local env.
- deleted 3y ago[deleted]
- tetha 3y agoFor critical and overall... fiddly things, we've grown into a culture of writing down reviewable plans and possibly executing these plans in pairs. We tend to go ahead and either use a runbook, or whatever experience we might have, to setup a pretty detailed plan of what to run on which systems with which purpose. You can then throw these plans at someone else to review. Sure, it takes an hour or two more to setup a solid plan and waiting for a review takes time as well. But this has turned into a great tool to build up experience in weird parts of the infrastructure.
- berkle4455 3y ago> Crucially, it held onto the Redis connection for the duration of that fallback condition and allowed errors from SET to fail the entire operation. What? Was this inside a MULTI (transaction) or something? This isn't a flaw of Redis being single-threaded. Honestly all of these "footguns" sound like amateur programmer mistakes and have zero to do with Redis.
- philbo 3y agoNo. As it explains at the beginning of the paragraph you're quoting: > If you're particularly naive, like I was on one occasion, you'll exacerbate these failures with some poorly-implemented application logic. Then a few paragraphs above that is this sentence: > The gotchas that follow were all occasions when I didn't use it correctly. I'm not sure how to make it more clear that I'm criticising myself, not Redis, in the post, but that's the intention. If you have suggestions how I could make it more obvious, please let me know.
- berkle4455 3y agoThe title comes across like these are faults of Redis and that if you're not particularly careful about you'll shoot yourself in the foot. > I'm not sure how to make it more clear that I'm criticising myself "Mistakes I made while building applications on Redis"
- stlava 3y agoMy team manages a handful of clusters at work and I wrote on an internal redis client proxy (it's on my todo list to opensource). A few things I tell other teams to set them up for success (we use Elasticache): - Connection pooling / pipelining and circuit breaking is a must at scale. The clients are a lot better than they used to be but it's important developers understand the behavior of the client library they are using. Someone suggested using Envoy as sidecar proxy, I personally wouldn't after our experience with it with redis but it's an easy option. - Avoid changing the cluster topology if the CPU load is over 40%. This is primarily in case of unplanned failures during a change. - If something goes wrong shed load application side as quick as possible because Redis won't recover if it's being hammered. You'll need to either have feature flags of be able to scale down your application. - Having replicas won't protect you from data loss so don't treat it as a source of truth. Also, don't rely on consistency in clustered mode. - Remember Redis is single threaded so an 8xl isn't going to be super useful with all those unused cores. Things we have alarms on by default: - Engine utilization - Anomalies in replication lag - Network throughput (relative to throughput of the underlying EC2 instance) - Bytes used for cache - Swap usage (this is the oh shit alarm)
- rmbyrro 3y agoYou need two line breaks after each item so they'll show stacked as a list
- unlitioldem 3y ago[flagged]
- resonious 3y ago> One common mistake is serialising objects to JSON strings before storing them in Redis. This works for reading and writing objects as atomic units but is inefficient for reading or updating individual properties within an object I would love to see some numbers on this. My intuition says there are probably some workloads where JSON strings are better and some where one key per property is better.
- yawaramin 3y agoRedis is great, it's a great piece of software. One way I shot myself in the foot (kinda) with it: used it with a 1:N query fanout pattern. I.e., issued N queries to Redis for 1 incoming query to my service. My service by design needs to do N queries (it's a long story). But Redis is not really designed to be used like this and I was putting it under very high load. I swapped it out with an SQLite cache recently and got rid of the errors that would pop up from putting extreme stress on the Redis server.