3 ms·
Once you reach this level of enlightenment ("any session should be revokable and fail closed"), there are still higher levels of abuse you'll need to prevent.
by echelon 5y ago
Once you reach this level of enlightenment ("any session should be revokable and fail closed"), there are still higher levels of abuse you'll need to prevent.
Let's say you keep your user sessions in Redis and replicate active-active between multiple regions. Your service allows for more than one active session per user, and has a second index of such "live sessions" so that users can terminate all of their sessions at once (eg. "Log out all of my devices").
You use optimistic concurrency to update multiple indices and pieces of information. Session mutation events and validations can occur anywhere.
What if an attacker slowly logs in thousands of different accounts, hundreds of sessions apiece, then logs everything out in unison?
Redis is single threaded, and all of your validation reads are going to get jammed with impossible to clear logout events. Replication is going to play these events out in all of your Redis clusters, meaning each of your regions will fall over. Nothing in logged-in flow can be processed, and you're stuck.
That was fun to fix.
I've got so many horror stories... Redis is great, but engineering at scale is tough even with great tools.