4 ms·
> Also if I am reading it correctly, it sounds like the rate limiting was being done per-IP, which sounds strange. Seems to me that the main mistake was not ra
by CPLX 7y ago
> Also if I am reading it correctly, it sounds like the rate limiting was being done per-IP, which sounds strange.
Seems to me that the main mistake was not rate limiting per account right? If you get 200k password requests for a single user something is severely wrong.
Or maybe they did and it didn't work, he makes reference to a race condition in the original post but doesn't elaborate.
- zrobotics 7y agoLikely this is to prevent malicious attempts to block access to an account- I can definitely imagine someone trying to block a popular account by abusing the password reset mechanism. Imagine attempting to reset a password of a popular account by spamming the p/W reset mechanism, if the limit is per-account instead of per-IP then you could fairly easily deny access to any account.
- ttsda 7y agoHow would you be denying access? The block is only by the password reset system, if you know the password you can still access your account.
- hwstack 7y agoHe didn't really explain it, but I think what was going on is the rate limiting is done per account, and the race condition was a way to circumvent that. He has to make all the requests very quickly because the first thing all the requests are doing is determining if new requests for this account should be ignored. All the requests are received around the same time, they all make this check and decide they are valid requests, then they all report that an attempt was made for that account (locking it). The rate limiting for IPs is probably global (not related to the reset endpoint).
- yardstick 7y agoI think you are dead on, yeah it’s the quick rate of large numbers of requests that avoid the per-account rate limiting. Curious how they resolved this— run all authentication requests for a given user serially and in a consolidated fashion at some point. Exclusive lock the relevant db record before checking the code and recording the failure?
- hwstack 7y agoYeah, my first thought was make every attempt acquire some per user lock with a timeout. It's pretty much the same thing. Either one would have a negligible effect on legitimate requests and would solve the problem.
- roblabla 7y agoCould start by incrementing the value, then checking if it's below the threshold, similar to an atomic fetch_add operation. PostgreSQL has RETURNING clauses, SQL Server has OUTPUT clauses, etc...
- holy_city 7y agoYou could pipe the requests into a FIFO without back pressure so excessive requests are dropped by default.
- chatmasta 7y agoDistributed rate limiting is hard. He could hit multiple front ends simultaneously before they have a chance to catchup to the correct counts.
- yardstick 7y agoYeah it is hard. The enforcement would need to be done on a single backend. Not all users need to have their auth done by the same specific backend, but each user individually should always have their auth go to the same backend (or same concurrency domain, if distributed locking applies to the architecture).
- floatingatoll 7y agoIf you use a sharded “eventual consistency” database cluster to store the number of reset attempts per interval, then sufficient IPs can be purchased to overwhelm your “eventual” threshold unless each reset attempt compels consistency before evaluating whether the limit is reached. That would qualify as a “race condition” in language terms, even if not being the classical sort of one considered such by programmers, and present the easy described fix (take a consistent read lock rather than an inconsistent one or none at all) when checking the rate limit.