12 ms·
It seems to me like one thing that would have made these failures way more manageable would be a solid adaptive load throttling framework. Presumably, the total
by HALtheWise 5y ago
It seems to me like one thing that would have made these failures way more manageable would be a solid adaptive load throttling framework. Presumably, the total load only exceeded their maximum capacity by a couple percent, so a good identity-aware throttling framework should be able to maintain full functionality for ~97% of users by shedding requests early in the flow, or probably better if some automated traffic is prioritized lower and can be dropped first. If I remember correctly, Tencent had a nice paper about how they do this, but I can't find a link now.
- HALtheWise 5y agoEdit: https://www.cs.columbia.edu/~ruigu/papers/socc18-final100.pdf https://www.cs.columbia.edu/~ruigu/papers/socc18-final100.pd...
- getcrunk 5y agoTldr?
- HALtheWise 5y agoWhen a request hits the first server in the backend, it gets assigned a priority based on the service priority (i.e. wechat pay is prioritized over games) and a hash of the user's ID and the current date. When any service gets overloaded, it propagates a message to the other services that depends on it telling them to drop traffic below some threshold priority. That means the traffic gets dropped as early as possible, and the username hashing means people don't just keep refreshing until they can get through.
- getcrunk 5y agoOh wow. To both ur tldr and the method. Hmm gives me ... Many ideas