4 ms·
I can totally imagine how that happens... The software isn't performing well enough, so some programmer says "I can rewrite the request queueing system to pr
by londons_explore 5y ago
I can totally imagine how that happens... The software isn't performing well enough, so some programmer says "I can rewrite the request queueing system to prioritize requests just before the 1ms deadline and deprioritize the ones where the deadline has already been missed". End result, a step...
- eru 5y agoDeep in the internals of Google, we actually had a system that was putting incoming requests not in a queue (first-in-first-out), but in a stack (last-in-first-out). The system in question was essentially not meant to have any backlog of request under normal operations. But when overloaded, for this particular system it was better to serve the requests it could serve very fast, and just shed load on the overflow. (I don't remember more specifics, and even if I could, I would probably not be allowed to give them..)
- willvarfar 5y agoReminded me of this old post by Facebook, well worth a read :) https://engineering.fb.com/2014/11/14/production-engineering/solving-the-mystery-of-link-imbalance-a-metastable-failure-state-at-scale/ https://engineering.fb.com/2014/11/14/production-engineering...
- londons_explore 5y agoI really wish network routers would do this when delivering packets... The other end gets far better information about congestion that way, and congestion control algorithms could be made much smarter. Everyone says "jitter is bad in networks", but the reality is if the jitter is giving you network state information, it is a net positive - especially when you can use erasure coding schemes so the application need not see the jitter.
- CodesInChaos 5y agoCurrently you can assume that if you receive a packet with a higher sequence number, earlier packets were probably lost, and resend them. This heuristic won't work anymore with a LIFO buffer. LIFO sounds annoying for bursty traffic, since you can only start processing the burst after the buffer has cleared.
- londons_explore 5y agoUsing erasure coding, there is no need to know which packets were lost. You just keep sending more erasure coded packets till you get an acknowledgement that all data has been decoded. If a packet from a while ago randomly reappears, it usually isn't wasted throughput either - it can be used to replace any other lost packet, either forwards or backwards in the packet stream (up to some limit determined by the applications end-to-end latency requirement).
- eru 5y agoYou might like BBR: https://blog.apnic.net/2017/05/09/bbr-new-kid-tcp-block/ https://blog.apnic.net/2017/05/09/bbr-new-kid-tcp-block/
- jeffbee 5y agoLIFO is a perfectly reasonable mode, when overloaded. FIFO will give you a stable equilibrium where every request takes too long and fails. LIFO breaks out of that operating point. This is the key feature of the CoDel solution to buffer bloat.
- kristjansson 5y agoI have exactly that pattern in an internal system. Under load, the caller will give up after a fixed timeout and retry, so why waste time on an old request that where the caller has probably already hung up?