Without knowing more about the source, it's hard to say for sure, but would an explicit test for a failure to connect to memcached have uncovered this? That is

Without knowing more about the source, it's hard to say for sure, but would an explicit test for a failure to connect to memcached have uncovered this? That is, if there is common code to connect to memcached, and a test to purposely generate a timeout, it seems like it would have jumped out as taking far too long. If there's no common code to connect, and every user of it has its own implementation, there might be more bugs like this lurking. Every user of memcache in this particular code base would need to have its own "what if memcached is slow or down?" type test. It seems such a test did not already exist. Following the events in this post, is there now a test to explicitly make sure the delays are reasonable, or was the extra multiplication just removed?