4 ms·

We built a system called „friendly fire“ that nukes a server every 10 minutes. It has changed the mindset of all engineers and made our infrastructure missile-p

• brianpayne2 7y ago
  That's awesome. Probably has helped your overall uptime numbers too. We couldn't get SLT approval to implement this, we had to settle for war games in dev/stage
• jaggederest 7y ago
  We used to boot ~3x the servers we needed, run a hard load on them for a while, performance test them, and kill the "weakest" 2/3rds. You can get a bunch of nodes on iffy hardware or greedy neighbors (or all on the same physical box) and see significant performance improvements that way. Of course this was a decade ago, but I think the fundamentals are still sound, as far as being skeptical about the quality and longevity of your nodes in a virtual environment.
  • isbvhodnvemrwvn 7y ago
    At my previous job we did that for DB nodes on AWS. Still definitely a known technique.
• agumonkey 7y ago
  It reminds me about the feeling of competition. You might fail during a tournament, but you know perfectly where you stand now.