Another part of my career, I worked on a team that ran an automated content detection service (like YouTube ContentId.) The database holding signatures for kno

Another part of my career, I worked on a team that ran an automated content detection service (like YouTube ContentId.) The database holding signatures for known music samples was sharded by artist. Not as crazy as you might think. You get a trial sample and you send it to all the shards which chunk on it in parallel then you just wait for the servers hosting each shard to respond. But then Prince died. The queue for the server that owned the shard Prince was in backed up. Then our proxy that distributed trials to each shard backed up. Then our regular reverse proxy backed up. Then our anemic load balancer fell over. What we learned for about the bazillioneth time is keep a backup of each shard handy so you can add it into the rotation and pay an intern to watch Facebook looking for news about recently deceased musicians.