I remember hearing on a podcast a story about turning a "backend timed out" failure into "send that data in email and tell the user 'ok, we got it come back and

I remember hearing on a podcast a story about turning a "backend timed out" failure into "send that data in email and tell the user 'ok, we got it come back and check later' instead of an error". I thought that was really clever, and filed it away as something I'd use in an emergency in the future. It also reminded me of a WebTV incident with a database overload. I came up with the idea of shutting down all our last-hop mail delivery servers, which needed a database connection. That caused email to queue in the SMTP inbound Servers, which we drained later when the database was no longer overloaded. The lesson is: use batch-oriented processing from queues to take load off of systems that are designed to work in a transaction-oriented manner, but are failing to do so.