I've never operated a SAAS app at large scale (ie. millions of customers with 50-100+ machines, etc.) but for smaller deploys I must say that things haven't eve

I've never operated a SAAS app at large scale (ie. millions of customers with 50-100+ machines, etc.) but for smaller deploys I must say that things haven't ever gotten that bad. In some of my own projects I've only gotten bitten by little things a few times over the last 5 years. Like an SSL cert not getting recreated successfully, but this could have been prevented at the time if I had registered the LE account with an email address to get notified it wasn't getting renewed in time. If you put in your due diligence with writing tests, run them automatically as part of your CI pipeline, stick with stable software / tools and keep things as simple as possible until they no longer work then you'll set yourself up for a strong base to work off of. Then as you encounter issues, you automate fixing them as soon as possible. Having monitoring in place to prevent disasters helps too. Like getting notified of unusual CPU / memory / disk usage and getting warned before it becomes a real problem. Sure this requires being messaged but it also means you probably have at least a day's notice before you need to take action. That means you don't need to be glued to a pager and respond in 5 minutes because your site is down. Big difference. This sort of applies to customer support too. I currently do personal customer support for 30,000+ folks who take one of my programming related courses. From the outside you would think I'd be slammed with requests to the point where every day involves answering questions for 2 hours but really it's nothing like that. With a strong base (a working course that stays updated) it's a handful of emails most days and quite often times nothing.