The product I work on started life as an on-prem-only software. We're mostly hosting customers in AWS these days. The smallest on-prem customers run on a singl

The product I work on started life as an on-prem-only software. We're mostly hosting customers in AWS these days. The smallest on-prem customers run on a single server. We have multiple separate web applications, and a couple separate backend services. In AWS, we have each separate web app/backend service on its own set of auto-scaling EC2 instances, and have ELB+haproxy and some scripts to manage everything. Each customer (100s) get their own DB, but the DBs are all hosted across a few actual DB servers. We use the same codebase for on-prem and AWS, and there's really not much difference in operation: as far as the applications are concerned, on-prem is just the same multi-tenant environment that happens to have a single tenant. However, I would absolutely go all-cloud if I could. Two major reasons: AWS/cloud services, and database. The first point is easy: we just can't really use any AWS or other 3rd party SaaS stuff ourselves, other than for our own support infrastructure. The app has to deploy to a self-contained on-prem environment. This is really both a blessing and a curse. On one hand, we have to re-invent some things that could otherwise be off-the-shelf from AWS or other offerings. On the other hand, while switching to another cloud would not be trivial, there's nothing in our core application code that relies on AWS at all. This also makes our technology stack a bit constrained: we could benefit immensely from a time-series database, but instead we use a RDBMS (MS SQL Server) because, well, we have enough trouble with on-prem administering the database as it is. The second point -- database -- is the cause of so, so much frustration. Very few people can run databases well. It's the same old story: when you're small, it's no problem, but as you grow the DB performance becomes critical (for reference, while our smaller customers have DBs that are 1-2GB, dozens of our customers have DBs in the 100s of GB, and a couple are over 1TB, and typically there's about 15% inbound data per day, though after processing not that much gets stored). I/O speed is always an issue.. we've had customers trying to run with their I/O on a NAS that's also shared with a dozen other applications ("We spent 100k on our NAS and the sales guy said it'll do everything, so it must be your application that's slow"). I have heard our customers say things like "But I can't make a full database backup, I don't have enough disk space". There's also the inevitable "what hardware do I need to buy?" question, that even our own sales team is frustrated by. Overspec, and they are upset at cost, or -- probably worse -- are frustrated you made them spend all that money when their machine sits mostly idle. Underspec, and it's now your problem that it doesn't work, because they bought what you told them. It took me months before I convinced even our own team the best we can do is provide a couple examples of hardware setups, and caveat the heck out of it ("with this many devices monitored, with an average of this many parameters, at this frequency, with this many alerts configured, this many concurrent users," etc). It can be hard enough to isolate the performance problems under load when you operate the whole stack -- it's incredibly difficult when you can only approximate a customer's setup, and even then, you have to somehow convince them their DB config or I/O is inadequate and then need to spend money to fix it.