I worked on a system with the shared app + shared database model. At its core, we received events (5-10KB) from customers and did something with those events. I

I worked on a system with the shared app + shared database model. At its core, we received events (5-10KB) from customers and did something with those events. In total, we were receiving 8K-10K events per second. In terms of security and privacy, isolating an individual tenant from others wasn't so much of a concern as each tenant was a customer within the organization with the same data classification level. So from a security perspective, we were "okay". Where this gets interesting is that one tenant would suddenly decide to push a massive volume of data. Now processing events within a specific SLA was a critical non-functional requirement with this system. So then our on-call engineers would get alerts because shared messaging queues were getting backed up since Mr. Bob had decided to give us 3-5x his typical volume. The traffic spike from one customer, which could last from minutes to hours, would negatively impact our SLAs with the other customers. Now all the customers would be upset. ^_0 Being internal customers, they were willing to pay for the excess traffic, but we didn't really have the tooling to auto scale. Our customers also didn't want us to rate limit them. Their expectation was that when they have traffic spikes, we need to be able to deal with it. Now – we didn't want to run extra machines that sat idle like 90% of the time. And when we had these traffic spikes, we'd see our metrics and find maxed out CPU, memory, and even worse, we'd consume all disk space from the log volume on the machines filling everything up. The hosts would become zombies until someone logged in and manually freed up disk. There were a few lessons learned: 1. Rate limit your customers (if your organization allows). 2. If your customers are adamant that in some instances each month, they need to be able to send you 5x the traffic without any notice, then you can't just rate limit them and be done with it. We adopted a solution where we would let our queues back up while some monitors would detect the excessive CPU or memory usage and would start scaling out the infrastructure. Once our monitors saw the message queues were looking normal again, they'd wait a little while and then scale back down. 3. When you're processing from a message queue, you need to capture metrics to track which customer is sending you what volume. Otherwise, you can have metrics on the message queues themselves and have one queue per customer. 4. If it's a matter of life and death (it wasn't, but that's how one customer described it), something you can do is stop logging when disk space usage exceeds a specific amount. 5. Also – when you have a high throughput system, think very carefully about every log statement you have. What is its purpose? Does it really add value?