My work is mainly massive data pipelines (I use Project Reactor, MongoDB, etc.) Massive is subjective: in my case this is trying to fit, on a single small machi

My work is mainly massive data pipelines (I use Project Reactor, MongoDB, etc.) Massive is subjective: in my case this is trying to fit, on a single small machine tens of millions of complex objects per second, hundreds of processing steps, gigabytes of data from MongoDB every second sustained over days, weeks and even months, using wide range of techniques, exotic data structures to try to speed up stream processing. Then if the problem requires, scaling it up. Some stuff I learned over the years somebody might find useful: 1) Tune your app for max load. This is the most important parameter, nobody really cares what happens when you have 10% of the load -- what is important is how it behaves when you are close to, at or above capacity. If your app can behave nicely as it reaches its limit it should also be able to behave nicely at less than it. Why scale down your thread pools? Threads do not use extra resources when they are not running -- if you had resources at max capacity you now have resources laying around to have an idle thread. Just find the number you need at your max load and leave it at that. Your framework should be able to allocate concurrency until it hits the limit at which point it will be in the state as tuned for max load. 2) A trick I found is to make sure the application becomes more efficient per-transaction as the load increases. This means things like increasing batch sizes, etc. Increasing concurrency can have opposite effect. When per-transaction efficiency increases the application tends to behave nicely as it approaches its max throughput and it is much easier to keep stable at or close to the limit. 3) One way of increasing per-transaction efficiency is increase batch sizes. Imagine a system where every user login requires an object to be fetched from the database. Rather than generate database request for each login, pipe them together and batch them every 100ms. Then take that 100ms batch and generate single request to the database engine, fetch the results, distribute to clients. Now as more and more users log in your application does not create more requests to the database -- it creates 10 requests every second regardless of the load. Larger requests should translate to higher efficiency per item. When the number of users logging in reaches limit you have option of buffering requests (preserving the database from getting overloaded) but this is more likely going to dramatically drop the efficiency. Instead, you can drop requests and require client to implement exponential backoff when retrying the requests. 4) I actually use control engineering techniques to change parameters of the system (just not the concurrency). For example, you can try to regulate processing latency or server load by modifying the limit of transactions in flight. You can even try to regulate database load (I have MongoDB report its state to app layer and app layer adjust its parameters to keep database in safe operating conditions). The reason to do this is because the system is incredibly complex and with all asynchronous stuff happening there is nobody that really understands what is happening. Feedback control helps stabilising system in much wider range of circumstances without people having to change any configuration which is one of big reliability improvements. Feedback control will regulate faster and kick in before anybody notices anything and has much less chance of messing stuff up. One warning, though, it is important to put some boundaries on safe operating space for the control. Whatever control you implement there are going to be limits to what it can and can't do. There are ways to implement stuff so that you can ensure your app stays within limits -- for example start dropping request when the app reaches throughput limit.