There are several components I'm responsible for. The main one is a real-time analytical engine (think OLAP, for billions of records, sub-second response time

There are several components I'm responsible for. The main one is a real-time analytical engine (think OLAP, for billions of records, sub-second response time and dynamic queries, not pre-baked). The core products depend on it. Each client has its own cluster due to data privacy and isolation (some infra is shared, analytics it's not), some are FedRAMP compliant which makes matters even harder. Most clusters are small (4-5 small nodes, running on Kubernetes or Docker), some are huge (many terabytes of RAM across dozens of machines/containers). For example, we saw a shift in growth patterns (both in the datasets and product talk about features and the future) a couple of years back, so we invested in distributing the dataset and the query engine. We started early because we knew it would take time to build. So we projected our "death date" and built the project backwards from that. By the time the storm came, the scalability was there. We had to rush the deployment a bit, but we were almost there (perf testing, forcing failures and recovery, updating SOPs, etc.) At about the same time, we were somewhat blindsided in part by a 1000x to 10000x increase in data volume on one particular part of the product related to text analytics data (it came after an acquisition), which forced us to scramble and change some of the internal data structures to better compress this data, plus product changes to reduce the volume needed. In the end, the impact was under 10x, which we were able to handle with the distribution work I mentioned earlier. In any case this "scramble" took a quarter, and the strategy we used to increase compression was something that we had thought about internally and discussed before, but we didn't have an incentive to implement it until the demand was there. Sometimes it is like that, we play likely scenarios (usually looking for what will kill us next), think roughly about how to address them, assess the likelihood of that happening and when and decide if it's something we need to do now or later. Most scrambling is to bring forward something we planned for later and adjust it to the new reality.