3 ms·

I work platform, a quick project is one or two quarters, a decent sized one is multi year, will likely take several deployments to roll out and at least a quart

by juancn 6y ago

I work platform, a quick project is one or two quarters, a decent sized one is multi year, will likely take several deployments to roll out and at least a quarter for testing and stabilization, but when we deliver it to the several thousands clusters it will serve, the chance of a serious incident is close to zero. For this to work, we need to anticipate future requirements. We monitor usage patterns, make proyections, study the market our company serves and try to see trends and shifts. We care about what competitors are pushing and what complementary industries are doing. We are aware of the business, but not in the same way product does. When our plans work (and they mostly do), we usually deliver at the right time when the company needs it. We saw a shift in market patterns that would affect load a year ago, so we started working on scaling certain part of the system. It took us a year to get there, but when it was needed, we had it ready. Sometimes people ask for things that come out of the blue. We were unable to anticipate them and the work to get them done will take at least several quarters. These scenarios are the hardest, when you need to push back, but it's not that we're not doing anything, we're reshuffling several quarters of planning to adjust to the shift in reality, to correct the blind spot, so the next time you ask for something, it's ready, because we started a year ago.

lifeisstillgood 6y ago
Can you give some more detail - I am trying to work out what is deployed to thousands of clusters - is this like a base image containing a new rollout of say Postgres? Or internal tools ? I am genuinely interested - it sounds like you really know your onions and I am trying to work out what scale you do onions on :-)
- juancn 6y ago
  There are several components I'm responsible for. The main one is a real-time analytical engine (think OLAP, for billions of records, sub-second response time and dynamic queries, not pre-baked). The core products depend on it. Each client has its own cluster due to data privacy and isolation (some infra is shared, analytics it's not), some are FedRAMP compliant which makes matters even harder. Most clusters are small (4-5 small nodes, running on Kubernetes or Docker), some are huge (many terabytes of RAM across dozens of machines/containers). For example, we saw a shift in growth patterns (both in the datasets and product talk about features and the future) a couple of years back, so we invested in distributing the dataset and the query engine. We started early because we knew it would take time to build. So we projected our "death date" and built the project backwards from that. By the time the storm came, the scalability was there. We had to rush the deployment a bit, but we were almost there (perf testing, forcing failures and recovery, updating SOPs, etc.) At about the same time, we were somewhat blindsided in part by a 1000x to 10000x increase in data volume on one particular part of the product related to text analytics data (it came after an acquisition), which forced us to scramble and change some of the internal data structures to better compress this data, plus product changes to reduce the volume needed. In the end, the impact was under 10x, which we were able to handle with the distribution work I mentioned earlier. In any case this "scramble" took a quarter, and the strategy we used to increase compression was something that we had thought about internally and discussed before, but we didn't have an incentive to implement it until the demand was there. Sometimes it is like that, we play likely scenarios (usually looking for what will kill us next), think roughly about how to address them, assess the likelihood of that happening and when and decide if it's something we need to do now or later. Most scrambling is to bring forward something we planned for later and adjust it to the new reality.
  - lifeisstillgood 6y ago
    Thank you. It seems more and more that, like terrorists and freedom fighters, one persons platform engineer is another mans product engineer

by juancn 6y ago

lifeisstillgood 6y ago

Can you give some more detail - I am trying to work out what is deployed to thousands of clusters - is this like a base image containing a new rollout of say Postgres? Or internal tools ? I am genuinely interested - it sounds like you really know your onions and I am trying to work out what scale you do onions on :-)

juancn 6y ago
There are several components I'm responsible for. The main one is a real-time analytical engine (think OLAP, for billions of records, sub-second response time and dynamic queries, not pre-baked). The core products depend on it. Each client has its own cluster due to data privacy and isolation (some infra is shared, analytics it's not), some are FedRAMP compliant which makes matters even harder. Most clusters are small (4-5 small nodes, running on Kubernetes or Docker), some are huge (many terabytes of RAM across dozens of machines/containers). For example, we saw a shift in growth patterns (both in the datasets and product talk about features and the future) a couple of years back, so we invested in distributing the dataset and the query engine. We started early because we knew it would take time to build. So we projected our "death date" and built the project backwards from that. By the time the storm came, the scalability was there. We had to rush the deployment a bit, but we were almost there (perf testing, forcing failures and recovery, updating SOPs, etc.) At about the same time, we were somewhat blindsided in part by a 1000x to 10000x increase in data volume on one particular part of the product related to text analytics data (it came after an acquisition), which forced us to scramble and change some of the internal data structures to better compress this data, plus product changes to reduce the volume needed. In the end, the impact was under 10x, which we were able to handle with the distribution work I mentioned earlier. In any case this "scramble" took a quarter, and the strategy we used to increase compression was something that we had thought about internally and discussed before, but we didn't have an incentive to implement it until the demand was there. Sometimes it is like that, we play likely scenarios (usually looking for what will kill us next), think roughly about how to address them, assess the likelihood of that happening and when and decide if it's something we need to do now or later. Most scrambling is to bring forward something we planned for later and adjust it to the new reality.
- lifeisstillgood 6y ago
  Thank you. It seems more and more that, like terrorists and freedom fighters, one persons platform engineer is another mans product engineer