3 ms·

I think you are putting together two concepts that are actually different: "backend instance per customer" and "deploy the service in the customer's infrastruct

I think you are putting together two concepts that are actually different: "backend instance per customer" and "deploy the service in the customer's infrastructure". I don't have much to say about the 2nd one, but my team does the first one for our (internal) customers, so we deploy and manage in our own accounts a whole service stack for each of our customers. It's a nightmare. We are going to do some rearchitecture work next year to move away from it, because it drains so much of our time in operational load. One example of a major pain point that we have is encountering random failures in 3rd party services, such as AWS. For a more classic service that you deploy 2 or 3 times a week, if CloudFormation deployments fail once every thousand deployments for random errors, you'll have one failure per year. Well, we deploy thousands of instances of our service 2-3 times a week. So we have failures every single time we deploy. Oncall is a nightmare (I don't love oncall anymore, I created my login in my previous team) because we just fix a tsunami of similar tickets that are each from a different instance of our service, for a single customer, but often with a different root cause. We probably have half of our headcount dedicated to initiatives that wouldn't need to exist if we had a more classic 1 service = 1 instance approach. Just don't do it.

kylecordes 4y ago
The connection between the concepts is: if your multi-tenant strategy is to deploy (hopefully only while in active use) an instance per customer, then the additional effort to provide on-site installations is relatively low, compared to other multi-tenant strategies. Whether on site installs are a good thing to offer, of course depends on how valuable that is to your customers versus the cost/effort to support it. As you point out there are major trade-offs! If your product architecture requires substantial multi-service complexity per tenant, that points away from the instance per customer strategy.
travisjungroth 4y ago
I can see how this is horrible for internal tools. The incentives are all messed up.
gregplaysguitar 4y ago
Out of interest are you deploying exactly the same software to each instance? Or are there customisations? My work deploys an instance per customer, and I haven’t encountered the same problem, but they number in the tens at this stage so not at the same scale yet. Would love to hear any dos or don’ts you’ve picked up, being further ahead in terms of scale!
- iLoveOncall 4y ago
  They're all exactly the same software, the only customization is configuration (more or less servers, getting data from this or that S3 bucket, etc.). To be honest it comes down a lot to the tools you are using, you have to make sure they scale. For example we deploy one CloudFormation stack per customer (3-4 actually but it's not important), and those are deployed all as part of a single deployment pipeline (think AWS CodePipeline type of thing). Well, turns out this pipeline thing does not like deploying to thousands of targets at once and only supports a few hundreds at most. We had to refactor the way we do that part because just loading the page with all the targets was taking 5 minutes. We also had to split our targets across 2 pipelines, and soon we'll have to go to 3, and it's a pain in the ass and a big engineering headache every time. The main issue comes from operations, invest in automation ASAP, and make sure you make your customers accountable for their own mistakes. Half of our tickets are raised because a customer changed something on their side without thinking of changing the configuration in our system, which then breaks. If we alarmed them instead of ourselves it would be a lot easier.

kylecordes 4y ago

The connection between the concepts is: if your multi-tenant strategy is to deploy (hopefully only while in active use) an instance per customer, then the additional effort to provide on-site installations is relatively low, compared to other multi-tenant strategies. Whether on site installs are a good thing to offer, of course depends on how valuable that is to your customers versus the cost/effort to support it. As you point out there are major trade-offs! If your product architecture requires substantial multi-service complexity per tenant, that points away from the instance per customer strategy.

travisjungroth 4y ago

I can see how this is horrible for internal tools. The incentives are all messed up.

gregplaysguitar 4y ago

Out of interest are you deploying exactly the same software to each instance? Or are there customisations? My work deploys an instance per customer, and I haven’t encountered the same problem, but they number in the tens at this stage so not at the same scale yet. Would love to hear any dos or don’ts you’ve picked up, being further ahead in terms of scale!

iLoveOncall 4y ago
They're all exactly the same software, the only customization is configuration (more or less servers, getting data from this or that S3 bucket, etc.). To be honest it comes down a lot to the tools you are using, you have to make sure they scale. For example we deploy one CloudFormation stack per customer (3-4 actually but it's not important), and those are deployed all as part of a single deployment pipeline (think AWS CodePipeline type of thing). Well, turns out this pipeline thing does not like deploying to thousands of targets at once and only supports a few hundreds at most. We had to refactor the way we do that part because just loading the page with all the targets was taking 5 minutes. We also had to split our targets across 2 pipelines, and soon we'll have to go to 3, and it's a pain in the ass and a big engineering headache every time. The main issue comes from operations, invest in automation ASAP, and make sure you make your customers accountable for their own mistakes. Half of our tickets are raised because a customer changed something on their side without thinking of changing the configuration in our system, which then breaks. If we alarmed them instead of ourselves it would be a lot easier.