6 ms·

If an "account" is an "enterprise" customer (SMB or large, anything with multiple user accounts in it), then yes, I know at least a few successful companies, an

by gomox 6y ago

If an "account" is an "enterprise" customer (SMB or large, anything with multiple user accounts in it), then yes, I know at least a few successful companies, and I would argue in a lot of scenarios, it's actually advantageous over conventional multitenancy. The biggest advantage is flexibility to handle customers requirements (e.g. change management might have restrictions on versioning updates) and reduced impact of any failures during upgrade processes. It's easier to roll out upgrades progressively with proven conventional tools (git branches instead of shoddy feature flags). Increased isolation is also great from a security standpoint - you're not a where clause away from leaking customer data to other customers. I would go as far as saying this should be the default architecture for enterprise applications. Cloud infrastructure has eliminated most of the advantages of conventional multitenancy. If an account is a single user then no. PS: I have a quite a lot of experience with this so if you would like more details just ask.

polote 6y ago
What would be the best, different tables per customer or different db?
- brianwawok 6y ago
  There’s overhead both ways. Multi-tenant is more efficient. What is the most possible users you could have in 5 years? Can your choice be supported in that technology?
  - gomox 6y ago
    Enterprise systems have little to no cross-customer functionality, and customer isolation is paramount. I think you are assuming that they will have a database per individual user, but the original question pointed to "tenants" (i.e. companies using the system).
- gomox 6y ago
  What I have experience with is separate DB, and separate applications. Then basically you orchestrate a bunch of applications, a task for which a myriad of tools already exist.
- brigandish 6y ago
  There's always schema to consider too, you can have thousands of them too. An often overlook tool.
robot 6y ago
Do you not have to have a large dev ops team to handle upgrades or changes for each customer? I'd be against it to do more with fewer people, especially for an unproven < 5 year old startup.
- gomox 6y ago
  Actually, I think conventional multitenant (shared database with tenant_id on database rows) is a bad idea (as well as premature optimization) in the previously described context. One of the companies I am thinking of had no dedicated ops team until 300+ customers, running on 300+ instances of the setup. Probably 40+ employees at the time the first dedicated sysadmin was hired. The trick is that since the production environment is extremely similar to the developer environment, all the tools to upgrade versions, migrate databases, etc, exist and are constantly used by all the developers. When the customer base grows, horizontal expansion is baked into the architecture. When your largest customer grows beyond what you can host, you scale up with trivial mechanisms (database replication, multiple application servers). All you need is orchestration of extremely common, well understood and mature tools. Orchestration is home baked anyway at every company that I've had experience with (there is no off the shelf "multitenant application management system"), but by doing it this way you rely on extremely mature components for all the underlying tools, because they have been used to scale websites since 2000.
  - bearjaws 6y ago
    For context, we implemented your exact model at my job prior to switching to "conventional multi-tenant". I agree with your premise that there are a lot of off the shelf tools to help this, but in enterprise I feel like we hit dozens of other issues and our customers didn't even care if they shared the same database as someone else. If the production environment is extremely similar to dev, the problems are numerous, how do you ensure that a bug in one app server doesn't let users access other request data? We solved this via multiple app servers, but then how do you solve deploying to hundreds of individual app servers? We have 12 services per customer, replicated to 4 physical locations, that means 124500(customers) 24,000 containers to roll out any upgrade completely. (99.99% SLA) Also some customers get different services depending on their tier, so now we need to bake in some form of Salesforce integration into our deployment so we know which services to start up(we didn't do this unfortunately, so all services started up and some didn't do anything). If you split into many physical database servers vs many "logical" databases, how do you monitor that an individual customers performance is adequate? What about APM? One NewRelic instance per customer? One ELK cluster per customer? How do your developers / customer support engineers get access to a specific customer instances both in the software, in the database and APM? I cannot fathom having to manage 500 ELK clusters, even with AWS managed Elastic search. How do you enable ETL and reporting solutions that support all these databases. What if you have data aggregation rights and need to combine different customer's data? What happens when your product team needs to report across all customers to determine what features to work on (gotta be data driven)? Maybe this is only related to healthcare (where I work), but in our industry I cannot imagine our health systems and manufacturers not wanting answers to these. These answers are infinitely simpler in our new multi-tenant model vs our old multiple host model. Writing the processes and documentation around guaranteeing segmentation took less than 8 hours and is only 3 pages of our entire security model. Hi-trust specifically calls out many of these points, and I am sure SOC-2 does as well. In my experience, not a single one of our health systems or auditors has had an opinion either way on this. They only care when you aren't doing APM properly, or taking the necessary steps to prevent attacks both from external parties and by internal parties. We do in-depth third party audits that are shared with our customers. Ensuring good processes for managing risk is always more important than wishful thinking that a design pattern will make your enterprise model work. Our total lines of code for logical segmentation takes up less than 300 lines in all of our services combined. Finally, given that Postgres10+ now natively supports row level security, I fail to see why anyone starting a new project would choose the multiple database option when you can bake in authorization straight into the DB at the lowest level. Sorry this is really long, but I wanted to share my experience & thoughts.
randlet 6y ago
I run a very small SaaS and do this. Each customer gets its own isolated AWS account all organized under a root account for consolidated billing. It works great and has allowed me to develop client specific features where required (using git branches). Customers like the fact that their data is completely isolated from everyone else. Rolling out updates is simple, I just run the same deploy script for each client (after scheduling with the client if need be) and if a deploy goes wrong, you'll usually notice it after it affects one customer rather than all of them. The main downside for me is cost. The cost per customer scales linearly which hurts as you don't get the same economies of scale like you would in single tenant solutions.
- gomox 6y ago
  I don't know your application enough to comment, but that seems like an extreme version of what I'm proposing, which would be to share application and DB servers between customers, but not application code and databases (i.e. you can still run 3 customers on the same EC2 and RDS instances, they just have separate application code and separate database schemas). One of the main benefits, like you describe, is lowered costs. Running an AWS account for each customer is pretty significant overhead and would only be worth it to me if each customer was quite substantial (50K/yr or above?).
  - randlet 6y ago
    The way you are thinking would be much cheaper than what I'm doing with the downside that clients data exists on the same server (but maybe they won't care) and the fact that a performance issue for one client can impact other clients. I haven't found an AWS account per customer to be to bad to manage yet and makes keeping track of your AWS spend per client simple. It's also trivial to run your resources in say Sydney for your Australian clients and London for your European clients.
    - gomox 6y ago
      If it works, then it works :) Company B I mentioned above still runs location specific infrastructure, but each one of these clusters can host several customers. Of course there is some overhead (i.e. very new regions sometimes operate at a loss if there is not a critical mass of customer in that cluster, but they are subsidized by more profitable regions until they reach a critical mass of customers to pay for the whole cluster costs).
juped 6y ago
I think the standard term is "tenant" for that organizational level, no?
bearjaws 6y ago
Your biggest advantage is actually a disadvantage, you have literally enabled your individual customers to boss you around and fork your codebase into a mess of different functionality per customer. Are you a contract shop where you get SOWs to expand your software? This seems like a terrible idea for both product team and the development team, who now need to know what bastardized version of the software each customer is running. What happens when a customer goes "I don't want that update"? Unless you can guarantee that each customer ends up on the same software quickly, this will get messy.
- gomox 6y ago
  Flexible and disciplined is a good alternative to rigid, and for young software business, probably the safer one!

by gomox 6y ago

polote 6y ago

What would be the best, different tables per customer or different db?

brianwawok 6y ago
There’s overhead both ways. Multi-tenant is more efficient. What is the most possible users you could have in 5 years? Can your choice be supported in that technology?
- gomox 6y ago
  Enterprise systems have little to no cross-customer functionality, and customer isolation is paramount. I think you are assuming that they will have a database per individual user, but the original question pointed to "tenants" (i.e. companies using the system).
gomox 6y ago
What I have experience with is separate DB, and separate applications. Then basically you orchestrate a bunch of applications, a task for which a myriad of tools already exist.
brigandish 6y ago
There's always schema to consider too, you can have thousands of them too. An often overlook tool.

robot 6y ago

Do you not have to have a large dev ops team to handle upgrades or changes for each customer? I'd be against it to do more with fewer people, especially for an unproven < 5 year old startup.

gomox 6y ago
Actually, I think conventional multitenant (shared database with tenant_id on database rows) is a bad idea (as well as premature optimization) in the previously described context. One of the companies I am thinking of had no dedicated ops team until 300+ customers, running on 300+ instances of the setup. Probably 40+ employees at the time the first dedicated sysadmin was hired. The trick is that since the production environment is extremely similar to the developer environment, all the tools to upgrade versions, migrate databases, etc, exist and are constantly used by all the developers. When the customer base grows, horizontal expansion is baked into the architecture. When your largest customer grows beyond what you can host, you scale up with trivial mechanisms (database replication, multiple application servers). All you need is orchestration of extremely common, well understood and mature tools. Orchestration is home baked anyway at every company that I've had experience with (there is no off the shelf "multitenant application management system"), but by doing it this way you rely on extremely mature components for all the underlying tools, because they have been used to scale websites since 2000.
- bearjaws 6y ago
  For context, we implemented your exact model at my job prior to switching to "conventional multi-tenant". I agree with your premise that there are a lot of off the shelf tools to help this, but in enterprise I feel like we hit dozens of other issues and our customers didn't even care if they shared the same database as someone else. If the production environment is extremely similar to dev, the problems are numerous, how do you ensure that a bug in one app server doesn't let users access other request data? We solved this via multiple app servers, but then how do you solve deploying to hundreds of individual app servers? We have 12 services per customer, replicated to 4 physical locations, that means 124500(customers) 24,000 containers to roll out any upgrade completely. (99.99% SLA) Also some customers get different services depending on their tier, so now we need to bake in some form of Salesforce integration into our deployment so we know which services to start up(we didn't do this unfortunately, so all services started up and some didn't do anything). If you split into many physical database servers vs many "logical" databases, how do you monitor that an individual customers performance is adequate? What about APM? One NewRelic instance per customer? One ELK cluster per customer? How do your developers / customer support engineers get access to a specific customer instances both in the software, in the database and APM? I cannot fathom having to manage 500 ELK clusters, even with AWS managed Elastic search. How do you enable ETL and reporting solutions that support all these databases. What if you have data aggregation rights and need to combine different customer's data? What happens when your product team needs to report across all customers to determine what features to work on (gotta be data driven)? Maybe this is only related to healthcare (where I work), but in our industry I cannot imagine our health systems and manufacturers not wanting answers to these. These answers are infinitely simpler in our new multi-tenant model vs our old multiple host model. Writing the processes and documentation around guaranteeing segmentation took less than 8 hours and is only 3 pages of our entire security model. Hi-trust specifically calls out many of these points, and I am sure SOC-2 does as well. In my experience, not a single one of our health systems or auditors has had an opinion either way on this. They only care when you aren't doing APM properly, or taking the necessary steps to prevent attacks both from external parties and by internal parties. We do in-depth third party audits that are shared with our customers. Ensuring good processes for managing risk is always more important than wishful thinking that a design pattern will make your enterprise model work. Our total lines of code for logical segmentation takes up less than 300 lines in all of our services combined. Finally, given that Postgres10+ now natively supports row level security, I fail to see why anyone starting a new project would choose the multiple database option when you can bake in authorization straight into the DB at the lowest level. Sorry this is really long, but I wanted to share my experience & thoughts.

randlet 6y ago

I run a very small SaaS and do this. Each customer gets its own isolated AWS account all organized under a root account for consolidated billing. It works great and has allowed me to develop client specific features where required (using git branches). Customers like the fact that their data is completely isolated from everyone else. Rolling out updates is simple, I just run the same deploy script for each client (after scheduling with the client if need be) and if a deploy goes wrong, you'll usually notice it after it affects one customer rather than all of them. The main downside for me is cost. The cost per customer scales linearly which hurts as you don't get the same economies of scale like you would in single tenant solutions.

gomox 6y ago
I don't know your application enough to comment, but that seems like an extreme version of what I'm proposing, which would be to share application and DB servers between customers, but not application code and databases (i.e. you can still run 3 customers on the same EC2 and RDS instances, they just have separate application code and separate database schemas). One of the main benefits, like you describe, is lowered costs. Running an AWS account for each customer is pretty significant overhead and would only be worth it to me if each customer was quite substantial (50K/yr or above?).
- randlet 6y ago
  The way you are thinking would be much cheaper than what I'm doing with the downside that clients data exists on the same server (but maybe they won't care) and the fact that a performance issue for one client can impact other clients. I haven't found an AWS account per customer to be to bad to manage yet and makes keeping track of your AWS spend per client simple. It's also trivial to run your resources in say Sydney for your Australian clients and London for your European clients.
  - gomox 6y ago
    If it works, then it works :) Company B I mentioned above still runs location specific infrastructure, but each one of these clusters can host several customers. Of course there is some overhead (i.e. very new regions sometimes operate at a loss if there is not a critical mass of customer in that cluster, but they are subsidized by more profitable regions until they reach a critical mass of customers to pay for the whole cluster costs).

juped 6y ago

I think the standard term is "tenant" for that organizational level, no?

bearjaws 6y ago

Your biggest advantage is actually a disadvantage, you have literally enabled your individual customers to boss you around and fork your codebase into a mess of different functionality per customer. Are you a contract shop where you get SOWs to expand your software? This seems like a terrible idea for both product team and the development team, who now need to know what bastardized version of the software each customer is running. What happens when a customer goes "I don't want that update"? Unless you can guarantee that each customer ends up on the same software quickly, this will get messy.

gomox 6y ago
Flexible and disciplined is a good alternative to rigid, and for young software business, probably the safer one!