6 ms·
> The answer is to start siloing customers or application concerns out into separate clusters depending on your operating model. This is such an underrated sol
by Ozzie_osman 5y ago
> The answer is to start siloing customers or application concerns out into separate clusters depending on your operating model.
This is such an underrated solution (siloing or sharding your data in some way). I think people don't do it because:
1. The tooling doesn't make it super-easy (e.g. good luck sharding Postgres unless you're willing to pay for Citus)
2. "Trendy" companies in the past decade have been network-type products (social networks, etc), where the structure of the data makes it much harder to silo (people need to follow each other, interact with each other's content, etc)
3. We as an industry took a several-year detour over to NoSQL land as a promised solution to scalability.
Life would be a lot easier if you could say something like:
* I want a Postgres node.
* I'm happy to shard my data by some key (customerId, city, etc) and am willing to accept responsibility for thinking through that sharding key.
* My application has some logic that easily knows which DB to read/write from depending on shard key.
* There's some small amount of "global application data" that might need to live on a single node.
- simonw 5y agoFor your "life would be easier" version, isn't that pretty much what Citus does? And Citus is open source so paying for the managed version is optional.
- cellis 5y agosaying "Citus is open source" doesn't mean its as simple as running `yarn install citus; yarn run citus`. Its is much much more complicated than typical open source. You pay for support for a very good reason.
- rad_gruchalski 5y ago> The tooling doesn't make it super-easy (e.g. good luck sharding Postgres unless you're willing to pay for Citus) There’s YugabyteDB.
- seanhunter 5y agoSharding by customer has a lot of benefits for b2b applications also. You can run each customer in a seperate instance in their own vpc if you want to, and guarantee to your customers that their data is never commingled with another customer. This makes a lot of sense if you have relatively low number relatively high value customers. It radically reduces the infosec/screw up blast radius. For example it means however badly you screw up it is just about impossible to accidentally show data from customer A to a user from customer B. You can let customers specify region/availability zone and bring their own keys for encryption making lots of enterprise security compliance things easier (for regulated use cases). There is no one true way of scaling that suits all possible use cases and as engineers, we should think for ourselves and build a solution that works well in our situation, rather than thinking you have to do a certain thing because that's what google/facebook etc do. Their scaling problems are most likely really different to yours.
- hinkley 5y agoOn paper I’m not dealing with the same problem as the people in this thread, but I can’t help thinking that the root causes are the same. I have a team that won’t even look at the fact that we measure performance by response time, but we are lumping together write traffic and two classes of read traffic into a single monolithic chunk of code. We are SaaS where our customers write and their customers read. And of course search queries are by far the slowest traffic. But there’s some sort of mental block about splitting things up that is only slowly changing, and our app is so “flexible” that a load balancer would struggle to tell which urls involve search functionality. I think people just want to “know where to look” but if your code is on a cluster there already is no “there”. You’re probably looking at some log aggregator anyway, so who gives a tinker’s damn if they come from separate log files on the same box or the same log file?