4 ms·

> And then, when you start growing, that's when you break out an independently scalable part of the system into a microservice Respectfully disagree. At this p

> And then, when you start growing, that's when you break out an independently scalable part of the system into a microservice Respectfully disagree. At this point you start refactoring your monolith into components and actually look at performance measurements via tracing. Do not do microservices when you're growing (or ever, most of the time). And by the love of god, don't split your data. Data is much more complex to manage than code.

Aeolun 3y ago
> And by the love of god, don't split your data. I think people feel that if they introduce a new data store, they won’t have to deal with the existing nearly unusable massive data store. Of course that just exacerbates the problem.
ngc248 3y ago
There is a way to splitting data. The service which owns the data always does the writes and others who need reads on that data can store replicas. ofc the complication then will be in replicating data, but this will enable services to massively scale and eliminate SPOFs
- code_biologist 3y ago
  The other part of this pattern you mention (replicating data) seems solved in many cases by the data warehouse patterns of the last few years. Stuff it all in Snowflake or BigQuery then readers can query as they see fit. Query engines like Trino can paper over data storage heterogeneity / breakdowns in centralization. I'm not a fan of the "lakehouse" terminology, but it is the thing. There are downsides to coupling more loosely with data consumers, but it keeps service owners moving without wasting time vending data.
- amne 3y ago
  As long as you're not replicating the data model also which means you're now replicating the business logic that go along with the data model. An ETL approach from the start will save you a lot of headache.
theshrike79 3y ago
Splitting data is just a hardcore way of splitting the responsibility for the data. You can't have 42 different classes directly poking the User-table for example. You need one clear location that has the responsibility for the data. If you move the User-table to a different database schema, other places CANNOT touch it because they won't have access to it =)
- aledalgrande 3y ago
  And then the order table will need to set user ids as a foreign key and you will have two sources of truth lol Will be fun when you decide that you want to have organizations to be linked to orders instead of users. All this to say, data management and having more than one source of truth complicates your world a lot. The responsibility for writing the data has to be solved at the appropriate layer: only one entity writing it.
  - rewmie 3y ago
    > And then the order table will need to set user ids as a foreign key and you will have two sources of truth lol Why should your order manager service be tightly-coupled to your IAM service when the only responsibility it has is to store a single ID?
  - fragmede 3y ago
    > And then the order table will need to set user ids as a foreign key and you will have two sources of truth You might be databasing wrong if that results in two sources of truth.
Copenjin 3y ago
> And by the love of god, don't split your data. Most data problems can be fixed on the frontend with a few relatively simple graphql queries. /s
- renegade-otter 3y ago
  You are trying to get me going, aren't you?
sethammons 3y ago
At several places, the biggest challenge to engineering velocity and the ability to innovate is that models are joined in interesting ways that prevent them from being decoupled. A user model tied to the package model preventing independent scaling of either. Every module reaching into all the tables. Joins, joins, and joins. Vast scans. The queries become inefficient and even the mighty postgres slows down at 2k rps. It is just too tempting to reach into another team's datastore. Now the two teams are in lockstep and can't alter their own datastore because others have assumptions on how it is stored and they access it directly. At SendGrid, we had an few tables to deal with IP management. Over time, the like 5 tables were in use by 15+ services owned by different teams with competing priorities. When we finally had to scale the database, it took over three quarters of working with other teams while we supported both legacy and the new hotness. It is hard to see early on where whole teams can sprout up and have domain ownership and to know which areas will be common/platform-like for other teams. But as soon as two services share a table, you should see a train coming at you.
- dgroshev 3y ago
  There is severe hindsight bias in play here. How many tables weren't problematic despite all those joins and just chugged along scaling fine? How many startups were bogged down by unnecessary complexity or trying to keep a semblance of consistency instead of shipping features and just didn't reach the scale when performance becomes a problem? Joins are good, actually, and normalisation should be the default. It's much easier to keep things consistent by default rather than trying to make them somewhat consistent afterwards. When something becomes a problem there is a bunch of tools (from denormalisation or materialised views to specialised datastores) to deal with that, but before that it's just premature optimisation.
- deleted 3y ago
  [deleted]
rewmie 3y ago
> Respectfully disagree. At this point you start refactoring your monolith into components and actually look at performance measurements via tracing. No matter how hard you modularize and instrument your service, that won't give you regional deployments, which you absolutely need to have ti be able to drive down customer-facing latencies from many hundreds of milliseconds to a few tens of milliseconds. How do you pull that off with a monolith when you have a global userbase?
- rokkitmensch 3y ago
  Do the next simplest thing: shard geographically. If you really need to handle users transiting geos, that's still dramatically simpler than meegroserbusses.
  - aledalgrande 3y ago
    Sharding is part of how we do it.
deleted 3y ago
[deleted]

Aeolun 3y ago

> And by the love of god, don't split your data. I think people feel that if they introduce a new data store, they won’t have to deal with the existing nearly unusable massive data store. Of course that just exacerbates the problem.

ngc248 3y ago

There is a way to splitting data. The service which owns the data always does the writes and others who need reads on that data can store replicas. ofc the complication then will be in replicating data, but this will enable services to massively scale and eliminate SPOFs

code_biologist 3y ago
The other part of this pattern you mention (replicating data) seems solved in many cases by the data warehouse patterns of the last few years. Stuff it all in Snowflake or BigQuery then readers can query as they see fit. Query engines like Trino can paper over data storage heterogeneity / breakdowns in centralization. I'm not a fan of the "lakehouse" terminology, but it is the thing. There are downsides to coupling more loosely with data consumers, but it keeps service owners moving without wasting time vending data.
amne 3y ago
As long as you're not replicating the data model also which means you're now replicating the business logic that go along with the data model. An ETL approach from the start will save you a lot of headache.

theshrike79 3y ago

Splitting data is just a hardcore way of splitting the responsibility for the data. You can't have 42 different classes directly poking the User-table for example. You need one clear location that has the responsibility for the data. If you move the User-table to a different database schema, other places CANNOT touch it because they won't have access to it =)

aledalgrande 3y ago
And then the order table will need to set user ids as a foreign key and you will have two sources of truth lol Will be fun when you decide that you want to have organizations to be linked to orders instead of users. All this to say, data management and having more than one source of truth complicates your world a lot. The responsibility for writing the data has to be solved at the appropriate layer: only one entity writing it.
- rewmie 3y ago
  > And then the order table will need to set user ids as a foreign key and you will have two sources of truth lol Why should your order manager service be tightly-coupled to your IAM service when the only responsibility it has is to store a single ID?
- fragmede 3y ago
  > And then the order table will need to set user ids as a foreign key and you will have two sources of truth You might be databasing wrong if that results in two sources of truth.

Copenjin 3y ago

> And by the love of god, don't split your data. Most data problems can be fixed on the frontend with a few relatively simple graphql queries. /s

renegade-otter 3y ago
You are trying to get me going, aren't you?

sethammons 3y ago

At several places, the biggest challenge to engineering velocity and the ability to innovate is that models are joined in interesting ways that prevent them from being decoupled. A user model tied to the package model preventing independent scaling of either. Every module reaching into all the tables. Joins, joins, and joins. Vast scans. The queries become inefficient and even the mighty postgres slows down at 2k rps. It is just too tempting to reach into another team's datastore. Now the two teams are in lockstep and can't alter their own datastore because others have assumptions on how it is stored and they access it directly. At SendGrid, we had an few tables to deal with IP management. Over time, the like 5 tables were in use by 15+ services owned by different teams with competing priorities. When we finally had to scale the database, it took over three quarters of working with other teams while we supported both legacy and the new hotness. It is hard to see early on where whole teams can sprout up and have domain ownership and to know which areas will be common/platform-like for other teams. But as soon as two services share a table, you should see a train coming at you.

dgroshev 3y ago
There is severe hindsight bias in play here. How many tables weren't problematic despite all those joins and just chugged along scaling fine? How many startups were bogged down by unnecessary complexity or trying to keep a semblance of consistency instead of shipping features and just didn't reach the scale when performance becomes a problem? Joins are good, actually, and normalisation should be the default. It's much easier to keep things consistent by default rather than trying to make them somewhat consistent afterwards. When something becomes a problem there is a bunch of tools (from denormalisation or materialised views to specialised datastores) to deal with that, but before that it's just premature optimisation.
deleted 3y ago
[deleted]

rewmie 3y ago

> Respectfully disagree. At this point you start refactoring your monolith into components and actually look at performance measurements via tracing. No matter how hard you modularize and instrument your service, that won't give you regional deployments, which you absolutely need to have ti be able to drive down customer-facing latencies from many hundreds of milliseconds to a few tens of milliseconds. How do you pull that off with a monolith when you have a global userbase?

rokkitmensch 3y ago
Do the next simplest thing: shard geographically. If you really need to handle users transiting geos, that's still dramatically simpler than meegroserbusses.
- aledalgrande 3y ago
  Sharding is part of how we do it.

deleted 3y ago

[deleted]