4 ms·
At several places, the biggest challenge to engineering velocity and the ability to innovate is that models are joined in interesting ways that prevent them fro
by sethammons 3y ago
At several places, the biggest challenge to engineering velocity and the ability to innovate is that models are joined in interesting ways that prevent them from being decoupled. A user model tied to the package model preventing independent scaling of either. Every module reaching into all the tables. Joins, joins, and joins. Vast scans. The queries become inefficient and even the mighty postgres slows down at 2k rps. It is just too tempting to reach into another team's datastore. Now the two teams are in lockstep and can't alter their own datastore because others have assumptions on how it is stored and they access it directly.
At SendGrid, we had an few tables to deal with IP management. Over time, the like 5 tables were in use by 15+ services owned by different teams with competing priorities. When we finally had to scale the database, it took over three quarters of working with other teams while we supported both legacy and the new hotness.
It is hard to see early on where whole teams can sprout up and have domain ownership and to know which areas will be common/platform-like for other teams. But as soon as two services share a table, you should see a train coming at you.
- dgroshev 3y agoThere is severe hindsight bias in play here. How many tables weren't problematic despite all those joins and just chugged along scaling fine? How many startups were bogged down by unnecessary complexity or trying to keep a semblance of consistency instead of shipping features and just didn't reach the scale when performance becomes a problem? Joins are good, actually, and normalisation should be the default. It's much easier to keep things consistent by default rather than trying to make them somewhat consistent afterwards. When something becomes a problem there is a bunch of tools (from denormalisation or materialised views to specialised datastores) to deal with that, but before that it's just premature optimisation.
- deleted 3y ago[deleted]