5 ms·
The kitchen sink database used by everybody is such a common problem, yet it is repeated over and over again. If you grow it becomes significant tech debt and a
by Scubabear68 3y ago
The kitchen sink database used by everybody is such a common problem, yet it is repeated over and over again. If you grow it becomes significant tech debt and a performance bottleneck.
Fortunately, with managed DBs like RDS it is really easy to run individual DB clusters per major app.
- sgarland 3y agoThe downside is then you have many, many DBs to fight with, to monitor, to tune, etc. This is rarely a problem when things are small, but as they grow, the bad schema decisions made by empowering DBA-less teams to run their own infra become glaringly obvious.
- Scubabear68 3y agoNot a downside to me. Each team maintains their own DB and pays for their own choices. In the kitchen sink model all teams are tied together for performance and scalability, and some bad apple applications can ruin the party for everyone. Seen this countless times doing due diligence on startups. The universal kitchen sink DB is almost always one of the major tech debt items.
- sgarland 3y agoI’m a DBRE, which means it’s somehow always my fault until proven otherwise. And even then, it’s usually on me to work around the insane schema dreamt up by the devs. Multi-tenant DBs can work fine as long as every app has its own users, everyone goes through a connection pooler / load balancer, and every user has rate limits. You want to write shitty queries that time out? Not my problem. Your GraphQL BFF bullshit is trying to make 10,000 QPS? Nope, sorry, try again later. EDIT: I say “not my problem,” but as mentioned, it inevitably becomes my problem. Because “just unblock them so the site is functional” is far more attractive to the C-Suite than “slow down velocity to ensure the dev teams are doing things right.”
- CoolCold 3y agoYou forgot the modern mantra - dev team is always right!
- Scubabear68 3y agoI agree. My gripe was everybody in the same schema with a global “app” user.
- dalyons 3y agoOr, you just avoid doing multi tenet from the start and none of those become your problem to unblock. What’s the downside?
- sgarland 3y agoDone that as well; it still becomes my problem because teams without RDBMS knowledge eventually break it, and… then I get paged. Full Stack is a lie, and the sooner companies accept that and allow people to specialize again, and to pay for the extra headcount, the better off everyone will be.
- dalyons 3y agoI disagree I guess. Multiple companies I’ve worked at have broken up their shared db into many dbs that individual teams own the operations of, and it works just fine. At significant scale in traffic and # of eng. No central dbas needed - smaller databases require much less skills to manage. The teams that own them learn enough.
- maccard 3y ago> Not a downside to me. Each team maintains their own DB and pays for their own choices. This is how you end up with the infamous "jira and confluence have two different markdown flavors" issue.
- vrosas 3y agoBad schema decisions are made regardless of whether you’re one database or 50. At least with many databases the problems are localized.
- sgarland 3y agoBut then the DB Team – if you have one – is responsible for 50 databases, each full of their own unique problems. This will undoubtedly go over poorly, but honestly I think every data decision should be gated through the DB Team (again, if you have them). Your proposed schema isn’t normalized? Straight to jail. You don’t want to learn SQL? Also straight to jail. You want to use a UUIDv4 as a primary key? Believe it or not, jail. The most performant and referentially sound app in the world, because of jail.
- inquist 3y agoWhat’s wrong with uuidv4 as PK?
- marcosdumay 3y agoSerial integers always work better than any uuid as PKs, but the thing with uuid4 is that it disrupts any kind of index or physical ordering you decide to put on your data. Uuids are really for external communication, not in-system organization.
- calvinmorrison 3y agoIt's because I hate databases and programming separately. I would rather slow code then have to dig into some database procdure. Its just another level of separation thats too mentally hard to manage. Its like... my queries go into a VM and now I have to worry about how the VM is performing. I wish and maybe there is a programming language with first class database support. I mean really first class not just let me run queries but almost like embedded into the language in a primal way where I can both deal with my database programming fancyness and my general development together. Sincerely someone who inherited a project from a DBA.
- sgarland 3y ago> I mean really first class not just let me run queries but almost like embedded into the language Not quite embedded into the OS, but Django is a damn good ORM. I say that as a DBRE, and someone obsessed with performance (inherent issues with interpreted languages aside).
- leetharris 3y agoThe closest thing to what you're describing is Prisma in Node. It generates a Typescript file from your schema so you get code completion on your data. And it exists somewhere between a query builder and a traditional ORM. I have worked in many languages with many ORMs and this has been my personal favorite.
- sgarland 3y agoUntil Prisma can manage JOINs [0] there is no way I can recommend it. [0]: https://github.com/prisma/prisma/discussions/12715 https://github.com/prisma/prisma/discussions/12715
- kkarimi 3y agoThe support for JOINs is coming, currently under a feature flag [0] [0]: https://github.com/prisma/prisma/issues/5184#issuecomment-1843505785 https://github.com/prisma/prisma/issues/5184#issuecomment-18...
- el_benhameen 3y agoLots of interesting comments on this one. Anyone have any good resources for learning how not to fuck up schema/db design for those of us who will probably never have a DBA on the team?
- magicalhippo 3y agoGood question. We don't have a DBA either. I've learned SQL as needed and while I'm not terrible, it's still daunting when making the schema for a new module that might require 10-20 tables or more. One thing that has worked well for us is to alway include the top-most parent key in all child tables down yhe hierarchy. This way we can load all the data for say an order without joins/exists. Oh and never use natural keys. Each time I thought finally I had a good use-case, it has bitten me in some way. Apart from that we just try to think about the required data access and the queries needed. Main thing is that all queries should go against indexes in our case, so we make sure the schema supports that easily. Requires some educated guesses at times but mostly it's predictable IME. Anyway would love to see a proper resource. We've made some mistakes but I'm sure there's more to learn.
- AznHisoka 3y agoNot to pick on you, but is SQL not basic knowledge for every software engineer these days? Or have times changed?
- rswail 3y agoTimes have changed. If you have C# programmers and they can't do it in Entity Framework/LINQ, then they can't do it.
- neonsunset 3y agoThis seems like a stereotype from 2010s and disconnected from reality today.
- 3y ago
- nitwit005 3y agoThe moment you have two databases is the moment you need to deal with data consistency problems. If you can't do something like determine if you can delete data, as the article mentions, you won't be able to produce an answer to how to deal with those problems.
- eduction 3y agoManagement problem masquerading as a tech problem. Being shared between applications is literally what databases were invented to do. That’s why you learn a special dsl to query and update them instead of just doing it in the same language as your application. The problem is that data is a shared resource. The database is where multiple groups in an organization come together to get something they all need. So it needs to be managed. It could be a dictator DBA or a set of rules designed in meetings and administered by ops, or whatever. But imagine it was money. Different divisions produce and consume money just like data. Would anyone imagine suggesting either every team has their own bank account or total unfettered access to the corporate treasury? Of course not. You would make a system. Everyone would at least mildly hate it. That’s how databases should generally be managed once the company is any real size.
- dalyons 3y agoWhy would you make it a shared resource if you don’t have to? Decades of experience have shown us the massive costs of doing so - the crippled velocity and soul crushing agony of dba change control teams, the overhead salary of database priests, the arcane performance nightmares, the nuclear blast radius, the fundamental organizational counter-incentives of a shared resource . Why on earth would we choose to pay those terrible prices in this day and age, when infrastructure is code, managed databases are everywhere and every team can have their own thing. You didn’t have a choice previously, now you do.
- eduction 3y agoYou wouldn’t but in any decent sized organization you will have to. If it is an organization that needs to exist there will be some common set of critical data.
- webo 3y agoIn my experience, isolated (repeated) data storage paradigm is even more common at large organizations. They share data via services, ETLs, event buses, etc.
- 3y ago