6 ms·

> No single data model can support all use cases. In theory, there is no domain (or finite set of domains) that cannot be accurately modeled using tuples of th

> No single data model can support all use cases. In theory, there is no domain (or finite set of domains) that cannot be accurately modeled using tuples of things and their relations. Practically speaking, the scope of a given database/schema is generally restricted to one business or problem area, but even this doesn't matter as long as the types aren't aliasing inappropriately. You could put a web retailer and an insurance company in the same schema and it would totally work if you are careful with naming things. Putting everything into exactly one database is a superpower. The #1 reason I push for this is to avoid the need to conduct distributed transactions across multiple datastores. If all business happens in one transactional system, your semantics are dramatically simplified.

baq 3y ago
This this this. > Putting everything into exactly one database is a superpower. Especially this. $1M big iron DB server is much cheaper than a redundant array of inexpensive databases when people come to actually use the data; be it developers, analysts or leadership, everybody saves time, except perhaps a few DBAs.
- riku_iki 3y ago
  > $1M big iron DB server is much cheaper than a redundant array of inexpensive databases but you need some redundancy to avoid service downtime if that server/rack/datacenter will have a failure.
- bunderbunder 3y ago
  It can be, but I've discovered that in tech it's common to assume that your employees' time costs $0, for some reason. I couldn't even begin to count the number of times I've had to walk a manager back from deciding that we should spend a person-month of engineering time to develop an in-house solution that lets us avoid a $500/year licensing fee. I see this a lot in RDBMSes. I've used plenty of 'em, and I have nothing against projects like PostgreSQL; they're amazing, especially for the price. But computers cost money, and maintenance time costs money, and development time costs money, and I know from experience that I can squeeze a lot more performance out of the same hardware using MSSQL than I can PostgreSQL, and I'll have an easier time doing it, too. I can even point to some of the specific design and feature differences that lead to the discrepancy. It's to the point where, whenever I see an article talking about the limitations of relational databases, I automatically mentally insert a "that you can get for free" into the sentence.
- asah 3y ago
  well I dunno, we've only tested this architecture for 60 years and a few million companies... in all seriousness, putting the data in one place also makes it a lot easier to optimize and to amortize the optimization efforts.
  - alexchamberlain 3y ago
    Is there any medium size company that keeps all its data in 1 place?
    - fiddlerwoaroof 3y ago
      It’s still fairly common for the majority of critical data to be in a single database. Even if microservice initiatives have moved some data out.
    - Spivak 3y ago
      I've worked at two of them -- all data, sans pointers to content addressed immutable object storage, is stored in a single database. It worked well for us; it's not rainbows and unicorns wow all of our problems disappeared but we got what we needed with sufficient performance for an app that was not well optimized. There were challenges like the difficulty of working with a few huge tables that existed since the company's inception and were practically infeasible to do any schema changes on, but that's more a modeling issue. It was also a single point of failure service. If the db goes down so to does everything but constant sense of impending doom it was less of a problem than I thought with ndb.
      - vmfunction 3y ago
        It is good to see people on HN are supporting this single db approach. And with development such OrioleDB, it would solve the horizontal scaling. And all the access to data are native.
      - hasty_pudding 3y ago
        Y'all couldn't have some read-only clusters that could just take over as the master?
- opportune 3y ago
  As someone bootstrapping a business, my cost structure favors horizontally scaling databases with low/no fixed overhead vs vertically scaling databases with high overhead. And I don’t think this is exactly an uncommon situation - plenty of people, even within large well-funded organizations, would have a hard time justifying such a large expenditure if it were uncertain that it’d be fully utilized. It’s also a lot easier for me to handle tenancy/isolation/different data “realms” this way. And I sleep better at night knowing a mistake (I have no dbas, and I don’t have time to invest in tons of safeguards as a solo dev when there are much higher EV tasks) won’t nuke absolutely everything.
  - smt88 3y ago
    For small companies, I've never started with horizontal scaling. With vertical scaling and maybe 2-3 instances, I don't have to worry about hasty ORM code that generates suboptimal queries. If I have horizontal scaling, sometimes even a reasonable query that's missing an explicit index can turn into a 30s query that I have to spend dev time debugging.
    - kstrauser 3y ago
      I wish I could upvote this more. 95+% of startups will never need to scale past one reasonably sized single-instance database. Maybe they'll want to have a couple of replicas running for a fast failover, but from a performance perspective, they should spend their time writing efficient queries instead of horizontally scaling. It's good to think about the what-ifs along the way. Can we shard this setup if we land a huge customer, or one that's hyper-sensitive about having their data separated? If/when that time comes, you'll have ideas of what to do about it. But realistically, most companies never hit that point. Every hour spent worrying how to make their product FAANG-scale is an hour spent not making their product.
      - pi-e-sigma 3y ago
        I think most even semi-experienced developers know this. But doing the right thing is against their career grow goals so they overcomplicate the architecture on purpose.
- doctorpangloss 3y ago
  You can never have 1 database. Something operates the database. That thing is going to need a database. Postgres and etcd are very different. SQLite is very different. Look at this microk8s thread - https://github.com/canonical/microk8s/issues/3227 https://github.com/canonical/microk8s/issues/3227 - as a canonical example. Context is they use dqlite, which is multi-master SQLite with raft. Here people come to actually use the data and it’s a disaster. Just use etcd! Then people will deploy Postgres on top of Kubernetes. Great. Maybe they will deploy it differently, and their database is a bunch of files in /etc/. Whatever. But you can’t run Kubernetes on Postgres, you wouldn’t want to anyway, the choice to use etcd is very thoughtful. This is an environment designed to simplify the layers of bootstrapping, Kubernetes’s superpower is that it runs as much of its own stuff on top of itself as safe as it is to do so. It still needs a database that is going to be separate from your applications, and unfortunately, for all interesting applications, the line between deployment and business-logical concerns is blurred. So IMO, you will have multiple databases, because logical and deployment concerns are intertwined; deployment concerns need databases; and thus to solve chicken and egg, you’ll have at least 2, hopefully the 1st is super easy to administer. Perhaps you disagree. Maybe you should use dqlite as the only database. It’s so naive to think anything is CP as long as there’s consensus in front of it. And if there’s no strong opinion about the arcane issue I just said, worse if it’s meaningless gobbledygook; or if the strong opinion is “use Postgres,” the data is screwed and the users are screwed.
- tomxor 3y ago
  It's also entirely practical for everything that's not twitter-like, i.e almost everything. But the most noise comes from people with the (admittedly more interesting) use case requirements at the data store level. So long as your application doesn't throw space and performance away at every opportunity, vertical scaling goes really far these days.
nathanmarz 3y ago
Being able to model a use case with tuples and relations does not mean the database can meet the performance requirements of that use case. If it can't meet the performance requirements, then the use case is unsupported. It's the same way how no single data structure or combination of data structures can support all regular programming use cases. Sometimes you need a map, sometimes you need a list, sometimes you need a set, sometimes you need a combination, and sometimes you need something completely different.
- nineteen999 3y ago
  Sometimes you just need a DBA to help you optimise your DB for your workload and future growth. In general, developers make shitty DBA's and vice versa.
  - sharadov 3y ago
    As a DBA, who has managed enough monolithic databases at mid and large sized organizations, there are enough safeguards in today's date to avoid the scenario you described - backups, read replicas, replication to avoid the need for unnecessary distributed databases.
    - nineteen999 3y ago
      I suspect you responded to the wrong parent.
  - Foobar8568 3y ago
    Performance tuning is not a skillset enterprise DBAs usually have.
    - tomnipotent 3y ago
      Every DBA I've worked with has had performance tuning as one of their top skills, both at the installation-level and query-level. Sometimes it's optimizing a query using some hard-earned knowledge of the RDBMS, and sometimes it's keeping logs and data on separate storage.
lupusreal 3y ago
> In theory, there is no domain (or finite set of domains) that cannot be accurately modeled using tuples of things and their relations. That seems right to me, but is it really provably true?
- Scarblac 3y ago
  At least, if it can be modelled by some type of finite database stored on disk or in memory, then you can store the entire contents of that database as one binary blob. Now you just need one tuple ("our_system", "the_db", the_binary_blob) QED :)
- bob1029 3y ago
  Provably? I'll leave that up to the academics. Practically? I've never had someone throw a problem at me I couldn't model in SQL. Not saying I can guarantee performance, but in terms of abstract modeling I've never encountered something that can't be done "clean". I'd like to use the analogy of painting. What can't you paint on that canvas that isn't trivially-hindered by the available media? Can you actually describe that space of things? In my estimation, premature optimization is why most people can't work in these terms. You wind up with a tuple like WorldObjects and immediately go off the deep end about a billion row scale before you are done naming things.
- Legend2440 3y ago
  No. Many real domains like natural language are too complex and fuzzy to accurately model with tuples. To be truly universal, your model needs to be computationally stronger - it needs to be turing complete. For example fractals could never be modeled by any number of tuples, but can be perfectly represented by very short computer programs.
  - Mikhail_Edoshin 3y ago
    Can a program be modeled by a finite number of tuples?
    - bostik 3y ago
wolverine876 3y ago
It's a dream, a truly beautiful one, but it never works (does it? has anyone pulled it off?). Taking it to an extreme, the entire world needs one integrated database and user authorizations. Obviously data structures vary, performance is required, and it becomes a bottleneck because it's so critical that mortals can't touch it and every change must be ultra-safe. And then there's security: what if someone finds a way from the development bug section to the HR or executive-only financials sections? Generally, anyone who has tried to implement universal ERM systems knows how difficult and painful integrated systems can be. But those are the extremes. I'd be interested to know how far people have gotten, and how they did it, pursuing this ideal? I've never seen a business run on one. How about a personal knowledge-management system? Does everything fit? Do you still use spreadsheets for something 'quick', text files for free-form, etc.?
- Kinrany 3y ago
  I once toyed with the idea of viewing all the available compute and storage in the world as one big decentralized computer, with multi-tenancy and auth baked into the OS.
xnx 3y ago
> Putting everything into exactly one database is a superpower. T Amen. Previous co. had a cargo-cult microservices setup including a separate DB for each app. This made things unnecessarily complicated and expensive for no business benefit (definite resume-padding benefit). Lesson: Don't complicate things until you're forced to.
- adra 3y ago
  Hard disagree. You don't need different physical databases (though it may become necessary at some point), but you should absolutely value data isolation when you're developing smaller services. Oh you need to rename a column and 6 services call in to read it write this table? You took a 1x solution and made it 5x more difficult to make this change and coordinate, etc. If instead 5 services asked the 1 holder of truth service about the data instead, all of 1 service needs to change when you update the database schema. You can then pick and choose when each dependent service is moved over to the new schema's representation without blocking all services from a simple upgrade path . If on the other hand, you're fine yolo bringing down your entire tech stack for potentially hours force upgrading all services at once, then you probably shouldn't be using microservices to begin with. (And yes there are strategies to mitigate needing a big bang for renames as well as long as writers are all consistent, I just wanted to describe why everyone touching the same data stores leads to terrible outcomes like people being unable to make changes because "it's hard", which nobody says out loud but it's definitely visible in the tech stack).
- zerbinxx 3y ago
  The thing with the microservice + do separation problem is that by the time you’re forced to do it, the cost to migrate and split data out of a single source into many (potentially all at once) is extremely great. Last at work week a trio of services which rely on the same physical db had a problem where they were both locked out because of a problem with the logic in one of them. This caused an outage that lasted 5 hours and resulted in a (possible) loss of several months worth of their dev team’s funding. At a certain scale you try to proactively lessen the impact of possible events like these because the math works out. If you’re at that scale, then great, do that. But I also fully agree that cargo-culting that kind of decision out of the gate can be a massive waste of money.
NikolaNovak 3y ago
I work in ERP space, which is the ultimate uncool from typical HN perspective; but sometimes "uncool" has a Venn diagram overlap with "mature". A modern top tier RDBMS is an incredibly powerful, fast, mature, stable tool - when wielded wisely by the knowledgeable wizard. To be fair, the developers must have a solid understanding of programming language, and business domain, and SQL / data dictionary / ERD - which I've always taken for granted but eventually realized is not the norm.
- mamcx 3y ago
  I also work on this space, and kind of see the value of this idea. Having a log store make sense, and having a super-VIEW replacement that can re-compute the data (so I can denormalize o build secondary tables/indexes that could or not be stored) is a common pattern. That is fine. What is NOT fine, and that normal DBS avoid, is to add "eventual consistency". I NEED to get exact counts, sum, avg, and RIGTH NOW, not a second later. The more "eventual" the data become the worse things are, and that is the major way to add bugs and problems. One of the reasons FK are important is they eliminate the chance of getting the relationships wrong, but also, to get it right NOW.
samaysharma 3y ago
Agreed. Putting data for all your use cases in a single database is a huge deal. Having different databases for each part of the application bloats so much so easily from a cost, complexity and skill set standpoint. Also, makes it extremely hard to debug issues when you have to chart the path of data through 5 different tools. There is the challenge of workload separation and scaling each component separately but that can be resolved by pulling out challenging workloads into their own "database" albeit on the same stack.
refset 3y ago
> If all business happens in one transactional system, your semantics are dramatically simplified. 100% agreed. One of the biggest issues SQL databases have faced is that the scope & scale of "one transactional system" has evolved a lot more quickly than any off-the-shelf database architecture has been able to keep up with, resulting in an explosion of workaround technologies and approaches that really shouldn't need to exist. We're now firmly in the cloud database era and can look at things like Spanner to judge how far away Postgres remains from satisfying modern availability & scaling expectations. But it will be great to see OSS close the gap (hopefully soon!).
- ddorian43 3y ago
  YugabyteDB has PostgreSQL on top of Spanner architecture and is apache2 licensed (I work there).
  - refset 3y ago
    "YugabyteDB Anywhere" is open core though right? Half the value & complexity is in the orchestration stack. It's definitely a step forward, but such licensing is likely still too restrictive to ever supplant Postgres itself (it terms of breadth of adoption).
    - ddorian43 3y ago
      > "YugabyteDB Anywhere" is open core though right? No, it's all open source. YugabyteDB Anywhere is just automation & 24/7 core engineering support. All db features are open source. You can re-create Anywhere without needing to touch c/c++. Note: All features have always been open source (besides incremental backup which will be soon).
0xbadcafebee 3y ago
Big databases are not new. If that was all that was needed, people would have been doing just that for the past 40 years. Turns out it doesn't always work, and sometimes (often?) it's terrible. Even if theoretically it was all you ever needed, the other constant problem is the implementation. Most developers, today anyway, are morons. They don't understand how databases work or how to use them, so they use them poorly. In order to get away from that fact, they invented new databases (like NoSQL) so they could use their brains less and just write more glue code. Turned out that was horrible as well. Pretty soon the tides will turn yet again and "just having a big database" will again be seen as not in vogue and another paradigm of "simplicity" will come around (instead of "one big database" being simple it'll be "many small databases" like we had with microservices). Those who don't understand history are doomed to repeat it.
karencarits 3y ago
Is there any good opensource software with gui to manage such databases/graphs? I guess Wikibase is an alternative, although not very user friendly In contrast, pocketbase [0] was discussed two days ago, and there are many user-friendly alternatives for relational data [e.g. 1-4] [0] https://news.ycombinator.com/item?id=38898934 https://news.ycombinator.com/item?id=38898934 [1] https://www.getgrist.com/ https://www.getgrist.com/ [2] https://nocodb.com/ https://nocodb.com/ [3] https://baserow.io/ https://baserow.io/ [4] https://www.rowy.io/ https://www.rowy.io/
felipellrocha 3y ago
I've heard so many people claim this, but I've never seen this put into practice in production, and at scale. I don't even think that that proves that this cannot be done... And I've seen A LOT of people try. But, things invariably start to break down, and at this point I just think that the statement above is more wishy washy than anything, though I want to believe...

baq 3y ago

This this this. > Putting everything into exactly one database is a superpower. Especially this. $1M big iron DB server is much cheaper than a redundant array of inexpensive databases when people come to actually use the data; be it developers, analysts or leadership, everybody saves time, except perhaps a few DBAs.

riku_iki 3y ago
> $1M big iron DB server is much cheaper than a redundant array of inexpensive databases but you need some redundancy to avoid service downtime if that server/rack/datacenter will have a failure.
bunderbunder 3y ago
It can be, but I've discovered that in tech it's common to assume that your employees' time costs $0, for some reason. I couldn't even begin to count the number of times I've had to walk a manager back from deciding that we should spend a person-month of engineering time to develop an in-house solution that lets us avoid a $500/year licensing fee. I see this a lot in RDBMSes. I've used plenty of 'em, and I have nothing against projects like PostgreSQL; they're amazing, especially for the price. But computers cost money, and maintenance time costs money, and development time costs money, and I know from experience that I can squeeze a lot more performance out of the same hardware using MSSQL than I can PostgreSQL, and I'll have an easier time doing it, too. I can even point to some of the specific design and feature differences that lead to the discrepancy. It's to the point where, whenever I see an article talking about the limitations of relational databases, I automatically mentally insert a "that you can get for free" into the sentence.
asah 3y ago
well I dunno, we've only tested this architecture for 60 years and a few million companies... in all seriousness, putting the data in one place also makes it a lot easier to optimize and to amortize the optimization efforts.
- alexchamberlain 3y ago
  Is there any medium size company that keeps all its data in 1 place?
  - fiddlerwoaroof 3y ago
    It’s still fairly common for the majority of critical data to be in a single database. Even if microservice initiatives have moved some data out.
  - Spivak 3y ago
    I've worked at two of them -- all data, sans pointers to content addressed immutable object storage, is stored in a single database. It worked well for us; it's not rainbows and unicorns wow all of our problems disappeared but we got what we needed with sufficient performance for an app that was not well optimized. There were challenges like the difficulty of working with a few huge tables that existed since the company's inception and were practically infeasible to do any schema changes on, but that's more a modeling issue. It was also a single point of failure service. If the db goes down so to does everything but constant sense of impending doom it was less of a problem than I thought with ndb.
    - vmfunction 3y ago
      It is good to see people on HN are supporting this single db approach. And with development such OrioleDB, it would solve the horizontal scaling. And all the access to data are native.
    - hasty_pudding 3y ago
      Y'all couldn't have some read-only clusters that could just take over as the master?
opportune 3y ago
As someone bootstrapping a business, my cost structure favors horizontally scaling databases with low/no fixed overhead vs vertically scaling databases with high overhead. And I don’t think this is exactly an uncommon situation - plenty of people, even within large well-funded organizations, would have a hard time justifying such a large expenditure if it were uncertain that it’d be fully utilized. It’s also a lot easier for me to handle tenancy/isolation/different data “realms” this way. And I sleep better at night knowing a mistake (I have no dbas, and I don’t have time to invest in tons of safeguards as a solo dev when there are much higher EV tasks) won’t nuke absolutely everything.
- smt88 3y ago
  For small companies, I've never started with horizontal scaling. With vertical scaling and maybe 2-3 instances, I don't have to worry about hasty ORM code that generates suboptimal queries. If I have horizontal scaling, sometimes even a reasonable query that's missing an explicit index can turn into a 30s query that I have to spend dev time debugging.
  - kstrauser 3y ago
    I wish I could upvote this more. 95+% of startups will never need to scale past one reasonably sized single-instance database. Maybe they'll want to have a couple of replicas running for a fast failover, but from a performance perspective, they should spend their time writing efficient queries instead of horizontally scaling. It's good to think about the what-ifs along the way. Can we shard this setup if we land a huge customer, or one that's hyper-sensitive about having their data separated? If/when that time comes, you'll have ideas of what to do about it. But realistically, most companies never hit that point. Every hour spent worrying how to make their product FAANG-scale is an hour spent not making their product.
    - pi-e-sigma 3y ago
      I think most even semi-experienced developers know this. But doing the right thing is against their career grow goals so they overcomplicate the architecture on purpose.
doctorpangloss 3y ago
You can never have 1 database. Something operates the database. That thing is going to need a database. Postgres and etcd are very different. SQLite is very different. Look at this microk8s thread - https://github.com/canonical/microk8s/issues/3227 https://github.com/canonical/microk8s/issues/3227 - as a canonical example. Context is they use dqlite, which is multi-master SQLite with raft. Here people come to actually use the data and it’s a disaster. Just use etcd! Then people will deploy Postgres on top of Kubernetes. Great. Maybe they will deploy it differently, and their database is a bunch of files in /etc/. Whatever. But you can’t run Kubernetes on Postgres, you wouldn’t want to anyway, the choice to use etcd is very thoughtful. This is an environment designed to simplify the layers of bootstrapping, Kubernetes’s superpower is that it runs as much of its own stuff on top of itself as safe as it is to do so. It still needs a database that is going to be separate from your applications, and unfortunately, for all interesting applications, the line between deployment and business-logical concerns is blurred. So IMO, you will have multiple databases, because logical and deployment concerns are intertwined; deployment concerns need databases; and thus to solve chicken and egg, you’ll have at least 2, hopefully the 1st is super easy to administer. Perhaps you disagree. Maybe you should use dqlite as the only database. It’s so naive to think anything is CP as long as there’s consensus in front of it. And if there’s no strong opinion about the arcane issue I just said, worse if it’s meaningless gobbledygook; or if the strong opinion is “use Postgres,” the data is screwed and the users are screwed.
tomxor 3y ago
It's also entirely practical for everything that's not twitter-like, i.e almost everything. But the most noise comes from people with the (admittedly more interesting) use case requirements at the data store level. So long as your application doesn't throw space and performance away at every opportunity, vertical scaling goes really far these days.

nathanmarz 3y ago

Being able to model a use case with tuples and relations does not mean the database can meet the performance requirements of that use case. If it can't meet the performance requirements, then the use case is unsupported. It's the same way how no single data structure or combination of data structures can support all regular programming use cases. Sometimes you need a map, sometimes you need a list, sometimes you need a set, sometimes you need a combination, and sometimes you need something completely different.

nineteen999 3y ago
Sometimes you just need a DBA to help you optimise your DB for your workload and future growth. In general, developers make shitty DBA's and vice versa.
- sharadov 3y ago
  As a DBA, who has managed enough monolithic databases at mid and large sized organizations, there are enough safeguards in today's date to avoid the scenario you described - backups, read replicas, replication to avoid the need for unnecessary distributed databases.
  - nineteen999 3y ago
    I suspect you responded to the wrong parent.
- Foobar8568 3y ago
  Performance tuning is not a skillset enterprise DBAs usually have.
  - tomnipotent 3y ago
    Every DBA I've worked with has had performance tuning as one of their top skills, both at the installation-level and query-level. Sometimes it's optimizing a query using some hard-earned knowledge of the RDBMS, and sometimes it's keeping logs and data on separate storage.

lupusreal 3y ago

> In theory, there is no domain (or finite set of domains) that cannot be accurately modeled using tuples of things and their relations. That seems right to me, but is it really provably true?

Scarblac 3y ago
At least, if it can be modelled by some type of finite database stored on disk or in memory, then you can store the entire contents of that database as one binary blob. Now you just need one tuple ("our_system", "the_db", the_binary_blob) QED :)
bob1029 3y ago
Provably? I'll leave that up to the academics. Practically? I've never had someone throw a problem at me I couldn't model in SQL. Not saying I can guarantee performance, but in terms of abstract modeling I've never encountered something that can't be done "clean". I'd like to use the analogy of painting. What can't you paint on that canvas that isn't trivially-hindered by the available media? Can you actually describe that space of things? In my estimation, premature optimization is why most people can't work in these terms. You wind up with a tuple like WorldObjects and immediately go off the deep end about a billion row scale before you are done naming things.
Legend2440 3y ago
No. Many real domains like natural language are too complex and fuzzy to accurately model with tuples. To be truly universal, your model needs to be computationally stronger - it needs to be turing complete. For example fractals could never be modeled by any number of tuples, but can be perfectly represented by very short computer programs.
- Mikhail_Edoshin 3y ago
  Can a program be modeled by a finite number of tuples?
  - bostik 3y ago

wolverine876 3y ago

It's a dream, a truly beautiful one, but it never works (does it? has anyone pulled it off?). Taking it to an extreme, the entire world needs one integrated database and user authorizations. Obviously data structures vary, performance is required, and it becomes a bottleneck because it's so critical that mortals can't touch it and every change must be ultra-safe. And then there's security: what if someone finds a way from the development bug section to the HR or executive-only financials sections? Generally, anyone who has tried to implement universal ERM systems knows how difficult and painful integrated systems can be. But those are the extremes. I'd be interested to know how far people have gotten, and how they did it, pursuing this ideal? I've never seen a business run on one. How about a personal knowledge-management system? Does everything fit? Do you still use spreadsheets for something 'quick', text files for free-form, etc.?

Kinrany 3y ago
I once toyed with the idea of viewing all the available compute and storage in the world as one big decentralized computer, with multi-tenancy and auth baked into the OS.

xnx 3y ago

> Putting everything into exactly one database is a superpower. T Amen. Previous co. had a cargo-cult microservices setup including a separate DB for each app. This made things unnecessarily complicated and expensive for no business benefit (definite resume-padding benefit). Lesson: Don't complicate things until you're forced to.

adra 3y ago
Hard disagree. You don't need different physical databases (though it may become necessary at some point), but you should absolutely value data isolation when you're developing smaller services. Oh you need to rename a column and 6 services call in to read it write this table? You took a 1x solution and made it 5x more difficult to make this change and coordinate, etc. If instead 5 services asked the 1 holder of truth service about the data instead, all of 1 service needs to change when you update the database schema. You can then pick and choose when each dependent service is moved over to the new schema's representation without blocking all services from a simple upgrade path . If on the other hand, you're fine yolo bringing down your entire tech stack for potentially hours force upgrading all services at once, then you probably shouldn't be using microservices to begin with. (And yes there are strategies to mitigate needing a big bang for renames as well as long as writers are all consistent, I just wanted to describe why everyone touching the same data stores leads to terrible outcomes like people being unable to make changes because "it's hard", which nobody says out loud but it's definitely visible in the tech stack).
zerbinxx 3y ago
The thing with the microservice + do separation problem is that by the time you’re forced to do it, the cost to migrate and split data out of a single source into many (potentially all at once) is extremely great. Last at work week a trio of services which rely on the same physical db had a problem where they were both locked out because of a problem with the logic in one of them. This caused an outage that lasted 5 hours and resulted in a (possible) loss of several months worth of their dev team’s funding. At a certain scale you try to proactively lessen the impact of possible events like these because the math works out. If you’re at that scale, then great, do that. But I also fully agree that cargo-culting that kind of decision out of the gate can be a massive waste of money.

NikolaNovak 3y ago

I work in ERP space, which is the ultimate uncool from typical HN perspective; but sometimes "uncool" has a Venn diagram overlap with "mature". A modern top tier RDBMS is an incredibly powerful, fast, mature, stable tool - when wielded wisely by the knowledgeable wizard. To be fair, the developers must have a solid understanding of programming language, and business domain, and SQL / data dictionary / ERD - which I've always taken for granted but eventually realized is not the norm.

mamcx 3y ago
I also work on this space, and kind of see the value of this idea. Having a log store make sense, and having a super-VIEW replacement that can re-compute the data (so I can denormalize o build secondary tables/indexes that could or not be stored) is a common pattern. That is fine. What is NOT fine, and that normal DBS avoid, is to add "eventual consistency". I NEED to get exact counts, sum, avg, and RIGTH NOW, not a second later. The more "eventual" the data become the worse things are, and that is the major way to add bugs and problems. One of the reasons FK are important is they eliminate the chance of getting the relationships wrong, but also, to get it right NOW.

samaysharma 3y ago

Agreed. Putting data for all your use cases in a single database is a huge deal. Having different databases for each part of the application bloats so much so easily from a cost, complexity and skill set standpoint. Also, makes it extremely hard to debug issues when you have to chart the path of data through 5 different tools. There is the challenge of workload separation and scaling each component separately but that can be resolved by pulling out challenging workloads into their own "database" albeit on the same stack.

refset 3y ago

> If all business happens in one transactional system, your semantics are dramatically simplified. 100% agreed. One of the biggest issues SQL databases have faced is that the scope & scale of "one transactional system" has evolved a lot more quickly than any off-the-shelf database architecture has been able to keep up with, resulting in an explosion of workaround technologies and approaches that really shouldn't need to exist. We're now firmly in the cloud database era and can look at things like Spanner to judge how far away Postgres remains from satisfying modern availability & scaling expectations. But it will be great to see OSS close the gap (hopefully soon!).

ddorian43 3y ago
YugabyteDB has PostgreSQL on top of Spanner architecture and is apache2 licensed (I work there).
- refset 3y ago
  "YugabyteDB Anywhere" is open core though right? Half the value & complexity is in the orchestration stack. It's definitely a step forward, but such licensing is likely still too restrictive to ever supplant Postgres itself (it terms of breadth of adoption).
  - ddorian43 3y ago
    > "YugabyteDB Anywhere" is open core though right? No, it's all open source. YugabyteDB Anywhere is just automation & 24/7 core engineering support. All db features are open source. You can re-create Anywhere without needing to touch c/c++. Note: All features have always been open source (besides incremental backup which will be soon).

0xbadcafebee 3y ago

Big databases are not new. If that was all that was needed, people would have been doing just that for the past 40 years. Turns out it doesn't always work, and sometimes (often?) it's terrible. Even if theoretically it was all you ever needed, the other constant problem is the implementation. Most developers, today anyway, are morons. They don't understand how databases work or how to use them, so they use them poorly. In order to get away from that fact, they invented new databases (like NoSQL) so they could use their brains less and just write more glue code. Turned out that was horrible as well. Pretty soon the tides will turn yet again and "just having a big database" will again be seen as not in vogue and another paradigm of "simplicity" will come around (instead of "one big database" being simple it'll be "many small databases" like we had with microservices). Those who don't understand history are doomed to repeat it.

karencarits 3y ago

Is there any good opensource software with gui to manage such databases/graphs? I guess Wikibase is an alternative, although not very user friendly In contrast, pocketbase [0] was discussed two days ago, and there are many user-friendly alternatives for relational data [e.g. 1-4] [0] https://news.ycombinator.com/item?id=38898934 https://news.ycombinator.com/item?id=38898934 [1] https://www.getgrist.com/ https://www.getgrist.com/ [2] https://nocodb.com/ https://nocodb.com/ [3] https://baserow.io/ https://baserow.io/ [4] https://www.rowy.io/ https://www.rowy.io/

felipellrocha 3y ago

I've heard so many people claim this, but I've never seen this put into practice in production, and at scale. I don't even think that that proves that this cannot be done... And I've seen A LOT of people try. But, things invariably start to break down, and at this point I just think that the statement above is more wishy washy than anything, though I want to believe...