8 ms·

> In less than a decade, the world has become at least three orders of magnitude more transactional. And yet the SQL databases we still use to power this are 20

> In less than a decade, the world has become at least three orders of magnitude more transactional. And yet the SQL databases we still use to power this are 20-30 years old. Can they hold up? Errr yes. Without much sweat really. Just because something started ~30 years ago doesn't mean it hasn't updated with the times, and doesn't mean it was built on bad foundations.

dakiol 1y ago
I disagree. If we are talking about distributed systems in which we have N different databases, then distributed transactions are left as an exercise to the reader (that’s why we have things like Sagas). Within a single machine, yeah, relational dbs still work like a charm.
- Normal_gaussian 1y ago
  The other interesting thing to consider is that machines are a lot more powerful in all dimensions than they were 30 years ago - so many tasks that would have required distributed systems no longer do.
  - jorangreef 1y ago
    DBMS design has also changed. The bottleneck for a DBMS has moved from spinning disk to CPU and per core memory bandwidth. That's also key to TigerBeetle's design. For example, with static allocation and zero-copy techniques such as zero-deserialization. If you were to design an OLGP DBMS like Postgres today, it would look radically different. Same is true for OLTP.
  - jandrewrogers 1y ago
    The caveat is that the hardware constraints have fundamentally changed, so database architectures from 30 years ago are effectively incapable of properly using modern hardware environments. Databases have not been bottlenecked on storage bandwidth in a long time but most databases are designed as if this was still the case. Optimizing for memory bandwidth, the current bottleneck, leads to substantially different architectures than are commonly deployed.
- big_hacker 1y ago
  If it's gonna fit on one machine I'm picking Postgres, Mongo or Redis before TigerBeetle and let's be honest it's not the difference between the 4 that will make it not fit on one machine.
  - jorangreef 1y ago
    And to be fair, those are all great general purpose solutions. For example, you could also use Postgres or Redis as queue, instead of say Redpanda or Kafka. But our customers need separation of concerns in their architecture. While they could put the cash in the general purpose filing cabinet, they actually want the separation of concerns between OLGP system of reference (string database, i.e. PG) in the control plane, and OLTP system of record (integer/counting database, i.e. TB) in the data plane.
- pjmlp 1y ago
  I was part of a project that successfully deployed them into production using Oracle, sometimes those license costs are actually worth it.
  - eitland 1y ago
    Agree. But for anyone tempted by Oracle, do remember that the upfront, agreed licence costs are only a fraction of the true price: You’ll need someone who actually knows Oracle - either already in place or willing to invest a serious amount of time learning it. Without that, you’re almost certainly better off choosing a simpler database that people are comfortable with. There’s always a non-zero risk of an Oracle rep trying to “upsell” you. And by upsell, I mean threatening legal action unless you cough up for additional bits a new salesperson has suddenly decided you’re using. (A company I worked with sold Oracle licences and had a long, happy relationship with them - until one day they went after us over some development databases. Someone higher up at Oracle smoothed it over, but the whole experience was unnerving enough.) Incidental and accidental complexity: I’ve worked with Windows from 3.1 through to Server 2008, with Linux from early Red Hat in 2001 through to the latest production distros, plus a fair share of weird and wonderful applications, everything from 1980s/90s radar acquisition running on 2010 operating systems through a wide range of in house, commercial and open source software and up to modern microservices — and none of it comes close to Oracle’s level of pain. Edit: Installating Delphi 6 with 14 packages came close, I used 3 days when I had to find every package scattered on disks in shelves and drawers and across ancient web paces + posted as abandonware on source forge but I guess I could learn to do that in a day if I had to do it twice a month. Oracle consistently took me 3 days - if I did everything correct on first try and didn't have to start from scratch.
    - MobiusHorizons 1y ago
      Wait, are you saying that oracle lets you use features you don’t have a license for, but then threatens to sue you for using them? I guess I shouldn’t be surprised given oracle’s business model, but I am surprised they let you use the feature in the first place.
- sgarland 1y ago
  Tbf, this is a natural consequence of companies doing premature optimization / cargo culting. There are many, many reasons to NOT go into microservices. Kelsey Hightower had some thoughts on this [0] back in 2020. You can run a hell of a lot off of a small fleet of beefy servers fronted with a load balancer, all pointing to one DB cluster. [0]: https://changelog.com/posts/monoliths-are-the-future https://changelog.com/posts/monoliths-are-the-future
pkphilip 1y ago
True. The older databases run very well even on hardware which was far less powerful than what is available now.
- deleted 1y ago
  [deleted]
jorangreef 1y ago
Joran from TigerBeetle! Without much sweat for general purpose workloads. But transaction processing tends to have power law contention that kills SQL row locks (cf. Amdahl’s Law). We put a contention calculator on our homepage to show the theoretical best case limits and they’re lower than one might think: https://tigerbeetle.com/#general-purpose-databases-have-an-oltp-limit https://tigerbeetle.com/#general-purpose-databases-have-an-o...
- tux3 1y ago
  >Traditional SQL databases hold locks across the network; under Amdahl's Law, even modest contention caps write throughput at ≈100–1,000 TPS In fact large real world systems are not limited to 100-1000 TPS, or even 10 kTPS as the calculator tries to suggest. That's not because Amdahl's law is wrong, the numbers you're plugging in are just wildly off, so the conclusions are equally nonsensical. There might be some specific workloads where you saw those numbers, and your DB might be a good fit for this particular niche, but you shouldn't misrepresent general purpose workloads to try to prop up your DB. Claiming that SQL databases are limited to "100-1000 TPS" is unserious, it is not conductive to your cause.
  - cma256 1y ago
    Its a financial database built for use cases where this invariant holds and built for enabling new use cases where this invariant prevented businesses from expanding into new industries. The creator says as much: > Without much sweat for general purpose workloads.
    - tux3 1y ago
      Having a niche and expanding into new industries are all fine, there is no problem with having a DB filling a particular sub-segment of the market. But writing that traditional SQL databases cannot go above these "100-1000 TPS" numbers due to Amdahl's law is going to raise some eyebrows.
      - cma256 1y ago
Xelbair 1y ago
DNS still runs strong and it was ~~designed~~ released* in the November 1983. It still holds up basically whole of internet. in most cases SQL is good enough for 90% of workloads.
- seemaze 1y ago
  What would DNS look like if it were released in 2025?
  - Something1234 1y ago
    It would be completely centralized with a micropayment rent seeking solution to update records.
    - ochronus 1y ago
      I laughed at first, then I cried, because sadly, it's true.
    - oblio 1y ago
      I think it would actually be (pseudo-)decentralized and you'd have to mine a blockchain node containing each domain, where its creators will have reserved at least 10% of its node pool to make sure they became billionaires if it took off. Plus micropayments, of course :-)
    - TZubiri 1y ago
      Nono, it would run decentralized in blockchain with a mircropayment rent seeking solution to update records.
      - vvern 1y ago
        You’re right that it would run on a block chain, but that fact would primarily exist to power some marketing. Everybody would end up interacting with it through a single centralized web site and API because it’s the only usable way to get it to work.
yomismoaqui 1y ago
Are you telling me that new shiny things are not always better that established, time tested and boring tech? Sometimes I feel like we software engineers have the worst memory of any engineers.
- sgarland 1y ago
  I don’t even think it’s memory, it’s lack of experience. A lot of the battle-tested infrastructure is now abstracted away to the point that a lot of devs are hard-pressed to explain the difference between AWS’ ALB and NLB, much less name OSS versions of them. Actually, load balancers are a great example. The number of times I’ve seen a team re-implementing HAProxy, but poorly, is entirely too high. The reasoning is always something along the lines of “we need this extremely specific feature.” OK, and have you even read the manual for HAProxy? Or if that feature truly doesn’t exist, did you consider implementing that separately?
  - yomismoaqui 1y ago
    True, the only right answer for a question like "Should we use X here?" is "It depends". Maybe NIH is the right answer in some specific case but not on another that seems quite similar to the first. Some months ago I was re-enlightened when Anders Hejlsberg (creator of C# and TypeScript) explained why they chose Go for reimplementing the TypeScript compiler, instead of using any of those languages or something like Rust. The way they defined the problem, the tests they did and how they justify their pick is how these kind of decisions should be made if we want to call ourselves engineers.
- causenad 1y ago
  Kelsey Hightower gave an interesting talk on this - why understanding fundamentals will always matter more than the recent shiny new trend: https://youtu.be/Jlqzy02k6B8 https://youtu.be/Jlqzy02k6B8
cmrdporcupine 1y ago
Yeah, SQL is not the issue. At least not mostly. The relational model has shown itself to be exactly the flexible and powerful model that Codd said it was, even in its relatively-debased from in SQL. In fact the potential of it as a universal and flexible data model that abstracts away storage still remains to be fully unlocked. Now as for existing SQL databases, yes, many of them were built on foundational assumptions about the nature of memory and secondary storage that no longer hold true. Many of us in this industry still have our heads stuck in the past world of spinny spinny magnetic disks on platters with heads and cylinders and grinding noises. Real world hardware has moved on. We have a couple orders of magnitude higher IOPS generally available than we did just 10-15 years ago. So I'm excited about products like CedarDB and others (Umbra before it, etc) that are being built on a foundation from day one of managing with hybrid in-memory/disk. Throwing out SQL is not really the recipe for "performance" and the lessons of separating the storage system from the data model is still key, since the 1960s. I am willing to grant that a specialized transaction/ledger system like TigerBeetle might have its place as an optimization strategy in a specific industry/domain, but we should not make the mistake of characterizing this as a general problem with the relational model and data storage broadly.
- 9rx 1y ago
  > Throwing out SQL is not really the recipe for "performance" We've discovered hacks to work around the limitations of SQL, so you can maintain performance with sufficient hackiness, but you have to give up "purity" to get there. Worse is better applies, I suppose, but if we were starting over today the design would be very different. Throwing out SQL isn't strictly required, but like Rust isn't strictly required when you already have C, sometimes it is nice to take a step back and look at how you can improve upon things. Unfortunately, NoSQL turning into a story about "document databases" instead of "A better SQL" killed any momentum on that front.
  - cmrdporcupine 1y ago
    No, I'm arguing in fact the opposite and you'll have to prove the point wrong instead of drive-by insisting on this. The point of Codd's original paper was to separate concerns about data storage from its representation and to propose an elegant representational & transformation & query model based on first order logic and I believe he succeeded. SQL was the first serious attempt to translate this into a working system, and it's full of warts, but the underlying model it is reaching towards retains its elegance. But most importantly the principle of not tying the structure of data to its storage model is absolutely key. Not least because we can iteratively and flexibly improve the storage model over time, and not be stuck with decisions that tie us to one particular structure which is the problem that 1960s "network" and "hierarchical" databases (and modern day "key value" or "NoSQL" databases) cause.

dakiol 1y ago

I disagree. If we are talking about distributed systems in which we have N different databases, then distributed transactions are left as an exercise to the reader (that’s why we have things like Sagas). Within a single machine, yeah, relational dbs still work like a charm.

Normal_gaussian 1y ago
The other interesting thing to consider is that machines are a lot more powerful in all dimensions than they were 30 years ago - so many tasks that would have required distributed systems no longer do.
- jorangreef 1y ago
  DBMS design has also changed. The bottleneck for a DBMS has moved from spinning disk to CPU and per core memory bandwidth. That's also key to TigerBeetle's design. For example, with static allocation and zero-copy techniques such as zero-deserialization. If you were to design an OLGP DBMS like Postgres today, it would look radically different. Same is true for OLTP.
- jandrewrogers 1y ago
  The caveat is that the hardware constraints have fundamentally changed, so database architectures from 30 years ago are effectively incapable of properly using modern hardware environments. Databases have not been bottlenecked on storage bandwidth in a long time but most databases are designed as if this was still the case. Optimizing for memory bandwidth, the current bottleneck, leads to substantially different architectures than are commonly deployed.
big_hacker 1y ago
If it's gonna fit on one machine I'm picking Postgres, Mongo or Redis before TigerBeetle and let's be honest it's not the difference between the 4 that will make it not fit on one machine.
- jorangreef 1y ago
  And to be fair, those are all great general purpose solutions. For example, you could also use Postgres or Redis as queue, instead of say Redpanda or Kafka. But our customers need separation of concerns in their architecture. While they could put the cash in the general purpose filing cabinet, they actually want the separation of concerns between OLGP system of reference (string database, i.e. PG) in the control plane, and OLTP system of record (integer/counting database, i.e. TB) in the data plane.
pjmlp 1y ago
I was part of a project that successfully deployed them into production using Oracle, sometimes those license costs are actually worth it.
- eitland 1y ago
  Agree. But for anyone tempted by Oracle, do remember that the upfront, agreed licence costs are only a fraction of the true price: You’ll need someone who actually knows Oracle - either already in place or willing to invest a serious amount of time learning it. Without that, you’re almost certainly better off choosing a simpler database that people are comfortable with. There’s always a non-zero risk of an Oracle rep trying to “upsell” you. And by upsell, I mean threatening legal action unless you cough up for additional bits a new salesperson has suddenly decided you’re using. (A company I worked with sold Oracle licences and had a long, happy relationship with them - until one day they went after us over some development databases. Someone higher up at Oracle smoothed it over, but the whole experience was unnerving enough.) Incidental and accidental complexity: I’ve worked with Windows from 3.1 through to Server 2008, with Linux from early Red Hat in 2001 through to the latest production distros, plus a fair share of weird and wonderful applications, everything from 1980s/90s radar acquisition running on 2010 operating systems through a wide range of in house, commercial and open source software and up to modern microservices — and none of it comes close to Oracle’s level of pain. Edit: Installating Delphi 6 with 14 packages came close, I used 3 days when I had to find every package scattered on disks in shelves and drawers and across ancient web paces + posted as abandonware on source forge but I guess I could learn to do that in a day if I had to do it twice a month. Oracle consistently took me 3 days - if I did everything correct on first try and didn't have to start from scratch.
  - MobiusHorizons 1y ago
    Wait, are you saying that oracle lets you use features you don’t have a license for, but then threatens to sue you for using them? I guess I shouldn’t be surprised given oracle’s business model, but I am surprised they let you use the feature in the first place.
sgarland 1y ago
Tbf, this is a natural consequence of companies doing premature optimization / cargo culting. There are many, many reasons to NOT go into microservices. Kelsey Hightower had some thoughts on this [0] back in 2020. You can run a hell of a lot off of a small fleet of beefy servers fronted with a load balancer, all pointing to one DB cluster. [0]: https://changelog.com/posts/monoliths-are-the-future https://changelog.com/posts/monoliths-are-the-future

pkphilip 1y ago

True. The older databases run very well even on hardware which was far less powerful than what is available now.

deleted 1y ago
[deleted]

jorangreef 1y ago

Joran from TigerBeetle! Without much sweat for general purpose workloads. But transaction processing tends to have power law contention that kills SQL row locks (cf. Amdahl’s Law). We put a contention calculator on our homepage to show the theoretical best case limits and they’re lower than one might think: https://tigerbeetle.com/#general-purpose-databases-have-an-oltp-limit https://tigerbeetle.com/#general-purpose-databases-have-an-o...

tux3 1y ago
>Traditional SQL databases hold locks across the network; under Amdahl's Law, even modest contention caps write throughput at ≈100–1,000 TPS In fact large real world systems are not limited to 100-1000 TPS, or even 10 kTPS as the calculator tries to suggest. That's not because Amdahl's law is wrong, the numbers you're plugging in are just wildly off, so the conclusions are equally nonsensical. There might be some specific workloads where you saw those numbers, and your DB might be a good fit for this particular niche, but you shouldn't misrepresent general purpose workloads to try to prop up your DB. Claiming that SQL databases are limited to "100-1000 TPS" is unserious, it is not conductive to your cause.
- cma256 1y ago
  Its a financial database built for use cases where this invariant holds and built for enabling new use cases where this invariant prevented businesses from expanding into new industries. The creator says as much: > Without much sweat for general purpose workloads.
  - tux3 1y ago
    Having a niche and expanding into new industries are all fine, there is no problem with having a DB filling a particular sub-segment of the market. But writing that traditional SQL databases cannot go above these "100-1000 TPS" numbers due to Amdahl's law is going to raise some eyebrows.
    - cma256 1y ago

Xelbair 1y ago

DNS still runs strong and it was ~~designed~~ released* in the November 1983. It still holds up basically whole of internet. in most cases SQL is good enough for 90% of workloads.

seemaze 1y ago
What would DNS look like if it were released in 2025?
- Something1234 1y ago
  It would be completely centralized with a micropayment rent seeking solution to update records.
  - ochronus 1y ago
    I laughed at first, then I cried, because sadly, it's true.
  - oblio 1y ago
    I think it would actually be (pseudo-)decentralized and you'd have to mine a blockchain node containing each domain, where its creators will have reserved at least 10% of its node pool to make sure they became billionaires if it took off. Plus micropayments, of course :-)
  - TZubiri 1y ago
    Nono, it would run decentralized in blockchain with a mircropayment rent seeking solution to update records.
    - vvern 1y ago
      You’re right that it would run on a block chain, but that fact would primarily exist to power some marketing. Everybody would end up interacting with it through a single centralized web site and API because it’s the only usable way to get it to work.

yomismoaqui 1y ago

Are you telling me that new shiny things are not always better that established, time tested and boring tech? Sometimes I feel like we software engineers have the worst memory of any engineers.

sgarland 1y ago
I don’t even think it’s memory, it’s lack of experience. A lot of the battle-tested infrastructure is now abstracted away to the point that a lot of devs are hard-pressed to explain the difference between AWS’ ALB and NLB, much less name OSS versions of them. Actually, load balancers are a great example. The number of times I’ve seen a team re-implementing HAProxy, but poorly, is entirely too high. The reasoning is always something along the lines of “we need this extremely specific feature.” OK, and have you even read the manual for HAProxy? Or if that feature truly doesn’t exist, did you consider implementing that separately?
- yomismoaqui 1y ago
  True, the only right answer for a question like "Should we use X here?" is "It depends". Maybe NIH is the right answer in some specific case but not on another that seems quite similar to the first. Some months ago I was re-enlightened when Anders Hejlsberg (creator of C# and TypeScript) explained why they chose Go for reimplementing the TypeScript compiler, instead of using any of those languages or something like Rust. The way they defined the problem, the tests they did and how they justify their pick is how these kind of decisions should be made if we want to call ourselves engineers.
causenad 1y ago
Kelsey Hightower gave an interesting talk on this - why understanding fundamentals will always matter more than the recent shiny new trend: https://youtu.be/Jlqzy02k6B8 https://youtu.be/Jlqzy02k6B8

cmrdporcupine 1y ago

Yeah, SQL is not the issue. At least not mostly. The relational model has shown itself to be exactly the flexible and powerful model that Codd said it was, even in its relatively-debased from in SQL. In fact the potential of it as a universal and flexible data model that abstracts away storage still remains to be fully unlocked. Now as for existing SQL databases, yes, many of them were built on foundational assumptions about the nature of memory and secondary storage that no longer hold true. Many of us in this industry still have our heads stuck in the past world of spinny spinny magnetic disks on platters with heads and cylinders and grinding noises. Real world hardware has moved on. We have a couple orders of magnitude higher IOPS generally available than we did just 10-15 years ago. So I'm excited about products like CedarDB and others (Umbra before it, etc) that are being built on a foundation from day one of managing with hybrid in-memory/disk. Throwing out SQL is not really the recipe for "performance" and the lessons of separating the storage system from the data model is still key, since the 1960s. I am willing to grant that a specialized transaction/ledger system like TigerBeetle might have its place as an optimization strategy in a specific industry/domain, but we should not make the mistake of characterizing this as a general problem with the relational model and data storage broadly.

9rx 1y ago
> Throwing out SQL is not really the recipe for "performance" We've discovered hacks to work around the limitations of SQL, so you can maintain performance with sufficient hackiness, but you have to give up "purity" to get there. Worse is better applies, I suppose, but if we were starting over today the design would be very different. Throwing out SQL isn't strictly required, but like Rust isn't strictly required when you already have C, sometimes it is nice to take a step back and look at how you can improve upon things. Unfortunately, NoSQL turning into a story about "document databases" instead of "A better SQL" killed any momentum on that front.
- cmrdporcupine 1y ago
  No, I'm arguing in fact the opposite and you'll have to prove the point wrong instead of drive-by insisting on this. The point of Codd's original paper was to separate concerns about data storage from its representation and to propose an elegant representational & transformation & query model based on first order logic and I believe he succeeded. SQL was the first serious attempt to translate this into a working system, and it's full of warts, but the underlying model it is reaching towards retains its elegance. But most importantly the principle of not tying the structure of data to its storage model is absolutely key. Not least because we can iteratively and flexibly improve the storage model over time, and not be stuck with decisions that tie us to one particular structure which is the problem that 1960s "network" and "hierarchical" databases (and modern day "key value" or "NoSQL" databases) cause.