5 ms·

> few hundred nodes at most, and Oracle RAC/ExaData clusters can scale up that far too and how much license fee it will cost?

mike_hearn 1y ago
For over a hundred nodes? A lot, but these aren't little VMs with 4G of RAM and two vCPUs we are talking about there. Almost nobody has databases that big. For a more realistic deployment in the cloud with 2TB of data, 4TB of backup storage and peak of 64 cores with elastic scaling it's about 6k/month. So the moment you're spending more than a third of a decently skilled SWE on working around database limitations it's worth it. That's for a fully managed HA cluster with 99.95% uptime, rolling upgrades etc.
- riku_iki 1y ago
  > 2TB of data, 4TB of backup storage and peak of 64 cores with elastic scaling it's about 6k/month right, but foundationdb and similar is for cases where you actually need throughput of hundreds servers, for what you described OSS Postgres will work well.
  - mike_hearn 1y ago
    It's confusing because of terminology differences. What FoundationDB calls a "server" is a single thread, often running on a commodity cloud VM. What Postgres calls a server is a multi-process instance on a single machine, often a commodity cloud VM. What ExaData calls a server is a dedicated bare metal machine with at least half a terabyte of RAM and a couple hundred CPU cores, with a gazillion processes running on it, linked to other cluster nodes by 100Gb networks. Often the machines are bigger. Obviously then a workload that might require dozens or hundreds of "servers" in FoundationDB can fit on just one in an Oracle database. Hundreds of nodes in an ExaData cluster would mean tens of thousands of cores and hundreds of terabytes of RAM, not to even get into the disk sizes. The nodes in turn are linked by a dedicated high-bandwidth network, with user traffic being routed over the regular network. As far as I know, nobody has ever scaled a FoundationDB cluster to tens of thousands of servers. So you can argue then that ExaData scales far better than FoundationDB does. This makes sense for the design because in an RDBMS cluster the inputs and outputs are small. A little bit of SQL goes in, a small result set comes out (usually). But the data transfers and work required to compute that small result might be very large - in extremis, it may require reading the entire database from disk. Even with the best networks bandwidth inside a computer is much higher than bandwidth between computers, so you want the biggest machines possible before spilling horizontally. You just get much better bang-for-buck that way. In FoundationDB on the other hand nodes barely communicate at all, because it's up to the client to do most of the work. If you write a workload that requires reading the entire database it will just fail immediately because you'll hit the five second timeout. If you do it without caring about transactional consistency (a huge sacrifice), you'll probably end up reading the entire database over a regular commodity network before being able to process it at all - much slower.
    - riku_iki 1y ago
      > Postgres calls a server is a multi-process instance on a single machine, often a commodity cloud VM. What ExaData calls a server is a dedicated bare metal machine with at least half a terabyte of RAM and a couple hundred CPU cores, with a gazillion processes running on it, linked to other cluster nodes by 100Gb networks. Often the machines are bigger. You can also have PG cluster of beefy machines linked by 100Gb network, I didn't get what's the difference in this case.
      - mike_hearn 1y ago
        This is again an issue of terminology. A PG cluster is at best one write master and then some full read replicas. You don't necessarily need a fast interconnect for that, but your write traffic can never exceed the power of the one master, and storage isn't sharded but replicated (unless you host all the PG files on a SAN which would be slow). A RAC cluster is multi-write master, and each node stores a subset of the data. You can then add as many read caches (not replicas) as you need, and they'll cache only the hot data blocks. So they're not quite the same thing in that sense.
        riku_iki 1y ago
        PG ecosystem actually has multiple shared nothing cluster implementations, example is www.citusdata.com, unlike RAC, where masters need to sync to accept writes, so technically write load is not distributted across servers.
        mike_hearn 1y ago
        Citus is great but it's just sharding. Only some very specific use cases fit within those limits, and there are cost issues too (being ha requires replicas of each node etc). RAC masters don't need to sync writes to each other, that's not how it works. Every node can be writing independently, they only communicate when one node has exclusive ownership over a block another node needs to write to, and the transfer then occurs peer to peer over rdma. But if writes are scattered they work independently.
- jwr 1y ago
  > fully managed HA cluster This is where Suspension of Disbelief stops for me — I've been taught by years of Jepsen analyses that "HA" really doesn't exist, especially in the SQL database world, and especially if there is a big company behind with lots of big complicated buzzwords. What you usually get is single master replication, and you get to pick up the pieces if the master dies.
  - mike_hearn 1y ago
    Jepsen doesn't work through databases in any kind of order as far as I can tell, and they haven't done an analysis of Oracle or many other popular databases. So I wouldn't take that as a representative sample. RAC clusters are multi-write-master and HA. The drivers know how to fail over between nodes, can continue sessions across a failover and this can work even across major version upgrades. The tech has existed for a long time already. If you're curious you can read about it more in the docs. "Application continuity" i.e. continuing a connection if a node dies without the application noticing: https://docs.oracle.com/en/database/oracle/oracle-database/23/racad/ensuring-application-continuity.html#GUID-C1EF6BDA-5F90-448F-A1E2-DC15AD5CFE75 https://docs.oracle.com/en/database/oracle/oracle-database/2... Fast failover and load balancing for cluster clients (here in the context of Java but it works for other languages too): https://docs.oracle.com/en/database/oracle/oracle-database/23/jjucp/overview-Oracle-RAC-features.html https://docs.oracle.com/en/database/oracle/oracle-database/2... I mean, I understand the skepticism because HN never covers this stuff. It's "enterprise" and not "startup" for whatever reason, unless Google do it. But Oracle has been selling databases into nearly every big company on earth for decades. Do you really think nobody has built an HA and horizontally scalable SQL based database? They have and people have been using it to run 24/7 businesses at scale for a long time.
    - jen20 1y ago
      I wonder if the clause in the Oracle license that prohibits publishing benchmarks and other studies could be to blame for Jepsen having not investigated..?

mike_hearn 1y ago

For over a hundred nodes? A lot, but these aren't little VMs with 4G of RAM and two vCPUs we are talking about there. Almost nobody has databases that big. For a more realistic deployment in the cloud with 2TB of data, 4TB of backup storage and peak of 64 cores with elastic scaling it's about 6k/month. So the moment you're spending more than a third of a decently skilled SWE on working around database limitations it's worth it. That's for a fully managed HA cluster with 99.95% uptime, rolling upgrades etc.

riku_iki 1y ago
> 2TB of data, 4TB of backup storage and peak of 64 cores with elastic scaling it's about 6k/month right, but foundationdb and similar is for cases where you actually need throughput of hundreds servers, for what you described OSS Postgres will work well.
- mike_hearn 1y ago
  It's confusing because of terminology differences. What FoundationDB calls a "server" is a single thread, often running on a commodity cloud VM. What Postgres calls a server is a multi-process instance on a single machine, often a commodity cloud VM. What ExaData calls a server is a dedicated bare metal machine with at least half a terabyte of RAM and a couple hundred CPU cores, with a gazillion processes running on it, linked to other cluster nodes by 100Gb networks. Often the machines are bigger. Obviously then a workload that might require dozens or hundreds of "servers" in FoundationDB can fit on just one in an Oracle database. Hundreds of nodes in an ExaData cluster would mean tens of thousands of cores and hundreds of terabytes of RAM, not to even get into the disk sizes. The nodes in turn are linked by a dedicated high-bandwidth network, with user traffic being routed over the regular network. As far as I know, nobody has ever scaled a FoundationDB cluster to tens of thousands of servers. So you can argue then that ExaData scales far better than FoundationDB does. This makes sense for the design because in an RDBMS cluster the inputs and outputs are small. A little bit of SQL goes in, a small result set comes out (usually). But the data transfers and work required to compute that small result might be very large - in extremis, it may require reading the entire database from disk. Even with the best networks bandwidth inside a computer is much higher than bandwidth between computers, so you want the biggest machines possible before spilling horizontally. You just get much better bang-for-buck that way. In FoundationDB on the other hand nodes barely communicate at all, because it's up to the client to do most of the work. If you write a workload that requires reading the entire database it will just fail immediately because you'll hit the five second timeout. If you do it without caring about transactional consistency (a huge sacrifice), you'll probably end up reading the entire database over a regular commodity network before being able to process it at all - much slower.
  - riku_iki 1y ago
    > Postgres calls a server is a multi-process instance on a single machine, often a commodity cloud VM. What ExaData calls a server is a dedicated bare metal machine with at least half a terabyte of RAM and a couple hundred CPU cores, with a gazillion processes running on it, linked to other cluster nodes by 100Gb networks. Often the machines are bigger. You can also have PG cluster of beefy machines linked by 100Gb network, I didn't get what's the difference in this case.
    - mike_hearn 1y ago
      This is again an issue of terminology. A PG cluster is at best one write master and then some full read replicas. You don't necessarily need a fast interconnect for that, but your write traffic can never exceed the power of the one master, and storage isn't sharded but replicated (unless you host all the PG files on a SAN which would be slow). A RAC cluster is multi-write master, and each node stores a subset of the data. You can then add as many read caches (not replicas) as you need, and they'll cache only the hot data blocks. So they're not quite the same thing in that sense.
      - riku_iki 1y ago
        PG ecosystem actually has multiple shared nothing cluster implementations, example is www.citusdata.com, unlike RAC, where masters need to sync to accept writes, so technically write load is not distributted across servers.
        mike_hearn 1y ago
        Citus is great but it's just sharding. Only some very specific use cases fit within those limits, and there are cost issues too (being ha requires replicas of each node etc). RAC masters don't need to sync writes to each other, that's not how it works. Every node can be writing independently, they only communicate when one node has exclusive ownership over a block another node needs to write to, and the transfer then occurs peer to peer over rdma. But if writes are scattered they work independently.
jwr 1y ago
> fully managed HA cluster This is where Suspension of Disbelief stops for me — I've been taught by years of Jepsen analyses that "HA" really doesn't exist, especially in the SQL database world, and especially if there is a big company behind with lots of big complicated buzzwords. What you usually get is single master replication, and you get to pick up the pieces if the master dies.
- mike_hearn 1y ago
  Jepsen doesn't work through databases in any kind of order as far as I can tell, and they haven't done an analysis of Oracle or many other popular databases. So I wouldn't take that as a representative sample. RAC clusters are multi-write-master and HA. The drivers know how to fail over between nodes, can continue sessions across a failover and this can work even across major version upgrades. The tech has existed for a long time already. If you're curious you can read about it more in the docs. "Application continuity" i.e. continuing a connection if a node dies without the application noticing: https://docs.oracle.com/en/database/oracle/oracle-database/23/racad/ensuring-application-continuity.html#GUID-C1EF6BDA-5F90-448F-A1E2-DC15AD5CFE75 https://docs.oracle.com/en/database/oracle/oracle-database/2... Fast failover and load balancing for cluster clients (here in the context of Java but it works for other languages too): https://docs.oracle.com/en/database/oracle/oracle-database/23/jjucp/overview-Oracle-RAC-features.html https://docs.oracle.com/en/database/oracle/oracle-database/2... I mean, I understand the skepticism because HN never covers this stuff. It's "enterprise" and not "startup" for whatever reason, unless Google do it. But Oracle has been selling databases into nearly every big company on earth for decades. Do you really think nobody has built an HA and horizontally scalable SQL based database? They have and people have been using it to run 24/7 businesses at scale for a long time.
  - jen20 1y ago
    I wonder if the clause in the Oracle license that prohibits publishing benchmarks and other studies could be to blame for Jepsen having not investigated..?