4 ms·

Do note that many installations of PostgreSQL already have some form of "separation of storage and compute" through a networked storage solution like NFS, EBS,

Do note that many installations of PostgreSQL already have some form of "separation of storage and compute" through a networked storage solution like NFS, EBS, or other SAN-like systems. The major part of what Neon does is remove the file system abstraction that is between that Storage and Compute, so that we can better utilize the available resources because we can better select what information is or isn't being lost. A good example of what removing the file system abstraction enables for us is effectively free PITR, (lagging) replicas, and data branching. This is because PostgreSQL's file-system-based storage engine expects to be the only one working on the data directory, which means that any FS attached to a replica cannot be shared. If you remove that file-system based storage engine and plug in a different storage engine, those expectations are removed too, and after some effort integrating into the smgr-APIs, we're now able to provide a storage layer that only needs to contain one copy of the data for N physical replicas, instead of N copies.

TruthWillHurt 4y ago
The fact some systems do it, doesn't mean it is correct or optimals. The latency of NFS & EBS or EFS is actually the reason many businesses *do not* use them for their databases. I've seen deployments that had to go bare-metal because the tiny latency of EBS caused their compute times to rise exponentialy (doing AI training).
- mattashii 4y ago
  I agree that it does have its limitations, but databases are all about trade-offs. Talking about the "correctness" of a choice between tradeoffs is weird though. Running your database on hard drives nowadays is not a great choice - yet people still did that because the cost of hard drives was way cheaper than that of memory. Running your database in a way that doesn't guarantee that a 'committed' response actually retains the changes that it was responding on - yet people still ran their database in such configuration to scrape the barrel on performance. All Neon does is put up another option: If you don't mind the implications of networked storage, then here's one database system that has zero-cost cloning and does not lose data on single-node failure.

TruthWillHurt 4y ago

The fact some systems do it, doesn't mean it is correct or optimals. The latency of NFS & EBS or EFS is actually the reason many businesses *do not* use them for their databases. I've seen deployments that had to go bare-metal because the tiny latency of EBS caused their compute times to rise exponentialy (doing AI training).

mattashii 4y ago
I agree that it does have its limitations, but databases are all about trade-offs. Talking about the "correctness" of a choice between tradeoffs is weird though. Running your database on hard drives nowadays is not a great choice - yet people still did that because the cost of hard drives was way cheaper than that of memory. Running your database in a way that doesn't guarantee that a 'committed' response actually retains the changes that it was responding on - yet people still ran their database in such configuration to scrape the barrel on performance. All Neon does is put up another option: If you don't mind the implications of networked storage, then here's one database system that has zero-cost cloning and does not lose data on single-node failure.