3 ms·

How it works is: PostgreSQL WAL is sent to 3 'Safekeeper' nodes, which provide temporary persistence of WAL on their local disks. This allows us to provide low

How it works is: PostgreSQL WAL is sent to 3 'Safekeeper' nodes, which provide temporary persistence of WAL on their local disks. This allows us to provide low commit latencies. After Safekeepers acknowledge the WAL, a PageServer will receive the WAL from these Safekeepers and transform it into LSM-tree "Layers" - blocks of lookup-optimized changelogs, which (when complete) are sent to S3. At that point, the data is considered fully persisted against most, if not all, outages. The PageServer (which serves as the long-term data server for the running compute nodes) maintains a local cache of Layers. Still, by design, that is only a cache -- it allows for fast responses but is not strictly necessary for the persistence model.

cpursley 4y ago
Can we set our own logical replication? I'm listening to the WAL the same way Supabase does it: https://github.com/cpursley/walex#logical-replication https://github.com/cpursley/walex#logical-replication
- mattashii 4y ago
  In our cloud offering logical replication is not yet supported. There's an open epic to support receiving a logical replication stream [0], but we've not yet checked on sending logical or physical replication streams. [0] https://github.com/neondatabase/neon/issues/1632 https://github.com/neondatabase/neon/issues/1632
  - netcraft 4y ago
    Everything you've described sounds unbelievably awesome - this is the only thing I can think of that would make it better. If we can get all of these features (I've been dreaming of branching for years, scale to zero!, unlimited storage - just not having to think ahead about it!) plus easy, even on by default logical replication, you will have found the holy grail of cloud databases imo.
IgorPartola 4y ago
Again I am super impressed with the technology involved but do want to clarify: in order to have D in ACID the update must be sent to S3, right? Is there a mode which makes it so that an INSERT, UODATE, and DELETE do it return until this happens? What kind of latency does that introduce and is that latency affected by throughput at all?
- mattashii 4y ago
  Kind of. S3 is the long-term low-cost durability guarantee, while our Safekeepers (3, each in a different zone) provide a high-cost short-term durability guarantee with their local persistent disks. Latency from PostgreSQL WAL to S3 depends on WAL throughput and the configured pageserver checkpoint distance (default 256MB, and this config field is not equal to that of PostgreSQL).
  - IgorPartola 4y ago
    When you say short term do you mean for hot data or that the guarantee is short term? As in, once it is written to the Safekeeprs is there any chance that the data will disappear?
    - mattashii 4y ago
      We keep it there for a short duration, until the changes are confirmed to also be written to S3. Writing to 3 instances in 3 availability zones is considered persistent enough while also maintaining a high performance, and even though it does not provide the 11 9s of durability that S3 has, 3 availability zones dropping out with loss of all instance-local storage is considered rare enough that we do not think that it will impact our availability and durability guarantees.
      - IgorPartola 4y ago
        That makes sense, thank you! Sounds pretty damn robust.
- 4y ago

cpursley 4y ago

Can we set our own logical replication? I'm listening to the WAL the same way Supabase does it: https://github.com/cpursley/walex#logical-replication https://github.com/cpursley/walex#logical-replication

mattashii 4y ago
In our cloud offering logical replication is not yet supported. There's an open epic to support receiving a logical replication stream [0], but we've not yet checked on sending logical or physical replication streams. [0] https://github.com/neondatabase/neon/issues/1632 https://github.com/neondatabase/neon/issues/1632
- netcraft 4y ago
  Everything you've described sounds unbelievably awesome - this is the only thing I can think of that would make it better. If we can get all of these features (I've been dreaming of branching for years, scale to zero!, unlimited storage - just not having to think ahead about it!) plus easy, even on by default logical replication, you will have found the holy grail of cloud databases imo.

IgorPartola 4y ago

Again I am super impressed with the technology involved but do want to clarify: in order to have D in ACID the update must be sent to S3, right? Is there a mode which makes it so that an INSERT, UODATE, and DELETE do it return until this happens? What kind of latency does that introduce and is that latency affected by throughput at all?

mattashii 4y ago
Kind of. S3 is the long-term low-cost durability guarantee, while our Safekeepers (3, each in a different zone) provide a high-cost short-term durability guarantee with their local persistent disks. Latency from PostgreSQL WAL to S3 depends on WAL throughput and the configured pageserver checkpoint distance (default 256MB, and this config field is not equal to that of PostgreSQL).
- IgorPartola 4y ago
  When you say short term do you mean for hot data or that the guarantee is short term? As in, once it is written to the Safekeeprs is there any chance that the data will disappear?
  - mattashii 4y ago
    We keep it there for a short duration, until the changes are confirmed to also be written to S3. Writing to 3 instances in 3 availability zones is considered persistent enough while also maintaining a high performance, and even though it does not provide the 11 9s of durability that S3 has, 3 availability zones dropping out with loss of all instance-local storage is considered rare enough that we do not think that it will impact our availability and durability guarantees.
    - IgorPartola 4y ago
      That makes sense, thank you! Sounds pretty damn robust.
4y ago