3 ms·

My employer (Neon) offers Postgres databases that run on top of a couple of caching layers at the end of which there is S3: https://neon.tech/docs/introduction/

by est31 3y ago

My employer (Neon) offers Postgres databases that run on top of a couple of caching layers at the end of which there is S3: https://neon.tech/docs/introduction/architecture-overview https://neon.tech/docs/introduction/architecture-overview Directly exposing every write to S3 gives you the partial overwrite issues as described. But one can collect a bunch of traffic and push state to S3 once it reaches a threshold. Instead, a few writes in the postgres WAL are held outside of S3 in a replicated on-disk cache.

chubot 3y ago
Thanks for the link. But I searched the docs for "durability" and got zero results. Before I use anything like this, I'd like to see what durability settings are used: https://www.postgresql.org/docs/current/non-durability.html https://www.postgresql.org/docs/current/non-durability.html Litestream documents the their data loss window, it seems like Neon should too: https://litestream.io/tips/ https://litestream.io/tips/ By default, Litestream will replicate new changes to an S3 replica every second. During this time where data has not yet been replicated, a catastrophic crash on your server will result in the loss of data in that time window. I also searched for "data loss" and got zero results -- this is important because Neon is almost certainly sacrificing durability for performance.
- yencabulator 3y ago
  Neon handles that by staging the WAL segments on 3x replicated Safekeeper nodes. Durability relies on not having all of those blow up at the same time. I'd expect it to be much safer than traditional Postgres replication mechanisms (with the trade-off having a comparatively large minimum node count; Neon really is built for multitenancy where that cost can be amortized across lots of databases).
- est31 3y ago
  > I searched the docs for "durability" and got zero results. The link I gave above explains it, right the sentence with "durability": > Safekeepers are responsible for durability of recent updates. Postgres streams Write-Ahead Log (WAL) to the Safekeepers, and the Safekeepers store the WAL durably until it has been processed by the Pageservers and uploaded to cloud storage. > Safekeepers can be thought of as an ultra reliable write buffer that holds the latest data until it is processed and uploaded to cloud storage. Safekeepers implement the Paxos protocol for reliability.
  - chubot 3y ago
    OK thanks, I glossed over that part. But really what I look for is (1) explicit claims about durability, and (2) a third party (e.g. Aphyr) actually tested the claims. If there's no claim, then it's impossible to test :) In particular, there are no numbers in the description you quoted. Litestream gives a relatively weak claim, but it could be tested, which actually gives me more confidence in it. If you look at what aphyr writes, a lot of it is claims from vendors that turned out to be false - https://aphyr.com/tags/jepsen https://aphyr.com/tags/jepsen

by est31 3y ago

chubot 3y ago

Thanks for the link. But I searched the docs for "durability" and got zero results. Before I use anything like this, I'd like to see what durability settings are used: https://www.postgresql.org/docs/current/non-durability.html https://www.postgresql.org/docs/current/non-durability.html Litestream documents the their data loss window, it seems like Neon should too: https://litestream.io/tips/ https://litestream.io/tips/ By default, Litestream will replicate new changes to an S3 replica every second. During this time where data has not yet been replicated, a catastrophic crash on your server will result in the loss of data in that time window. I also searched for "data loss" and got zero results -- this is important because Neon is almost certainly sacrificing durability for performance.

yencabulator 3y ago
Neon handles that by staging the WAL segments on 3x replicated Safekeeper nodes. Durability relies on not having all of those blow up at the same time. I'd expect it to be much safer than traditional Postgres replication mechanisms (with the trade-off having a comparatively large minimum node count; Neon really is built for multitenancy where that cost can be amortized across lots of databases).
est31 3y ago
> I searched the docs for "durability" and got zero results. The link I gave above explains it, right the sentence with "durability": > Safekeepers are responsible for durability of recent updates. Postgres streams Write-Ahead Log (WAL) to the Safekeepers, and the Safekeepers store the WAL durably until it has been processed by the Pageservers and uploaded to cloud storage. > Safekeepers can be thought of as an ultra reliable write buffer that holds the latest data until it is processed and uploaded to cloud storage. Safekeepers implement the Paxos protocol for reliability.
- chubot 3y ago
  OK thanks, I glossed over that part. But really what I look for is (1) explicit claims about durability, and (2) a third party (e.g. Aphyr) actually tested the claims. If there's no claim, then it's impossible to test :) In particular, there are no numbers in the description you quoted. Litestream gives a relatively weak claim, but it could be tested, which actually gives me more confidence in it. If you look at what aphyr writes, a lot of it is claims from vendors that turned out to be false - https://aphyr.com/tags/jepsen https://aphyr.com/tags/jepsen