4 ms·

I don't know if it's the same person, but someone working on Atlassian's cloud offerings said he reported this issue (along with a working patch) to Postgres an

I don't know if it's the same person, but someone working on Atlassian's cloud offerings said he reported this issue (along with a working patch) to Postgres and they declined it. Sounds like what they do for cloud services is run a lot of Postgres instances where the DB data is on a big NFS, and this bug was a bigger problem due to how NFS works. But after patching the fsync handling in Postgres, they continued using Postgres-on-NFS with great success. Unfortunately I can't find the original comment, but I think it was on another HN story about Postgres+fsync. Edit: I found it: https://news.ycombinator.com/item?id=19126601 https://news.ycombinator.com/item?id=19126601 and I may or may not have hallucinated the part about Atlassian.

dboreham 7y ago
In case someone reads this and gets the impression that running PG over NFS is in general safe or a good idea, I'm pretty sure it still isn't, unless like the OP you have a complete understanding of what you're doing.
- merb 7y ago
  OT: it depends on how much writes/reads you get. for most people nfs is probably an easier solution for HA on postgres than anything else.
- rosser 7y ago
  I have quite successfully run pg atop NFS — even, in limited, point-solution type roles, in production. My experience doing that didn't leave me with the impression that it was a particularly egregious thing to do, though I would definitely take many, many additional steps to ensure redundancy and availability if I were going to use it more generally. You're right though: you really do want to have some idea what you're doing, if you're going to go there. Source: my day job is PostgreSQL DBA, and has been for ~15 years now. EDIT: Phrasing.
  - cyphar 7y ago
    The main thing to keep in mind is that NFS exercises a bunch of VFS code paths that effectively no other filesystems touches. So you are more at risk to strange VFS bugs, as well as general NFS shenanigans, which other PG users may not have seen before.
  - macdice 7y ago
    I think the problem is mostly ENOSPC from fsync() which jettisons data just like EIO on Linux. If you ran out of space, PostgreSQL would only learn about that while checkpointing, and then retry and carry on. Boom, data loss. Today PostgreSQL would panic on the first ENOSPC from fsync() so the problem is mostly "fixed".
- georgebarnett 7y ago
  That’s pretty much computing. It’s usually the case that unless you have a complete understanding, there’s a pothole waiting to trip you. As an FYI - we no longer use Postgres on NFS as we have migrated the entire architecture underlying the systems in question.
pgaddict 7y ago
I think it's a bit strange to claim the PostgreSQL community just declined the patch. Just look at the thread on pgsql-hackers list: https://www.postgresql.org/message-id/flat/46ED7555-9383-45D2-A5C0-6373BE2FDCB6%40atlassian.com https://www.postgresql.org/message-id/flat/46ED7555-9383-45D... As anarazel already pointed out in the hackernews thread you linked, there's one person who was initially skeptical about the patch, and a couple of people chimed in with feedback (e.g. pointing out some mount options may fix it, which I think is valuable). And then the author of the patch just disappears from the discussion, so the thread kinda gets forgotten for a couple of months. I'm sure PostgreSQL community has its flaws, but I doubt this is a particularly clear example ...

dboreham 7y ago

In case someone reads this and gets the impression that running PG over NFS is in general safe or a good idea, I'm pretty sure it still isn't, unless like the OP you have a complete understanding of what you're doing.

merb 7y ago
OT: it depends on how much writes/reads you get. for most people nfs is probably an easier solution for HA on postgres than anything else.
rosser 7y ago
I have quite successfully run pg atop NFS — even, in limited, point-solution type roles, in production. My experience doing that didn't leave me with the impression that it was a particularly egregious thing to do, though I would definitely take many, many additional steps to ensure redundancy and availability if I were going to use it more generally. You're right though: you really do want to have some idea what you're doing, if you're going to go there. Source: my day job is PostgreSQL DBA, and has been for ~15 years now. EDIT: Phrasing.
- cyphar 7y ago
  The main thing to keep in mind is that NFS exercises a bunch of VFS code paths that effectively no other filesystems touches. So you are more at risk to strange VFS bugs, as well as general NFS shenanigans, which other PG users may not have seen before.
- macdice 7y ago
  I think the problem is mostly ENOSPC from fsync() which jettisons data just like EIO on Linux. If you ran out of space, PostgreSQL would only learn about that while checkpointing, and then retry and carry on. Boom, data loss. Today PostgreSQL would panic on the first ENOSPC from fsync() so the problem is mostly "fixed".
georgebarnett 7y ago
That’s pretty much computing. It’s usually the case that unless you have a complete understanding, there’s a pothole waiting to trip you. As an FYI - we no longer use Postgres on NFS as we have migrated the entire architecture underlying the systems in question.

pgaddict 7y ago

I think it's a bit strange to claim the PostgreSQL community just declined the patch. Just look at the thread on pgsql-hackers list: https://www.postgresql.org/message-id/flat/46ED7555-9383-45D2-A5C0-6373BE2FDCB6%40atlassian.com https://www.postgresql.org/message-id/flat/46ED7555-9383-45D... As anarazel already pointed out in the hackernews thread you linked, there's one person who was initially skeptical about the patch, and a couple of people chimed in with feedback (e.g. pointing out some mount options may fix it, which I think is valuable). And then the author of the patch just disappears from the discussion, so the thread kinda gets forgotten for a couple of months. I'm sure PostgreSQL community has its flaws, but I doubt this is a particularly clear example ...