5 ms·

Slightly OT, but, from a quick glance at the post, it looks like the postgres data is persisted on a network file system. Does anybody have any experience with

Slightly OT, but, from a quick glance at the post, it looks like the postgres data is persisted on a network file system. Does anybody have any experience with a big enough and this kind of configuration, and what is the performance compared with SSD local disks? Intuitively I wouldn't expect a networked disk to be a viable solution for postgres, but seeing it proposed like this makes me think that maybe my intuition was wrong...

travem 10y ago
There would no doubt be some performance differences between local and networked but it is very normal in enterprises to deploy databases on a SAN to take advantage of the array's data services. On the NAS side I believe Oracle also are often deployed on NetApp arrays, they use their own NFS client to ensure the DB isn't compromised by bad NFS client implementations.
TheIronYuppie 10y ago
QQ - how do you deal with the situation where a node goes away? GCP offers networked SSD, and performance is fairly good, but, as always, YMMV. Disclosure: I work at Google on Kubernetes
- detaro 10y ago
  I think the posters point is that the article talks about using NFS or other shared network filesystems, not a block storage protocol?
lobster_johnson 10y ago
People have run Postgres on network disks — Elastic Block Storage (EBS) volumes on AWS, or Persistent Disks on Google Cloud — for a long time. They're implemented like a software SAN, with dedicated network paths, and throughput/latency is very good. Not as good as local, but still good enough for most apps. On Google Cloud, you do have to option of using a local SSD [1], but it comes with a bunch of limitations. It maxes out at 375GB (though you can allocate up to 8 per VM). They're somewhat expensive. They're not durable; if you stop the VM, the disk is lost. They're classified as "scratch disks" in the UI, which is a good indicator of their intended purpose. Edit: Using a networked file system, however, is a bad idea. [1] https://cloud.google.com/compute/docs/disks/local-ssd https://cloud.google.com/compute/docs/disks/local-ssd
- koolba 10y ago
  > People have run Postgres on network disks — Elastic Block Storage (EBS) volumes on AWS, or Persistent Disks on Google Cloud — for a long time. NFS != SAN The former is a shared file system accessible by multiple machines simultaneously. The latter is a block device accessed over the network by a single machine at a time. It's generally a bad idea to run a database on NFS as it'll be fine until it's not. Network block devices are fine though as it's just like a disk except it "plugs in" through your network jack rather than something like SATA.
  - lobster_johnson 10y ago
    The parent poster referred to both "networked disk" and "networked file system". I didn't catch the implied reference to NFS, so I assumed he was talking about the former, so that's what my comment is about. I agree that NFS is a terrible idea for databases.
koolba 10y ago
Forget performance, I'd be more worried about file locking, file fsync, and directory fsync. All are critical for a database and an "optimized" NFS driver that lies about the latter two can lead to corruption or data loss.
- dboreham 10y ago
  Looking at the other comments here : how quickly people forget everything!
  - koolba 10y ago
    Those who don't learn the lessons of what's wrong trying to jam everything atop NFS are doomed to repeat them.
foxylion 10y ago
When running PostgreSQL in any cloud provider most of your data is served from network. I would say performance varies, but quite good. In comparison to magnetic disks much better. Most workloads you should be able to handle with network mounted storage.
jsjohnst 10y ago
Depends on your performance tolerance. Networked IO can be an order of magnitude faster or more than a local spinning disk, but sure, network latency will likely never top the performance of cutting edge SSD used locally. As a case in point, I've pushed greater than 600mb/s of simultaneous heavy read and write traffic to an AWS EBS volume with acceptably low mean latency, which is more than enough for many Postgres use cases, but there's definitely cases where even that level IO would be a major bottleneck.
- barrkel 10y ago
  EBS isn't a network file system.
dijit 10y ago
I err.. never thought I'd be talking about this seriously. We (3 companies ago) used to use postgresql databases backed by a huge netapp filer. Performance was as you would expect, horrible but tolderable. Why was it tolerable? Because high availability was more important than performance at the time and native HA wasn't in PostgreSQL yet. (8.1 had just come out) The first DBA ever on staff was an absurdly knowledgable guy (and paid handsomely for being so knowledgeable) and his first and strongest recommendation was, in this order: 1) Upgrade to the latest PostgreSQL to get a better vacuum implementation. 2) Get the data off the filer. I highly advise against it unless your application caches tremendously well.
barrkel 10y ago
Networked disks are fairly normal for production databases - fibre channel SANs are networked disks - but this is quite different to network file systems.
vidarh 10y ago
Assuming you mean a network block device rather than a network filesystem (be very, very vary about network filesystems that claim sufficient POSIX compliance to run a database; it may work, but test very carefully and see what people say about your specific setup as it's very easy to get it wrong and end up with corrupted data), it'll work quite well, but it will typicall be awfully performance wise unless you spend a fortune working around bottlenecks that won't be there in most local setups. It's a performance vs. availability/reliability tradeoff, and it's not a straightforward one.
- danmaz74 10y ago
  The post explicitly says "Network File System": > The example in this blog uses NFS for the Persistent Volumes, but any shared file system would also work (ex: ceph, gluster).
  - lobster_johnson 10y ago
    If one has Ceph in place, of course, the wisest choice is to the Ceph block device service, which would be akin to using EBS or a SAN.
  - vidarh 10y ago
    I must admit I just skimmed over the initial parts of the post. NFS is possible, but it's very easy to get the settings wrong, and it will depend on your specific NFS server whether or not you risk losing data in corner cases... I'd not consider NFS for a database for that reason without knowing precisely which NFS server and which version and testing that specific combination extensively (or verifying others have experience with exactly that combination). It will be as slow as a block store, but most of the block stores offer up much simpler semantics that are much easier to verify if makes sense for a database workload (they need to store the blocks they claim to have stored, and that's pretty much what we care about; for the rest it's down to trusting a well tested local filesystem). As an added note, the "any shared file system would also work" is patently false and horrible, dangerous advice, for a database deployment as it depends entirely on the file systems crash semantics in terms of what data is guaranteed to hit disk in a durable way, and whether or not there's any chance of writes being reordered etc.
brazzledazzle 10y ago
If you can maintain network quality (latency, throughput, packet loss) the tradeoffs of using a SAN/NAS solution can be well worth it. For example it makes moving services+storage between compute a trivial affair. To (inexactly) translate SAN/NAS into AWS terms for those that may not be familiar with these types of on-prem deployments: S3/EFS/Glacier is NAS–files are stored and retrieved over the network. EBS is SAN–the storage is exposed as a block level device over the network and appear as locally attached devices.
crb 10y ago
Generally you would get a network attached block storage disk (Google Cloud Persistent Disk, EBS volume, etc). The blog writes from the perspective of an on-prem deployment and suggests you could use NFS to provide a PersistentVolume to Kubernetes. While this is possible, it's probably not best practice, especially if a failure in your NFS will render all your nodes offline at once.

travem 10y ago

There would no doubt be some performance differences between local and networked but it is very normal in enterprises to deploy databases on a SAN to take advantage of the array's data services. On the NAS side I believe Oracle also are often deployed on NetApp arrays, they use their own NFS client to ensure the DB isn't compromised by bad NFS client implementations.

TheIronYuppie 10y ago

QQ - how do you deal with the situation where a node goes away? GCP offers networked SSD, and performance is fairly good, but, as always, YMMV. Disclosure: I work at Google on Kubernetes

detaro 10y ago
I think the posters point is that the article talks about using NFS or other shared network filesystems, not a block storage protocol?

lobster_johnson 10y ago

People have run Postgres on network disks — Elastic Block Storage (EBS) volumes on AWS, or Persistent Disks on Google Cloud — for a long time. They're implemented like a software SAN, with dedicated network paths, and throughput/latency is very good. Not as good as local, but still good enough for most apps. On Google Cloud, you do have to option of using a local SSD [1], but it comes with a bunch of limitations. It maxes out at 375GB (though you can allocate up to 8 per VM). They're somewhat expensive. They're not durable; if you stop the VM, the disk is lost. They're classified as "scratch disks" in the UI, which is a good indicator of their intended purpose. Edit: Using a networked file system, however, is a bad idea. [1] https://cloud.google.com/compute/docs/disks/local-ssd https://cloud.google.com/compute/docs/disks/local-ssd

koolba 10y ago
> People have run Postgres on network disks — Elastic Block Storage (EBS) volumes on AWS, or Persistent Disks on Google Cloud — for a long time. NFS != SAN The former is a shared file system accessible by multiple machines simultaneously. The latter is a block device accessed over the network by a single machine at a time. It's generally a bad idea to run a database on NFS as it'll be fine until it's not. Network block devices are fine though as it's just like a disk except it "plugs in" through your network jack rather than something like SATA.
- lobster_johnson 10y ago
  The parent poster referred to both "networked disk" and "networked file system". I didn't catch the implied reference to NFS, so I assumed he was talking about the former, so that's what my comment is about. I agree that NFS is a terrible idea for databases.

koolba 10y ago

Forget performance, I'd be more worried about file locking, file fsync, and directory fsync. All are critical for a database and an "optimized" NFS driver that lies about the latter two can lead to corruption or data loss.

dboreham 10y ago
Looking at the other comments here : how quickly people forget everything!
- koolba 10y ago
  Those who don't learn the lessons of what's wrong trying to jam everything atop NFS are doomed to repeat them.

foxylion 10y ago

When running PostgreSQL in any cloud provider most of your data is served from network. I would say performance varies, but quite good. In comparison to magnetic disks much better. Most workloads you should be able to handle with network mounted storage.

jsjohnst 10y ago

Depends on your performance tolerance. Networked IO can be an order of magnitude faster or more than a local spinning disk, but sure, network latency will likely never top the performance of cutting edge SSD used locally. As a case in point, I've pushed greater than 600mb/s of simultaneous heavy read and write traffic to an AWS EBS volume with acceptably low mean latency, which is more than enough for many Postgres use cases, but there's definitely cases where even that level IO would be a major bottleneck.

barrkel 10y ago
EBS isn't a network file system.

dijit 10y ago

I err.. never thought I'd be talking about this seriously. We (3 companies ago) used to use postgresql databases backed by a huge netapp filer. Performance was as you would expect, horrible but tolderable. Why was it tolerable? Because high availability was more important than performance at the time and native HA wasn't in PostgreSQL yet. (8.1 had just come out) The first DBA ever on staff was an absurdly knowledgable guy (and paid handsomely for being so knowledgeable) and his first and strongest recommendation was, in this order: 1) Upgrade to the latest PostgreSQL to get a better vacuum implementation. 2) Get the data off the filer. I highly advise against it unless your application caches tremendously well.

barrkel 10y ago

Networked disks are fairly normal for production databases - fibre channel SANs are networked disks - but this is quite different to network file systems.

vidarh 10y ago

Assuming you mean a network block device rather than a network filesystem (be very, very vary about network filesystems that claim sufficient POSIX compliance to run a database; it may work, but test very carefully and see what people say about your specific setup as it's very easy to get it wrong and end up with corrupted data), it'll work quite well, but it will typicall be awfully performance wise unless you spend a fortune working around bottlenecks that won't be there in most local setups. It's a performance vs. availability/reliability tradeoff, and it's not a straightforward one.

danmaz74 10y ago
The post explicitly says "Network File System": > The example in this blog uses NFS for the Persistent Volumes, but any shared file system would also work (ex: ceph, gluster).
- lobster_johnson 10y ago
  If one has Ceph in place, of course, the wisest choice is to the Ceph block device service, which would be akin to using EBS or a SAN.
- vidarh 10y ago
  I must admit I just skimmed over the initial parts of the post. NFS is possible, but it's very easy to get the settings wrong, and it will depend on your specific NFS server whether or not you risk losing data in corner cases... I'd not consider NFS for a database for that reason without knowing precisely which NFS server and which version and testing that specific combination extensively (or verifying others have experience with exactly that combination). It will be as slow as a block store, but most of the block stores offer up much simpler semantics that are much easier to verify if makes sense for a database workload (they need to store the blocks they claim to have stored, and that's pretty much what we care about; for the rest it's down to trusting a well tested local filesystem). As an added note, the "any shared file system would also work" is patently false and horrible, dangerous advice, for a database deployment as it depends entirely on the file systems crash semantics in terms of what data is guaranteed to hit disk in a durable way, and whether or not there's any chance of writes being reordered etc.

brazzledazzle 10y ago

If you can maintain network quality (latency, throughput, packet loss) the tradeoffs of using a SAN/NAS solution can be well worth it. For example it makes moving services+storage between compute a trivial affair. To (inexactly) translate SAN/NAS into AWS terms for those that may not be familiar with these types of on-prem deployments: S3/EFS/Glacier is NAS–files are stored and retrieved over the network. EBS is SAN–the storage is exposed as a block level device over the network and appear as locally attached devices.

crb 10y ago

Generally you would get a network attached block storage disk (Google Cloud Persistent Disk, EBS volume, etc). The blog writes from the perspective of an on-prem deployment and suggests you could use NFS to provide a PersistentVolume to Kubernetes. While this is possible, it's probably not best practice, especially if a failure in your NFS will render all your nodes offline at once.