4 ms·

Your definition of Kubernetes is accurate for maybe 5 years ago. These days Kubernetes is perfectly capable of running stateful services and supports them as a

Your definition of Kubernetes is accurate for maybe 5 years ago. These days Kubernetes is perfectly capable of running stateful services and supports them as a first class citizen.

crabbone 3y ago
Nothing can be further from the truth. Kubernetes has nothing to offer to anyone that wants to work with storage (but there's a myriad of CSIs). Here, let me give you an example of a system that offers storage in a similar way to how Kubernetes offers... well, it isn't really that good at offering compute, but, at least it kind of does it. So, Ceph -- that's something that makes sense to run PostgreSQL on (it's a storage provider). Kubernetes isn't a storage provider. It doesn't know how to manage it even... I.e. if you think that you run PosgreSQL on Kubernetes -- you are mistaken. Something else does it. Kubernetes is a proxy there, at best (but probably is completely irrelevant).
- qaq 3y ago
  Look at benchmarks of Postgres on Ceph
  - crabbone 3y ago
    Part of my job is to measure storage performance... I can tell you at leas this: there cannot be a meaningful "benchmark of Postgres on Ceph". Too many things will influence the benchmark way too much. You need to be a lot more specific when you talk about such benchmarks. Here are some things you will need to present: * Are OSDs connected to the node being tested through network or are they closer (NVMe / SAS / SATA?). If network, what's the bandwidth? What's the latency? What if it's stuff like reliable Ethernet that's used for iSCSI / NVMe over IP or something like that? * How much memory (relative to data at rest) does the node have? * What is the layout of memory buffers in PostgreSQL? * What is the setting used for synchronization in PostgreSQL? * How much replication is going to happen (Ceph pool size)? * Block sizes and frame sizes. * Type of workload. Surprisingly, some queries can exploit parallelism in I/O while other queries cannot. Surprisingly some queries will need a lot of synchronization while others don't. And there's more, it would be too tedious to try to give an exhaustive list of things to control for. And the problem is, at least those mentioned here can influence performance sometimes up to an order of magnitude, sometimes two orders of magnitude...
    - rjzzleep 3y ago
      I've run into all of these issues. In the end it's much cheaper to just buy a cheap SAN than it is probably to pay a month of your expertise. I guess that's why I struggle making cash-cow consulting gigs, because my clients are never long term dependent on me.
      - gbartolini 3y ago
        Why not dedicate some worker nodes using taints/tolerations/labels, even on bare metal, with locally attached storage? I wrote this many years ago now but that's the reason why we started CloudNativePG (OpenEBS might not be the answer today, but there are many storage engines now, including topolvm which brings LVM to the game): https://www.2ndquadrant.com/en/blog/local-persistent-volumes-and-postgresql-usage-in-kubernetes/ https://www.2ndquadrant.com/en/blog/local-persistent-volumes... It is ultimately your choice. I am a big fan of shared nothing architecture for the database. (I am a maintainer of CloudNativePG)
        sebasmannem 3y ago
        Yeah, and let Postgres take care of redundancy. I agree that this is an interesting proposition. AFAIK PortWorkx could do a similar thing, but then with storage redundancy. Basically: - storage is synced to 3 local storage devices spread across 3 different k8s nodes. This could be NVMe. - pod is only scheduled next to one of the three - reads are local, writes are local (for fsync) and synchronised to the other devices. I would love to test with pg_tps_optimizer against Portworkx
      - sebasmannem 3y ago
        I also only run short term. I enable customers with designing and implementing the proper (CEPH e.a.) environment for their workloads. But I don't run their systems. I always handover to the rest of the technical staff. Bringing in a cheap SAN basically shifts that responsibility from people like me to the SAN vendor, and they could bring in hardware that might do the job. Running on K8s, I feel you need two types of storage: - Block storage, with proper fsync (fast and reliable) - S3 storage Both MUST be CloudNative. I don't know if Cheap SAN is available with proper k8s CSI providers, but if they are, they could be up to the challenge. Note that people like me can enable customers with both (choosing the proper 'cheap SAN', but also designing a proper storage environment with CEPH or other software storage solutions.
  - sebasmannem 3y ago
    We have run benchmarks on our environment (CNPG, Openshift and CEPH in Dutch gov) and compared to Azure Postgres and (CNPG on) Azure AKS. Pgbench and pg_tps_optimizer. CEPH indeed is a 'high bandwidth / high latency' storage solution, and as such we could get to comparable TPS but required more clients. With 2 vCPU the max TPS was about 17k/18k with AKS and also with OpenShift. But with AKS we required 34 parallel clients and with OpenShift/CEPH we required 233 clients. More clients <=> more in parallel <=> more TPS on CEPH... If you are interested I can share some graphs.
- erulabs 3y ago
  To quote you: “nothing could be further from the truth” Ceph itself runs very well on K8s - see the Rook project. Of course you can run psql on k8s. Psql doesn’t need a storage orchestration system - hell it technically doesn’t need any storage at all!
  - crabbone 3y ago
    Rook is about using CSI... it doesn't run Ceph on Kubernetes. That's impossible because Ceph relies on functionality that exists in drivers (kernel modules) to run. CSI is the component that communicates between eg. rbd driver and the user-space (eg. Kubernetes controllers), but it doesn't run Ceph. It's in principle impossible to do anything about block devices in containers like those used by Kubernetes because those rely on Linux processes and associated namespaces. There isn't a Linux namespace for block devices, the closest you can get is the filesystem namespace. In other words, you cannot manage block devices purely in containers, you need some help from the host operating system. And this is why I mentioned CSIs in my previous post.
    - erulabs 3y ago
      It does run Ceph on Kubernetes. How else would you describe deploying OSDs to linux servers via Kubernetes other than "Ceph on Kubernetes"? > That's impossible because Ceph relies on functionality that exists in drivers (kernel modules) to run This statement doesn't make sense. All linux applications require kernel functionality. Yes, to deploy Ceph, you must run Linux systems with the desired kernel modules. Turns out, Rook sets that up for ya! This statement exposes a somewhat deep misunderstanding of what Kubernetes is. I run into you in every thread that mentions k8s and I sense extreme vitriol and a huge lack of experience / understanding. Don't mistake my future lack of replies for an unsaid "you've misunderstood".
rjzzleep 3y ago
Whenever someone runs Kubernetes onprem I tell them to buy a TrueNAS or another cheap SAN. A cheap SAN costs as much as a DevOps expert setting up your Ceph infrastructure and a lot less when you actually run into issues with that software defined storage solution. Once you do that Kubernetes is actually quite nice, because it gives you a base configuration of postgres that comes with automatic backup setup etc. That advice is rarely taken though ...
- ahoka 3y ago
  Solid advice. What driver/provisioner works fine for this?
  - rjzzleep 3y ago
    There used to be the external-storage-provisioner[1] but what you need today is the second link[2] [1] https://github.com/kubernetes-retired/external-storage/tree/master https://github.com/kubernetes-retired/external-storage/tree/... [2] https://github.com/kubernetes-sigs/sig-storage-lib-external-provisioner https://github.com/kubernetes-sigs/sig-storage-lib-external-...

Your definition of Kubernetes is accurate for maybe 5 years ago. These days Kubernetes is perfectly capable of running stateful services and supports them as a first class citizen.

crabbone 3y ago

Nothing can be further from the truth. Kubernetes has nothing to offer to anyone that wants to work with storage (but there's a myriad of CSIs). Here, let me give you an example of a system that offers storage in a similar way to how Kubernetes offers... well, it isn't really that good at offering compute, but, at least it kind of does it. So, Ceph -- that's something that makes sense to run PostgreSQL on (it's a storage provider). Kubernetes isn't a storage provider. It doesn't know how to manage it even... I.e. if you think that you run PosgreSQL on Kubernetes -- you are mistaken. Something else does it. Kubernetes is a proxy there, at best (but probably is completely irrelevant).

qaq 3y ago
Look at benchmarks of Postgres on Ceph
- crabbone 3y ago
  Part of my job is to measure storage performance... I can tell you at leas this: there cannot be a meaningful "benchmark of Postgres on Ceph". Too many things will influence the benchmark way too much. You need to be a lot more specific when you talk about such benchmarks. Here are some things you will need to present: * Are OSDs connected to the node being tested through network or are they closer (NVMe / SAS / SATA?). If network, what's the bandwidth? What's the latency? What if it's stuff like reliable Ethernet that's used for iSCSI / NVMe over IP or something like that? * How much memory (relative to data at rest) does the node have? * What is the layout of memory buffers in PostgreSQL? * What is the setting used for synchronization in PostgreSQL? * How much replication is going to happen (Ceph pool size)? * Block sizes and frame sizes. * Type of workload. Surprisingly, some queries can exploit parallelism in I/O while other queries cannot. Surprisingly some queries will need a lot of synchronization while others don't. And there's more, it would be too tedious to try to give an exhaustive list of things to control for. And the problem is, at least those mentioned here can influence performance sometimes up to an order of magnitude, sometimes two orders of magnitude...
  - rjzzleep 3y ago
    I've run into all of these issues. In the end it's much cheaper to just buy a cheap SAN than it is probably to pay a month of your expertise. I guess that's why I struggle making cash-cow consulting gigs, because my clients are never long term dependent on me.
    - gbartolini 3y ago
      Why not dedicate some worker nodes using taints/tolerations/labels, even on bare metal, with locally attached storage? I wrote this many years ago now but that's the reason why we started CloudNativePG (OpenEBS might not be the answer today, but there are many storage engines now, including topolvm which brings LVM to the game): https://www.2ndquadrant.com/en/blog/local-persistent-volumes-and-postgresql-usage-in-kubernetes/ https://www.2ndquadrant.com/en/blog/local-persistent-volumes... It is ultimately your choice. I am a big fan of shared nothing architecture for the database. (I am a maintainer of CloudNativePG)
      - sebasmannem 3y ago
        Yeah, and let Postgres take care of redundancy. I agree that this is an interesting proposition. AFAIK PortWorkx could do a similar thing, but then with storage redundancy. Basically: - storage is synced to 3 local storage devices spread across 3 different k8s nodes. This could be NVMe. - pod is only scheduled next to one of the three - reads are local, writes are local (for fsync) and synchronised to the other devices. I would love to test with pg_tps_optimizer against Portworkx
    - sebasmannem 3y ago
      I also only run short term. I enable customers with designing and implementing the proper (CEPH e.a.) environment for their workloads. But I don't run their systems. I always handover to the rest of the technical staff. Bringing in a cheap SAN basically shifts that responsibility from people like me to the SAN vendor, and they could bring in hardware that might do the job. Running on K8s, I feel you need two types of storage: - Block storage, with proper fsync (fast and reliable) - S3 storage Both MUST be CloudNative. I don't know if Cheap SAN is available with proper k8s CSI providers, but if they are, they could be up to the challenge. Note that people like me can enable customers with both (choosing the proper 'cheap SAN', but also designing a proper storage environment with CEPH or other software storage solutions.
- sebasmannem 3y ago
  We have run benchmarks on our environment (CNPG, Openshift and CEPH in Dutch gov) and compared to Azure Postgres and (CNPG on) Azure AKS. Pgbench and pg_tps_optimizer. CEPH indeed is a 'high bandwidth / high latency' storage solution, and as such we could get to comparable TPS but required more clients. With 2 vCPU the max TPS was about 17k/18k with AKS and also with OpenShift. But with AKS we required 34 parallel clients and with OpenShift/CEPH we required 233 clients. More clients <=> more in parallel <=> more TPS on CEPH... If you are interested I can share some graphs.
erulabs 3y ago
To quote you: “nothing could be further from the truth” Ceph itself runs very well on K8s - see the Rook project. Of course you can run psql on k8s. Psql doesn’t need a storage orchestration system - hell it technically doesn’t need any storage at all!
- crabbone 3y ago
  Rook is about using CSI... it doesn't run Ceph on Kubernetes. That's impossible because Ceph relies on functionality that exists in drivers (kernel modules) to run. CSI is the component that communicates between eg. rbd driver and the user-space (eg. Kubernetes controllers), but it doesn't run Ceph. It's in principle impossible to do anything about block devices in containers like those used by Kubernetes because those rely on Linux processes and associated namespaces. There isn't a Linux namespace for block devices, the closest you can get is the filesystem namespace. In other words, you cannot manage block devices purely in containers, you need some help from the host operating system. And this is why I mentioned CSIs in my previous post.
  - erulabs 3y ago
    It does run Ceph on Kubernetes. How else would you describe deploying OSDs to linux servers via Kubernetes other than "Ceph on Kubernetes"? > That's impossible because Ceph relies on functionality that exists in drivers (kernel modules) to run This statement doesn't make sense. All linux applications require kernel functionality. Yes, to deploy Ceph, you must run Linux systems with the desired kernel modules. Turns out, Rook sets that up for ya! This statement exposes a somewhat deep misunderstanding of what Kubernetes is. I run into you in every thread that mentions k8s and I sense extreme vitriol and a huge lack of experience / understanding. Don't mistake my future lack of replies for an unsaid "you've misunderstood".

rjzzleep 3y ago

Whenever someone runs Kubernetes onprem I tell them to buy a TrueNAS or another cheap SAN. A cheap SAN costs as much as a DevOps expert setting up your Ceph infrastructure and a lot less when you actually run into issues with that software defined storage solution. Once you do that Kubernetes is actually quite nice, because it gives you a base configuration of postgres that comes with automatic backup setup etc. That advice is rarely taken though ...

ahoka 3y ago
Solid advice. What driver/provisioner works fine for this?
- rjzzleep 3y ago
  There used to be the external-storage-provisioner[1] but what you need today is the second link[2] [1] https://github.com/kubernetes-retired/external-storage/tree/master https://github.com/kubernetes-retired/external-storage/tree/... [2] https://github.com/kubernetes-sigs/sig-storage-lib-external-provisioner https://github.com/kubernetes-sigs/sig-storage-lib-external-...