6 ms·

So they say it's hard to run it in Kubernetes because they weren't running it in StatefulSets. Then they say you can actually run it properly in a StatefulSet b

by vasco 9y ago

So they say it's hard to run it in Kubernetes because they weren't running it in StatefulSets. Then they say you can actually run it properly in a StatefulSet but hand-wave it away with "but people run it on top of Ceph which has issues with latency". That sounds like a shitty excuse to just dismiss the whole thing as too hard. If one particular underlying storage system wasn't good, use a better one. Kubernetes allows you to abstract the actual storage implementation to whatever you want so it's not like there's lack of options. Also I'd be curious to see if "most people" actually use Ceph vs network storage like EBS volumes where AWS guarantees me that I won't have data corruption issues in exchange for money.

SteveNuts 9y ago
> network storage like EBS volumes where AWS guarantees me that I won't have data corruption issues in exchange for money. Source? I've never heard that claim, and in fact we've lost data on EBS (though it has been a couple years)
- jdc0589 9y ago
  seconded. As far as I know, EBS makes no claim to replicate a copy outside of your availability zone or anything to garuntee data integrity in the event of a major region wide issue
  - dsmithatx 9y ago
    EBS is replicated inside of the AZ (Availability Zone) that it is in which is what makes it durable. An AZ is a data center. You could always sync to another volume in another region if you were concerned about a longer term region outage. Note that RDS already mostly uses EBS.
twakefield 9y ago
"That sounds like a shitty excuse to just dismiss the whole thing as too hard" Disclosure: I work at the company that published this post. I read it differently (albeit, I have much more context). I read it as a cautionary tail that Kubernetes makes it easier to get in trouble if you don't know what you are doing - so you better have a deep knowledge of you stateful workloads and technology. Perhaps obvious to some, but still a good reminder when dealing with a well-hyped technology like Kubernetes.
- toomuchtodo 9y ago
  I appreciate pragmatic replies like this, providing opportunities to gracefully get off the hype train for those who bought in.
  - tyingq 9y ago
    I agree. Kubernetes seems clear that "pets" isn't their strong suit. I'm surprised at the surprise here. They will likely get better at it over time. Until then, either run it outside of K8S or deal with the warts.
aberoham 9y ago
Sasha's experience running Postgres under Kubernetes started before StatefulSets were a thing, back in the days of PetSets. At the time of his tech diligence the best option appeared to be pg/stolon https://github.com/sorintlab/stolon https://github.com/sorintlab/stolon. The gist of the interview is that there are many more edge cases with HA Postgres + leader election than you can reasonably imagine at first glance. Our experience building and running 24x7x365 HA Kubernetes clusters for clients is almost exclusively in air-gapped or on-premise environments where EBS (or even an IOPS QoS guarantee) does not exist. Most SaaS and managed service provider teams we encounter would much prefer to leverage a cloud provider's primitives than DIY.
- rattray 9y ago
  Would you say that EBS is a good solution for Postgres-on-K8s storage, for users who can tolerate Cloud storage?
oppositelock 9y ago
I run a bunch of Kubernetes microservices which use Postgres, and I treat Postgres in the same way as etcd - it runs on its own box, or RDS or CloudSQL, and the containers talk to it there. When you do it like this, you can use all your favorite postgres management tools, and not have to worry about Kubernetes doing something unexpected. If you want any semblance of DB performance from Pg as a container, you've got to give it its own data volumes, mount that into its container, etc. Not sure what the benefit of containerizing it would be. Shared tenancy on kube workers makes your performance unpredictable as well, and if you use affinity to run it on a dedicated worker, what's the point?
willejs 9y ago
I think he makes some really great points here, and obviously has some pretty good experience at running data stores in the real world, at scale. I think his comment about stateful sets is more focused about running data stores on distributed filesystems, or network attached storage. This isn't a good idea generally, and I am not going to go into why, but Cassandra and Etcd advise against this for a reason, its well documented. It wouldn't just be using a 'better' storage system, it would be designing and implementing a very complex operator as he mentions. The real take away which I think we should all realise is that, this is shiny tech to run mission critical data stores on, unless you know what your doing and want to invest a lot of time and effort in to it. Right now it's probably better to use RDS/Cloud SQL, or just run a traditional postgres setup with a SAN and decent failover that is tried and tested.
- carterehsmith 9y ago
  >> obviously has some pretty good experience at running data stores in the real world, at scale. I missed that part. Did they mention it in the article? Perhaps they should mention that kind of stuff more prominently. Numbers... and such.

by vasco 9y ago

SteveNuts 9y ago

> network storage like EBS volumes where AWS guarantees me that I won't have data corruption issues in exchange for money. Source? I've never heard that claim, and in fact we've lost data on EBS (though it has been a couple years)

jdc0589 9y ago
seconded. As far as I know, EBS makes no claim to replicate a copy outside of your availability zone or anything to garuntee data integrity in the event of a major region wide issue
- dsmithatx 9y ago
  EBS is replicated inside of the AZ (Availability Zone) that it is in which is what makes it durable. An AZ is a data center. You could always sync to another volume in another region if you were concerned about a longer term region outage. Note that RDS already mostly uses EBS.

twakefield 9y ago

"That sounds like a shitty excuse to just dismiss the whole thing as too hard" Disclosure: I work at the company that published this post. I read it differently (albeit, I have much more context). I read it as a cautionary tail that Kubernetes makes it easier to get in trouble if you don't know what you are doing - so you better have a deep knowledge of you stateful workloads and technology. Perhaps obvious to some, but still a good reminder when dealing with a well-hyped technology like Kubernetes.

toomuchtodo 9y ago
I appreciate pragmatic replies like this, providing opportunities to gracefully get off the hype train for those who bought in.
- tyingq 9y ago
  I agree. Kubernetes seems clear that "pets" isn't their strong suit. I'm surprised at the surprise here. They will likely get better at it over time. Until then, either run it outside of K8S or deal with the warts.

aberoham 9y ago

Sasha's experience running Postgres under Kubernetes started before StatefulSets were a thing, back in the days of PetSets. At the time of his tech diligence the best option appeared to be pg/stolon https://github.com/sorintlab/stolon https://github.com/sorintlab/stolon. The gist of the interview is that there are many more edge cases with HA Postgres + leader election than you can reasonably imagine at first glance. Our experience building and running 24x7x365 HA Kubernetes clusters for clients is almost exclusively in air-gapped or on-premise environments where EBS (or even an IOPS QoS guarantee) does not exist. Most SaaS and managed service provider teams we encounter would much prefer to leverage a cloud provider's primitives than DIY.

rattray 9y ago
Would you say that EBS is a good solution for Postgres-on-K8s storage, for users who can tolerate Cloud storage?

oppositelock 9y ago

I run a bunch of Kubernetes microservices which use Postgres, and I treat Postgres in the same way as etcd - it runs on its own box, or RDS or CloudSQL, and the containers talk to it there. When you do it like this, you can use all your favorite postgres management tools, and not have to worry about Kubernetes doing something unexpected. If you want any semblance of DB performance from Pg as a container, you've got to give it its own data volumes, mount that into its container, etc. Not sure what the benefit of containerizing it would be. Shared tenancy on kube workers makes your performance unpredictable as well, and if you use affinity to run it on a dedicated worker, what's the point?

willejs 9y ago

I think he makes some really great points here, and obviously has some pretty good experience at running data stores in the real world, at scale. I think his comment about stateful sets is more focused about running data stores on distributed filesystems, or network attached storage. This isn't a good idea generally, and I am not going to go into why, but Cassandra and Etcd advise against this for a reason, its well documented. It wouldn't just be using a 'better' storage system, it would be designing and implementing a very complex operator as he mentions. The real take away which I think we should all realise is that, this is shiny tech to run mission critical data stores on, unless you know what your doing and want to invest a lot of time and effort in to it. Right now it's probably better to use RDS/Cloud SQL, or just run a traditional postgres setup with a SAN and decent failover that is tried and tested.

carterehsmith 9y ago
>> obviously has some pretty good experience at running data stores in the real world, at scale. I missed that part. Did they mention it in the article? Perhaps they should mention that kind of stuff more prominently. Numbers... and such.