11 ms·

Stateful Apps on Kubernetes: A quick primer

139 points by loiselleatwork 8y ago

daxfohl 8y ago
Has anyone looked at Service Fabric (Microsoft tech) for things like this? That has offered stateful services for years now. I'm pretty sure it runs on Linux, and I've seen that it's Docker compatible. I know it's kinda in the same space as K8s but I don't really know the details. Would SF be able to do something like this in a similar (or better?) way?
- zapita 8y ago
  It's complicated, because the definition of Service Fabric seems to be in flux. The "original" Service Fabric is a high-level framework which requires invasive source code changes (you can't just drop an existing app on top of it), but gives you lots of benefits (scale, reliability etc) if you make the effort. Recently container-based platforms - Docker, Kubernetes, etc - have come along with a different tradeoff: better compatibility with existing applications in exchange for less magical benefits. That approach is getting much more traction, and I think internally at Microsoft there is some infighting between the "Service Fabric camp" and the "Containers camp". One consequence of the infighting is that Service Fabric is extending its scope to include features like "container support". It's not clear to what extent that is done in collaboration with the "container people", or as a way to bypass them. I think they are still trying to decide whether to embrace Kubernetes, or replicate the functionality in-house. My prediction is that the container-based approach will win, but if will take time for the politics to fully play out. In the meantime things will continue to be confusing. Bottom line: when evaluating Service Fabric, watch out for confusing and inconsistent use of the brand. It's a common pattern with large vendors - for example IBM with "Bluemix", SAP with "Hana", etc.
  - daxfohl 8y ago
    Okay that's about what it looked like to me too. There's only so many magic words you can throw at a tech and expect it to work together happily. Looking into it, the stateful service side of SF doesn't seem particularly compatible with the container side of it. A stateful service is a stateful SF service, and a container service is its own thing. Maybe there's a way to plug them together but unfortunately I didn't see it.
- jrbancel 8y ago
  Disclaimer: I work at Microsoft, not on Service Fabric but I have built complex stateful services on top of Service Fabric. As zapita said, Service Fabric now handles containers but I think it is just because containers became trendy and FOMO kicked in. Where Service Fabric is decades ahead of the container orchestration solutions is as a framework to build truly stateful services, meaning the state is entirely managed by your code through SF, not externalized in a remote disk, Redis, some DB, etc... It offers high level primitives like reliable collections [0], as well as very low level primitives like a replicated log to implement custom replication between replicas [1]. I feel that publicly this is not advertised enough and it is unfortunate because it is a key differentiator for Service Fabric that the competitors won't have for a while, if ever because it is a completely opposite approach: containers are all about isolation, being self-contained and plateform independent while SF stateful services are deeply integrated with Service Fabric. [0] https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-reliable-services-reliable-collections https://docs.microsoft.com/en-us/azure/service-fabric/servic... [1] https://docs.microsoft.com/en-us/dotnet/api/system.fabric.fabricreplicator?view=azure-dotnet https://docs.microsoft.com/en-us/dotnet/api/system.fabric.fa...
rrdharan 8y ago
> Because Kubernetes itself runs on the machines that are running your databases, it will consume some resources and will slightly impact performance. In our testing, we found an approximately 5% dip in throughput on a simple key-value workload. 5% seems like a surprisingly large overhead. What is k8s doing in this situation that would have that kind of impact?
- smarterclayton 8y ago
  CPU Cache contention, network overhead introduced by Kubernetes service proxy model, even the liveness checks. We haven’t yet evolved Kubernetes services to prefer specific cores and avoid app workloads quite yet (although cpu management is getting closer). Docker is also somewhat hefty memory wise and you may contend on disk if not careful. 5% seems pretty reasonable to me in general, just as a consequence of having something heavier weight on the same node managing workloads.
  - a-robinson 8y ago
    Yeah, it appeared to just be general resource contention from having to share the machine -- CPU interrupts, less memory available, etc. I'll note, though, that the 5% number is when using host networking for both Cockroach and the client load generator. Using GKE's default cluster networking through the Docker bridge is closer to 15% worse than running directly on equivalent non-Kubernetes VMs.
- nimos 8y ago
  Hard to say without knowing what they ran on what. There is a non trivial amount of memory that gets eaten up by the various k8s processes, docker and networking if you are using small nodes. I have a completely empty k8s cluster up right now with 1 worker and 1 master and the worker has about ~230 mb of ram used up.
lowbloodsugar 8y ago
>Given its pedigree of literally working at Google-scale I understood that a team at google developed k8s but google doesn't actually run it for their "google-scale" workloads. Am I misinformed?
- bajsejohannes 8y ago
  You are correct. > [kubernetes is] a simplified clone of Google’s internal borg system https://medium.com/@steve.yegge/honestly-i-cant-stand-k8s-48c9a600e405 https://medium.com/@steve.yegge/honestly-i-cant-stand-k8s-48...
  - outside1234 8y ago
    That said, Google, to my understanding, does run a completely containerized infrastructure internally, including databases and other stateful things, so it is not wildly off to suggest running a database on Kubernetes.
    - kyrra 8y ago
      Packaging was (is?) a bit different, because it existed before Docker existed. Pdf of the presentation: https://www.usenix.org/sites/default/files/conference/protected-files/lisa_2014_talk.pdf https://www.usenix.org/sites/default/files/conference/protec... And a recording of the talk: https://www.usenix.org/conference/lisa14/conference-program/presentation/mcnutt https://www.usenix.org/conference/lisa14/conference-program/...
      - puzzle 8y ago
        Googlers, including the ones working on Kubernetes, like to pick on MPM, but that's probably because they haven't run major services deployed from Docker images.
    - puzzle 8y ago
      Before it ran on F1/Spanner, Adwords ran on sharded MySQL on Borg. Not for its entire early life, but for quite a few years. Later in life, Checkout and maybe Wallet ran on MySQL on Borg, too. So did YouTube, which used Vitess, a sharding layer (now ported to Kubernetes and open sourced). Bigtable, Spanner and even Colossus/D run in containers on Borg.
bajsejohannes 8y ago
I would recommend against running stateful apps in kubernetes. It's not really ready for it. Big problems include routing (it works fine for http requests, but not for DBs, message brokers, etc) and just the pain of setting up stateful sets. If you don't believe me, take it from someone who should know what they're talking about: https://twitter.com/kelseyhightower/status/963413508300812295 https://twitter.com/kelseyhightower/status/96341350830081229...
- williamstein 8y ago
  I've used Kubernetes a massive amount during the last two years for running stateful apps. In contrast, I do recommend it. Yes, it is challenging, since stateful app are. However, the challenges are all well worth solving in the context of Kubernetes (great benefits from health checks, automated reproducible deployments, etc.). The situation is pretty good these days in my experience; at least, a lot better than 2 years ago!
  - bajsejohannes 8y ago
    Good point about it getting better. A lot of the pain was from trying to do it before 1.8.
- loiselleatwork 8y ago
  Author of post here and CRL employee: just for some additional detail, we reached out to Kelsey about the problems he's seen running databases in Kubernetes. https://twitter.com/kelseyhightower/status/963471316572561409 https://twitter.com/kelseyhightower/status/96347131657256140... He said "You still need to worry about database backups and restores. You need to consider downtime during cluster upgrades." These things are totally true. K8s doesn't automate backups (edit: by default; though, it can) and if you need to take K8s down for upgrades, then everything is down. For its part, though, CockroachDB supports rolling upgrades with no downtime on Kubernetes. As for routing, that is tough problem if you want to run K8s across multiple regions, though we have some folks who've done it. And if one finds setting up StatefulSets challenging, we have a tutorial on how to do it written by a former Kubernetes engineer: https://www.cockroachlabs.com/docs/stable/orchestrate-cockroachdb-with-kubernetes.html https://www.cockroachlabs.com/docs/stable/orchestrate-cockro...
tapirl 8y ago
Are there any cloud providers providing remote disks without replications? It looks such needs are popular for deploying databases in which replications are maintained by the databases themselves.
- jen20 8y ago
  That's effectively what EBS is, no?
  - tapirl 8y ago
    I have the impression that EBS does replication automatically, is it wrong?
stefanatfrg 8y ago
I'd like to know how to solve the storage dilution problem with stateful apps in k8s where you have to buy 3-18x more raw capacity than desired to meet availability & durability guarantees. For example if you ran CDB on a baremetal cluster of 3 nodes with 30TB of raw capacity, 15TB is lost to RAID10, 10TB is lost to running a replicated database such as cockroach DB, leaving you with 5TB effective capacity which is a 1/6 dilution of your initial capacity. If you ran cockroach DB on a replicated network volume, with a replication factor of three, it gets worse. If you bought 30 TB of disks, you'd lose 20 TB to volume replication, ~6.67TB to CDB replication leaving you with 3.3TB of effective capacity or a 1/9 dilution. If those disks were configured with RAID your effective capacity would drop to a 1/18 dilution. You could achieve a 1/3 dilution which is the effective minimum for a replicated database if you didn't configure RAID, but you increase the impact of disk failure, in that it would take much much longer to recover a cluster.

Has anyone looked at Service Fabric (Microsoft tech) for things like this? That has offered stateful services for years now. I'm pretty sure it runs on Linux, and I've seen that it's Docker compatible. I know it's kinda in the same space as K8s but I don't really know the details. Would SF be able to do something like this in a similar (or better?) way?

zapita 8y ago
It's complicated, because the definition of Service Fabric seems to be in flux. The "original" Service Fabric is a high-level framework which requires invasive source code changes (you can't just drop an existing app on top of it), but gives you lots of benefits (scale, reliability etc) if you make the effort. Recently container-based platforms - Docker, Kubernetes, etc - have come along with a different tradeoff: better compatibility with existing applications in exchange for less magical benefits. That approach is getting much more traction, and I think internally at Microsoft there is some infighting between the "Service Fabric camp" and the "Containers camp". One consequence of the infighting is that Service Fabric is extending its scope to include features like "container support". It's not clear to what extent that is done in collaboration with the "container people", or as a way to bypass them. I think they are still trying to decide whether to embrace Kubernetes, or replicate the functionality in-house. My prediction is that the container-based approach will win, but if will take time for the politics to fully play out. In the meantime things will continue to be confusing. Bottom line: when evaluating Service Fabric, watch out for confusing and inconsistent use of the brand. It's a common pattern with large vendors - for example IBM with "Bluemix", SAP with "Hana", etc.
- daxfohl 8y ago
  Okay that's about what it looked like to me too. There's only so many magic words you can throw at a tech and expect it to work together happily. Looking into it, the stateful service side of SF doesn't seem particularly compatible with the container side of it. A stateful service is a stateful SF service, and a container service is its own thing. Maybe there's a way to plug them together but unfortunately I didn't see it.
jrbancel 8y ago
Disclaimer: I work at Microsoft, not on Service Fabric but I have built complex stateful services on top of Service Fabric. As zapita said, Service Fabric now handles containers but I think it is just because containers became trendy and FOMO kicked in. Where Service Fabric is decades ahead of the container orchestration solutions is as a framework to build truly stateful services, meaning the state is entirely managed by your code through SF, not externalized in a remote disk, Redis, some DB, etc... It offers high level primitives like reliable collections [0], as well as very low level primitives like a replicated log to implement custom replication between replicas [1]. I feel that publicly this is not advertised enough and it is unfortunate because it is a key differentiator for Service Fabric that the competitors won't have for a while, if ever because it is a completely opposite approach: containers are all about isolation, being self-contained and plateform independent while SF stateful services are deeply integrated with Service Fabric. [0] https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-reliable-services-reliable-collections https://docs.microsoft.com/en-us/azure/service-fabric/servic... [1] https://docs.microsoft.com/en-us/dotnet/api/system.fabric.fabricreplicator?view=azure-dotnet https://docs.microsoft.com/en-us/dotnet/api/system.fabric.fa...

rrdharan 8y ago

> Because Kubernetes itself runs on the machines that are running your databases, it will consume some resources and will slightly impact performance. In our testing, we found an approximately 5% dip in throughput on a simple key-value workload. 5% seems like a surprisingly large overhead. What is k8s doing in this situation that would have that kind of impact?

smarterclayton 8y ago
CPU Cache contention, network overhead introduced by Kubernetes service proxy model, even the liveness checks. We haven’t yet evolved Kubernetes services to prefer specific cores and avoid app workloads quite yet (although cpu management is getting closer). Docker is also somewhat hefty memory wise and you may contend on disk if not careful. 5% seems pretty reasonable to me in general, just as a consequence of having something heavier weight on the same node managing workloads.
- a-robinson 8y ago
  Yeah, it appeared to just be general resource contention from having to share the machine -- CPU interrupts, less memory available, etc. I'll note, though, that the 5% number is when using host networking for both Cockroach and the client load generator. Using GKE's default cluster networking through the Docker bridge is closer to 15% worse than running directly on equivalent non-Kubernetes VMs.
nimos 8y ago
Hard to say without knowing what they ran on what. There is a non trivial amount of memory that gets eaten up by the various k8s processes, docker and networking if you are using small nodes. I have a completely empty k8s cluster up right now with 1 worker and 1 master and the worker has about ~230 mb of ram used up.

lowbloodsugar 8y ago

>Given its pedigree of literally working at Google-scale I understood that a team at google developed k8s but google doesn't actually run it for their "google-scale" workloads. Am I misinformed?

bajsejohannes 8y ago
You are correct. > [kubernetes is] a simplified clone of Google’s internal borg system https://medium.com/@steve.yegge/honestly-i-cant-stand-k8s-48c9a600e405 https://medium.com/@steve.yegge/honestly-i-cant-stand-k8s-48...
- outside1234 8y ago
  That said, Google, to my understanding, does run a completely containerized infrastructure internally, including databases and other stateful things, so it is not wildly off to suggest running a database on Kubernetes.
  - kyrra 8y ago
    Packaging was (is?) a bit different, because it existed before Docker existed. Pdf of the presentation: https://www.usenix.org/sites/default/files/conference/protected-files/lisa_2014_talk.pdf https://www.usenix.org/sites/default/files/conference/protec... And a recording of the talk: https://www.usenix.org/conference/lisa14/conference-program/presentation/mcnutt https://www.usenix.org/conference/lisa14/conference-program/...
    - puzzle 8y ago
      Googlers, including the ones working on Kubernetes, like to pick on MPM, but that's probably because they haven't run major services deployed from Docker images.
  - puzzle 8y ago
    Before it ran on F1/Spanner, Adwords ran on sharded MySQL on Borg. Not for its entire early life, but for quite a few years. Later in life, Checkout and maybe Wallet ran on MySQL on Borg, too. So did YouTube, which used Vitess, a sharding layer (now ported to Kubernetes and open sourced). Bigtable, Spanner and even Colossus/D run in containers on Borg.

bajsejohannes 8y ago

I would recommend against running stateful apps in kubernetes. It's not really ready for it. Big problems include routing (it works fine for http requests, but not for DBs, message brokers, etc) and just the pain of setting up stateful sets. If you don't believe me, take it from someone who should know what they're talking about: https://twitter.com/kelseyhightower/status/963413508300812295 https://twitter.com/kelseyhightower/status/96341350830081229...

williamstein 8y ago
I've used Kubernetes a massive amount during the last two years for running stateful apps. In contrast, I do recommend it. Yes, it is challenging, since stateful app are. However, the challenges are all well worth solving in the context of Kubernetes (great benefits from health checks, automated reproducible deployments, etc.). The situation is pretty good these days in my experience; at least, a lot better than 2 years ago!
- bajsejohannes 8y ago
  Good point about it getting better. A lot of the pain was from trying to do it before 1.8.
loiselleatwork 8y ago
Author of post here and CRL employee: just for some additional detail, we reached out to Kelsey about the problems he's seen running databases in Kubernetes. https://twitter.com/kelseyhightower/status/963471316572561409 https://twitter.com/kelseyhightower/status/96347131657256140... He said "You still need to worry about database backups and restores. You need to consider downtime during cluster upgrades." These things are totally true. K8s doesn't automate backups (edit: by default; though, it can) and if you need to take K8s down for upgrades, then everything is down. For its part, though, CockroachDB supports rolling upgrades with no downtime on Kubernetes. As for routing, that is tough problem if you want to run K8s across multiple regions, though we have some folks who've done it. And if one finds setting up StatefulSets challenging, we have a tutorial on how to do it written by a former Kubernetes engineer: https://www.cockroachlabs.com/docs/stable/orchestrate-cockroachdb-with-kubernetes.html https://www.cockroachlabs.com/docs/stable/orchestrate-cockro...

tapirl 8y ago

Are there any cloud providers providing remote disks without replications? It looks such needs are popular for deploying databases in which replications are maintained by the databases themselves.

jen20 8y ago
That's effectively what EBS is, no?
- tapirl 8y ago
  I have the impression that EBS does replication automatically, is it wrong?

stefanatfrg 8y ago

I'd like to know how to solve the storage dilution problem with stateful apps in k8s where you have to buy 3-18x more raw capacity than desired to meet availability & durability guarantees. For example if you ran CDB on a baremetal cluster of 3 nodes with 30TB of raw capacity, 15TB is lost to RAID10, 10TB is lost to running a replicated database such as cockroach DB, leaving you with 5TB effective capacity which is a 1/6 dilution of your initial capacity. If you ran cockroach DB on a replicated network volume, with a replication factor of three, it gets worse. If you bought 30 TB of disks, you'd lose 20 TB to volume replication, ~6.67TB to CDB replication leaving you with 3.3TB of effective capacity or a 1/9 dilution. If those disks were configured with RAID your effective capacity would drop to a 1/18 dilution. You could achieve a 1/3 dilution which is the effective minimum for a replicated database if you didn't configure RAID, but you increase the impact of disk failure, in that it would take much much longer to recover a cluster.