4 ms·

Acknowledging the risk of sounding like "why can't I use rsync scripts instead of this Dropbox?", Can someone explain to me like I just got out of high school,

Acknowledging the risk of sounding like "why can't I use rsync scripts instead of this Dropbox?", Can someone explain to me like I just got out of high school, why do I need to complicate my setup with Kubernetes/Docker Swarm/Deis/Dolly/Flynn instead of spinning up&down containers using scripted LXC commands? Someone was talking about using Kubernetes for data pipelines. For such a case, why is Kubernetes better than a script that spins up pre-made container images? Not being snark at all. Genuinely asking a "why?". Appreciate any educating responses!

drdaeman 9y ago
Maybe you don't. I wouldn't recommend anyone to use something extremely complex for mission critical architecture, unless unavoidable. If you feel the same, it's probably a perfectly sound reason to avoid Kubernetes if there's no necessity. When I wrote "extremely complex" I did't mean all that models and concepts from user documentation - they're the easy part (although I haven't used K8s much and probably missing a lot about its user-facing parts). I mean the internals. Because when things break and cluster goes down it's the internals that matter. I use Docker instead of plain LXC because it adds convenience of layers and versioning while not introducing any significant complexity. I had some issues with libnetwork (which sometimes behaved oddly when containers crash under high load or docker-proxy gets somehow stuck) but I can live with it. I also prefer Swarm to manual networking. It's trivial to set up and just works under normal circumstances. And I feel confident that I can fix issues reasonably fast. I had experimented with it under some odd scenarios (like a heterogeneous cluster with a mix of x86_64 and armv7h nodes) and when it had (obviously) failed, I was able to quickly find the relevant source code, read it, understand the details and correct the problem (which was trivial but not documented). And if I wouldn't be able to, I'm absolutely certain I would be able to scrap the cluster and script reprovisioning with standalone nodes, Compose and ad-hoc VPNs - I did this when Rancher (old version) had failed me badly. Now, maybe I'm just too stupid but I can't say this for Kubernetes. I've tried to experiment with it - also under explicitly unsupported conditions - and tried to dig into its documentation and code when it had failed (obviously it did, I was asking for it). However, I was literally overwhelmed by how much there is. What I've learned is a little tiny bit about CNI internals but the primary result was a conclusion "sorry but nope, that's not something I feel comfortable dealing with if I'm to support it". And I don't mean that it fail-prone or anything - just that every software project has its bugs, and sometimes they manifest themselves at really inconvenient times. On the other hand, Kubernetes is very nice when it's not you who's responsible for its operation but you're just an user. Spin a cluster on GKE and you know Google would take care of it - you'd just have to write some YAML to describe your project and enjoy. In this scenario, I think I can recommend it - even though I haven't ran anything serious on K8s but some toy projects worked without issues and were easy to deploy and maintain. And Kubernetes feels more mature than other options, having lots of nice functionality, supposedly covering a lot of different use cases and scenarios. E.g. Swarm feels like it lacks a lot of stuff - I've subscribed to a lot of tickets for features that I wanted but that aren't yet there. Just a personal opinion.
DigitalJack 9y ago
I agree with the "because it's fun to learn" answers. But I'd also add the reliability factor of keeping a quorum of containers running. It's a tool for dealing with the unreliability of cloud infrastructure, and unexpected process death.
dguaraglia 9y ago
In general, I tend to think that any tooling I can avoid writing (and documenting, and debugging, and maintaining...) is an advantage. As someone who has done the whole "running my own containers from bash scripts and cronjobs", Kubernetes (and in particular the hosted Kubernetes clusters provided by Google Cloud, Azure and AWS) does a great job of managing a lot of crap I don't want to deal with, such as: - Networking setup and service discovery - Managing container failure - Secret storage - Deployments (in particular, sensibly-managed Rolling Deployments) - Container-cluster size management - Ingress management (load balancers are a pain in the ass to manage from bash scripts) - Resource monitoring (Kubernetes comes with a nice interface to watch the cluster and modify it dynamically) There's nothing stopping you from building all of that from scratch, but why bother?
nielsole 9y ago
Kubernetes is not supposed to be run on a single node, but on dozens or hundreds of machines. Scheduling, networking, and authorizing heterogeneous highly available workloads by possibly different teams + managing ingress + managing networked storage is what Kubernetes is built for. I wouldn't know how to do this with LXC
- colek42 9y ago
  The API for k8s is great. We mostly run large (50 core 500GB memory) machines co-located at client sites. We have hundreds of these to manage, but they are separated geographically. K8's API/Infrastructure would be great to manage/monitor them.
  - nielsole 9y ago
    Sounds like an interesting problem. What is your use-case? Caching servers in ISPs? Kubernetes kind of assumes that any node can fail any time and reschedules workloads. This assumption might not be in line with your requirements.
donaldguy 9y ago
Mostly the answer is "if you want to do it all yourself, go for it"; Things like this, and especially Kubernetes, are about creating a common and agreed upon language for infrastructure - both patterns and specific "off the shelf" applications It's very nice to be able to start a Proof of Concept data pipeline with `helm install incubator/kafka` (https://github.com/kubernetes/charts/tree/master/incubator/kafka https://github.com/kubernetes/charts/tree/master/incubator/k...) And likewise the common language/interface creates opportunities for wider coverage and easier adoption in tooling for monitoring, autoscaling, smart routing (see "service mesh"es), etc. You may not need all of k8s functionality immediately but it's the agreed upon common core for modern distributed systems functionality And to everyone's credit it's also being done pretty much as modularly and extensiblely as we could hope for
xorcist 9y ago
Well, if you took the risk of sounding snarky, I might as well take the risk of offending a number of projects as well. The following is my personal opinion only. From touching this stack quite a bit over the past year, my lasting impression is that most people don't actually need it, but that at the same time one shouldn't decry the value of toying with a new tech stack. (By the way, I get the same impression from many Hadoop users who carry this large infrastructural debt from building out much too early.) Many people gravitate towards these things in order to learn them better, which can be wise on a personal level should they become more popular. For me personally, I think the value with Kubernetes is if it becomes a provider independent cloud platform. Bascially what Openstack couldn't deliver, Kubernetes takes the core of and delivers an orchestration platform on that is actually useful. So while you shouldn't invest resources in it solely because it's a new stack, the design is sane and the community is there.
akvadrako 9y ago
Why would you write scripts or learn LXC commands when you can just write a k8 deployment? Even if you don't consider all the more advanced features, it's just easier and declarative and supported natively by several cloud providers.
manishjhawar 9y ago
I've been in your shoes using LXC way back but moved to Docker fairly quickly when it came out, so I kinda get your position. Now, Kubernetes/Docker provide better tooling with added benefits. Docker provides a Dockerfile based declarative approach of building the setup and storing it in an image and then running it as a container with greater flexibility and more features than a typical LXC based script. Kubernetes takes the running of multiple containers to another level capable of managing 100s or even 1000s of them for scalable operations. RTFM for the details :-) HTH!

drdaeman 9y ago

Maybe you don't. I wouldn't recommend anyone to use something extremely complex for mission critical architecture, unless unavoidable. If you feel the same, it's probably a perfectly sound reason to avoid Kubernetes if there's no necessity. When I wrote "extremely complex" I did't mean all that models and concepts from user documentation - they're the easy part (although I haven't used K8s much and probably missing a lot about its user-facing parts). I mean the internals. Because when things break and cluster goes down it's the internals that matter. I use Docker instead of plain LXC because it adds convenience of layers and versioning while not introducing any significant complexity. I had some issues with libnetwork (which sometimes behaved oddly when containers crash under high load or docker-proxy gets somehow stuck) but I can live with it. I also prefer Swarm to manual networking. It's trivial to set up and just works under normal circumstances. And I feel confident that I can fix issues reasonably fast. I had experimented with it under some odd scenarios (like a heterogeneous cluster with a mix of x86_64 and armv7h nodes) and when it had (obviously) failed, I was able to quickly find the relevant source code, read it, understand the details and correct the problem (which was trivial but not documented). And if I wouldn't be able to, I'm absolutely certain I would be able to scrap the cluster and script reprovisioning with standalone nodes, Compose and ad-hoc VPNs - I did this when Rancher (old version) had failed me badly. Now, maybe I'm just too stupid but I can't say this for Kubernetes. I've tried to experiment with it - also under explicitly unsupported conditions - and tried to dig into its documentation and code when it had failed (obviously it did, I was asking for it). However, I was literally overwhelmed by how much there is. What I've learned is a little tiny bit about CNI internals but the primary result was a conclusion "sorry but nope, that's not something I feel comfortable dealing with if I'm to support it". And I don't mean that it fail-prone or anything - just that every software project has its bugs, and sometimes they manifest themselves at really inconvenient times. On the other hand, Kubernetes is very nice when it's not you who's responsible for its operation but you're just an user. Spin a cluster on GKE and you know Google would take care of it - you'd just have to write some YAML to describe your project and enjoy. In this scenario, I think I can recommend it - even though I haven't ran anything serious on K8s but some toy projects worked without issues and were easy to deploy and maintain. And Kubernetes feels more mature than other options, having lots of nice functionality, supposedly covering a lot of different use cases and scenarios. E.g. Swarm feels like it lacks a lot of stuff - I've subscribed to a lot of tickets for features that I wanted but that aren't yet there. Just a personal opinion.

DigitalJack 9y ago

I agree with the "because it's fun to learn" answers. But I'd also add the reliability factor of keeping a quorum of containers running. It's a tool for dealing with the unreliability of cloud infrastructure, and unexpected process death.

dguaraglia 9y ago

In general, I tend to think that any tooling I can avoid writing (and documenting, and debugging, and maintaining...) is an advantage. As someone who has done the whole "running my own containers from bash scripts and cronjobs", Kubernetes (and in particular the hosted Kubernetes clusters provided by Google Cloud, Azure and AWS) does a great job of managing a lot of crap I don't want to deal with, such as: - Networking setup and service discovery - Managing container failure - Secret storage - Deployments (in particular, sensibly-managed Rolling Deployments) - Container-cluster size management - Ingress management (load balancers are a pain in the ass to manage from bash scripts) - Resource monitoring (Kubernetes comes with a nice interface to watch the cluster and modify it dynamically) There's nothing stopping you from building all of that from scratch, but why bother?

nielsole 9y ago

Kubernetes is not supposed to be run on a single node, but on dozens or hundreds of machines. Scheduling, networking, and authorizing heterogeneous highly available workloads by possibly different teams + managing ingress + managing networked storage is what Kubernetes is built for. I wouldn't know how to do this with LXC

colek42 9y ago
The API for k8s is great. We mostly run large (50 core 500GB memory) machines co-located at client sites. We have hundreds of these to manage, but they are separated geographically. K8's API/Infrastructure would be great to manage/monitor them.
- nielsole 9y ago
  Sounds like an interesting problem. What is your use-case? Caching servers in ISPs? Kubernetes kind of assumes that any node can fail any time and reschedules workloads. This assumption might not be in line with your requirements.

donaldguy 9y ago

Mostly the answer is "if you want to do it all yourself, go for it"; Things like this, and especially Kubernetes, are about creating a common and agreed upon language for infrastructure - both patterns and specific "off the shelf" applications It's very nice to be able to start a Proof of Concept data pipeline with `helm install incubator/kafka` (https://github.com/kubernetes/charts/tree/master/incubator/kafka https://github.com/kubernetes/charts/tree/master/incubator/k...) And likewise the common language/interface creates opportunities for wider coverage and easier adoption in tooling for monitoring, autoscaling, smart routing (see "service mesh"es), etc. You may not need all of k8s functionality immediately but it's the agreed upon common core for modern distributed systems functionality And to everyone's credit it's also being done pretty much as modularly and extensiblely as we could hope for

xorcist 9y ago

Well, if you took the risk of sounding snarky, I might as well take the risk of offending a number of projects as well. The following is my personal opinion only. From touching this stack quite a bit over the past year, my lasting impression is that most people don't actually need it, but that at the same time one shouldn't decry the value of toying with a new tech stack. (By the way, I get the same impression from many Hadoop users who carry this large infrastructural debt from building out much too early.) Many people gravitate towards these things in order to learn them better, which can be wise on a personal level should they become more popular. For me personally, I think the value with Kubernetes is if it becomes a provider independent cloud platform. Bascially what Openstack couldn't deliver, Kubernetes takes the core of and delivers an orchestration platform on that is actually useful. So while you shouldn't invest resources in it solely because it's a new stack, the design is sane and the community is there.

akvadrako 9y ago

Why would you write scripts or learn LXC commands when you can just write a k8 deployment? Even if you don't consider all the more advanced features, it's just easier and declarative and supported natively by several cloud providers.

manishjhawar 9y ago

I've been in your shoes using LXC way back but moved to Docker fairly quickly when it came out, so I kinda get your position. Now, Kubernetes/Docker provide better tooling with added benefits. Docker provides a Dockerfile based declarative approach of building the setup and storing it in an image and then running it as a container with greater flexibility and more features than a typical LXC based script. Kubernetes takes the running of multiple containers to another level capable of managing 100s or even 1000s of them for scalable operations. RTFM for the details :-) HTH!