6 ms·

Having used Docker Compose/Swarm for last two years, I remember having problems with them twice. One of which was an MTU setting which I didn't really understan

Having used Docker Compose/Swarm for last two years, I remember having problems with them twice. One of which was an MTU setting which I didn't really understand why, but overall I was relatively happy with them. Since Kubernetes seems to have won, I decided to learn it but got some disappointments. The first disappointment is setting up a local development environment. I failed to get minikube running on a Macbook Air 2013 and a Ubuntu Thinkpad. Both have VTx enabled and Docker and VirtualBox running flawlessly. Their online interactive tutorial was good though, enough for the learning purpose. Production setup is a bigger disappointment. The only easy and reliable ways to have a production grade Kubernetes cluster are to lock yourself into either a big player cloud provider, or an enterprise OS (Redhat/Ubuntu), or introduce a new layer on top of Kubernetes [1]. Locking myself into enterprise Ubuntu/Redhad is expensive, and I'm not comfortable with adding a new, moving, unreliable layer on top of Kubernestes which is built on top of Docker. One thing I like about the Docker movement is that they commoditize infrastructure and reduce lock-ins. I can design my infrastructure so it can utilize an open source based cloud product first and easily move to others or self-host if needed. With Kubernetes, things are going the other way. Even if I never moved out of the big 3 (AWS/Azure/GCloud), the migration process could be painful since their Kubernetes may introduce further lock-ins for logging, monitoring, and so on. [1]: https://kubernetes.io/docs/setup/pick-right-solution/ https://kubernetes.io/docs/setup/pick-right-solution/

hjacobs 8y ago
I never used Docker Swarm (so can't compare), but I don't fully understand your point about Kubernetes cloud lock-in. Certainly there are important differences in networking, load balancing, persistent volumes, and other cloud features, but that's not something any platform can just hide/eliminate (e.g. think about AWS ELB/ALB/NLB vs Google Load Balancer). The Kubernetes concepts (Deployment, Ingress, Service) still work mostly the same for the user across clouds. Some other details like non-standardized Ingress annotations are obviously due to not having them agreed in Kubernetes core API (nginx ingress supports other annotations than say Google LB or Skipper).
- bonesss 8y ago
  Most of the kubernetes toolchain provides nice support for delineating the requirements from separate cloud providers, too. Compared to most alternatives, a little HELM magic to support hybrid cloud installations is a piece of cake.
- fhrow4484 8y ago
  > I don't fully understand your point about Kubernetes cloud lock-in The kubernetes folks describe a tentative solution to cloud lock-in here: https://kubernetes.io/docs/concepts/cluster-administration/federation/#why-federation https://kubernetes.io/docs/concepts/cluster-administration/f... OP isn't the only one with those concerns. It would be nice when you can switch your cluster load from any of the cloud providers, or your own on-prem setup as you go. For instance, I could see people wanting to have a default small cluster on their on-prem setup, and be ready to scale on cloud when needed.
  - dankohn1 8y ago
    Federation is a solution for multi-cloud and hybrid cloud, which is a different problem than cloud portability.
  - parasubvert 8y ago
    Kubernetes federation doesn’t really help with lock-in. Its for multi-cluster management. It’s also kind of stalled / going to be rethought - it has too many issues to be used at scale. Some may still like it of course. There are two points of lock-in: beneath Kubernetes or on top of Kubernetes. You still need to install and manage your cluster, so the low level lock-in of a distro is hard to avoid. What you may be looking for is kubeadm. https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm/ https://kubernetes.io/docs/reference/setup-tools/kubeadm/kub... It is the closest primitive to help standardize “here’s how to install and run a K8s cluster in a certified way”. It’s unfortunately incomplete. As for what’s on top of Kubernetes, that will be an area ripe for completion and lock-in.
pepemon 8y ago
Out of curiosity, why do you need RHEL or subscription from Canonical for the production Kubernetes setup? What's wrong with plain Ubuntu or CentOS?
- y4mi 8y ago
  You probably don't want to configure kubernetes manually... Kops is a thing though, but it's still a lot error potential, if you want to go to production
  - codereflection 8y ago
    This. We use Rancher within our data center, because configuring and managing k8s is not trivial. Rancher eases a lot of that pain for our small team.
- parasubvert 8y ago
  It’s common to pay for things to make them easier to configure/manage. Red Hat OpenShift on RHEL, Pivotal Container Service on Ubuntu, Red Hat’s nextgen CoreOS based Kubernetes, Canonical’s Charmed Kubernetes Distribution on Ubuntu, etc. all have different config management , install, upgrade, patching mechanisms that vary from Ansible, to Terraform, to BOSH, to Juju. Some handle PXE bare metal, some don’t. Etc. There usually are free / no pay versions of the above that you can use self-supported, but then you’ll also need to coordinate your own upgrades and use community forums for q&a rather than being able to contractually have someone looking out for you and answering your questions. If you’d prefer to avoid lock-in, All of that plumbing would otherwise have to be configured and scripted yourself with your chosen toolchain plus the newer “k8s small tools” like Kubeadm, Kops, Kube-spray, etc. As the old saying goes, open source is only free (as in beer) if your time has no value.
  - jcoffland 8y ago
    > It’s common to pay for things to make them easier to configure/manage. Yes but in the case of RedHat specifically those goals are not achieved.
dankohn1 8y ago
> The only easy and reliable ways to have a production grade Kubernetes cluster are to lock yourself into either a big player cloud provider, or an enterprise OS (Redhat/Ubuntu), or introduce a new layer on top of Kubernetes [1] I think you might have misunderstood that page. The standard and universal way to deploy Kubernetes on to either your own bare metal or any cloud provider is to use kubeadm. However, if you would like a simpler and more automated solution and/or one backed by a vendor, you are welcome to pick any of the hosted platforms, distributions, or installers. CNCF has certified 70 conformant solutions: https://www.cncf.io/certification/software-conformance/ https://www.cncf.io/certification/software-conformance/ > Even if I never moved out of the big 3 (AWS/Azure/GCloud), the migration process could be painful since their Kubernetes may introduce further lock-ins for logging, monitoring, and so on. If you choose open source solutions for logging and monitoring like Fluentd and Prometheus, then you can avoid locking into anyone's value added services and remain completely portable. If you decide to go with a vendor's solution, you may trade convenience for higher switching costs. [1]: https://kubernetes.io/docs/setup/pick-right-solution/ https://kubernetes.io/docs/setup/pick-right-solution/ Disclosure: I'm executive director of CNCF and run the conformance program.
- pquerna 8y ago
  The point is not about the minimum conformance, but rather the lock-in provided by the maximum configuration / extensions of each vendor. Take AWS EKS as an example. Their feature page[1] does mention conformance. Then it mentions 20 other non-conformance focused features that create an effective lock-in. k8s is becoming like OpenStack in this regards. You need to embrace a vendor version of k8s in order to have a functional cluster without a massive team. [1] - https://aws.amazon.com/eks/features/ https://aws.amazon.com/eks/features/
  - devthane 8y ago
    This isn’t my experience at all. I as one person taught myself over the past couple years docker then Kubernetes and am now managing a small 3 node bare metal cluster on my own. But using rancher 2.0 has helped a bunch to ease me into it. Now I feel comfortable enough to start up a cluster on my own without it.
tyingq 8y ago
Digital Ocean's K8S offering is out of beta now: https://www.digitalocean.com/products/kubernetes/ https://www.digitalocean.com/products/kubernetes/
- clvx 8y ago
  It’s good, but the storage layer has some bugs. For example, if you create a pvc, then resize the volume according to their docs, the new size doesn’t reflect in k8s. Also you can create pv’s manually and it won’t show up in the dashboard.
  - farslan 8y ago
    Hi, I'm the maintainer of the csi-digitalocean driver. Thanks for the feedback! Unfortunately "resizing" is still not supported by csi yet (the resizer sidecar is still in development), hence there is no way to provide the resize functionality to the customers. Once the `external-resizer` sidecar is finalized, it'll be part of the csi-digitalocean driver. When you say dashboard, which dashboard do you mean? Happy to look at it. Thanks again for the feedback.
- hjacobs 8y ago
  DO K8s is pretty neat, but last time I checked it did not have the metrics server (CPU/mem metrics, also "kubectl top") yet.
- karakanb 8y ago
  Migrated my very small cluster from GKE to DigitalOcean's K8s a few weeks ago. I was using 3 nodes on GKE with 1 core & 3.75GB RAM per node, and the cost was around 100 $ per month including load balancer for the cheapest region, `us-central1-a`. Now, on DigitalOcean, I have 3 nodes with 1 core & 2GB RAM per node. The cost is exactly 40$ including load balancer. I am a pretty basic user, I have started using k8s on this project as a learning and 100$ was too much for the learning price, but now on DO I get a similar cluster for less than half of GKE price and I feel like it is worth it, considering all the simplicity and observability of deployments. Also, DO allows me to select regions without any price difference, so I was able to select Amsterdam to get 10 times better latency from where I live. My setup is quite basic, my app with aroud 8-10 pods, + additional stuff such as cert-manager and prometheus. YMMV, but so far I am really happy with DO's offering, both in terms of performance, simplicity and performance. I am not a power user and definitely operate at no scale, but using DO in general is much simpler than using GCP with GKE.
DavidWoof 8y ago
I've had small-ish docker swarms in production for a couple of years as well, and I really don't understand why it doesn't seem to be popular at all. I feel like I need to move to K8S just because swarm seems to be going away, but I'm really not seeing the technical advantages at all. If someone could point me to an article explaining why k8s is so much better than swarm, I'd really appreciate it. Are the big advantages only at 100-node scales?
- clofresh 8y ago
  I'm also constantly surprised at how unpopular docker swarm is given that everyone already uses docker itself. Why do you think swarm is going away though? I love the idea of just using my docker compose file as my deployment config.
  - hjacobs 8y ago
    I'm not familiar with Docker Swarm, but the Kubernetes API is the major strong point for me (not saying that it's perfect): it has the right abstractions (CronJob, StatefulSet, ..) and is extensible (Custom Resource Definitions). There are many ways to run containerized workloads (ECS, Mesos, Docker Swarm, ..), but the de-facto agreement on the Kubernetes API is a game changer: now we can start building things on top of it :-)
  - jamougha 8y ago
    My company tried using Swarm 2-3 years ago and ran into the problem that it didn't actually work. Containers would just go missing from the network. Consequently we switched to Kubernetes. I imagine it does work now but it seems to be too late. I've recently started using Kompose to autogenerate Helm charts from docker compose files and I've found that pretty satisfactory.
  - InTheArena 8y ago
    The second version of Swarm (the first version built into the stack) was really unstable. It got a bad reputation very quickly in one area that it could not afford to. That said, I was at a conference, talking with a CIO of a K8s related company, back during the initial days of that startup ecosystem. I was told off the record that Google and Redhat were offering an "marketing budget" for companies that would come on board the Kubernetes ecosystem. The person rightly stated that K8s was going to roll everyone else because of it. Can't validate that - it's hearsay, but I definitely think that it was a big factor
romeisendcoming 8y ago
Or you could just learn how to manage infrastructure the old fashioned way, which was never broken for small business and mid-sized enterprise environments. The only time you need the complexity and overhead of something like kubernetes is when you are truly large or when you have caught the in-fashion disease. It's quite simple for a 20 year SA to stand up a highly integrated environment with modular monitoring, directory service, virtualization and hybrid cloud options for all services in a week. Why don't you hire one of these for the job instead of recipe/containering yourself into 'doesn't work, I dunno' posts.
- Tehchops 8y ago
  >Why don't you hire one of these for the job instead of recipe/containering yourself into 'doesn't work, I dunno' posts. Because every.single.one of these "integrated environments" I've ever come across was an objective mess, poorly documented and littered with tech-debt. It was clear the "20-year SA" had forced 20-year old administration abstractions and ideas on top of modern infrastructure and application concerns. It was cheaper/better/easier to throw it out and rebuild on something like k8s than to make any attempting at "scaling" the existing solution. You're simply trading the "in-fashion" disease for the "I'm a 20-year Linuxbeard I know best and no one tells me different" disease.
  - romeisendcoming 8y ago
    There is some of that, certainly, but you can misapply any solution and perform poorly using any tool or personality. For your information I find the new breed and k8s FUD to be more of a reaction rather than an educated, seamless and practical systems approach.
- manigandham 8y ago
  Why not hire an SA? Because K8S is free and runs well from small to large and is available on every cloud where the IT infrastructure already is. Why is it better to spend money to rebuild a fraction of K8S with a patchwork of infrastructure put together by a single person?
jordanbeiber 8y ago
The kubeadm api for ”phases” going beta in 1.12 -> 1.13 has actually made rolling k8s clusters (almost) a breeze. It used to be really clunky, but these days all you need is a simple bash script or ansible play (or whatever you’re comfortable with) to get going. But yeah, no unix philosphy vibes from k8s as a whole...
resouer 8y ago
Kubernetes does not provide functionalities like logging & monitoring. The way how this works is totally a bunch of open source solutions like Prometheus & Fluentd. Actually, I barely saw any Kubernetes cloud provider provides meaningful service which can lock myself in, they are basically managed Kubernetes clusters with their cloud services as plugins. You can verify this by comparing GKE/AKE/EKS, you'll find they are almost same thing.

hjacobs 8y ago

I never used Docker Swarm (so can't compare), but I don't fully understand your point about Kubernetes cloud lock-in. Certainly there are important differences in networking, load balancing, persistent volumes, and other cloud features, but that's not something any platform can just hide/eliminate (e.g. think about AWS ELB/ALB/NLB vs Google Load Balancer). The Kubernetes concepts (Deployment, Ingress, Service) still work mostly the same for the user across clouds. Some other details like non-standardized Ingress annotations are obviously due to not having them agreed in Kubernetes core API (nginx ingress supports other annotations than say Google LB or Skipper).

bonesss 8y ago
Most of the kubernetes toolchain provides nice support for delineating the requirements from separate cloud providers, too. Compared to most alternatives, a little HELM magic to support hybrid cloud installations is a piece of cake.
fhrow4484 8y ago
> I don't fully understand your point about Kubernetes cloud lock-in The kubernetes folks describe a tentative solution to cloud lock-in here: https://kubernetes.io/docs/concepts/cluster-administration/federation/#why-federation https://kubernetes.io/docs/concepts/cluster-administration/f... OP isn't the only one with those concerns. It would be nice when you can switch your cluster load from any of the cloud providers, or your own on-prem setup as you go. For instance, I could see people wanting to have a default small cluster on their on-prem setup, and be ready to scale on cloud when needed.
- dankohn1 8y ago
  Federation is a solution for multi-cloud and hybrid cloud, which is a different problem than cloud portability.
- parasubvert 8y ago
  Kubernetes federation doesn’t really help with lock-in. Its for multi-cluster management. It’s also kind of stalled / going to be rethought - it has too many issues to be used at scale. Some may still like it of course. There are two points of lock-in: beneath Kubernetes or on top of Kubernetes. You still need to install and manage your cluster, so the low level lock-in of a distro is hard to avoid. What you may be looking for is kubeadm. https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm/ https://kubernetes.io/docs/reference/setup-tools/kubeadm/kub... It is the closest primitive to help standardize “here’s how to install and run a K8s cluster in a certified way”. It’s unfortunately incomplete. As for what’s on top of Kubernetes, that will be an area ripe for completion and lock-in.

pepemon 8y ago

Out of curiosity, why do you need RHEL or subscription from Canonical for the production Kubernetes setup? What's wrong with plain Ubuntu or CentOS?

y4mi 8y ago
You probably don't want to configure kubernetes manually... Kops is a thing though, but it's still a lot error potential, if you want to go to production
- codereflection 8y ago
  This. We use Rancher within our data center, because configuring and managing k8s is not trivial. Rancher eases a lot of that pain for our small team.
parasubvert 8y ago
It’s common to pay for things to make them easier to configure/manage. Red Hat OpenShift on RHEL, Pivotal Container Service on Ubuntu, Red Hat’s nextgen CoreOS based Kubernetes, Canonical’s Charmed Kubernetes Distribution on Ubuntu, etc. all have different config management , install, upgrade, patching mechanisms that vary from Ansible, to Terraform, to BOSH, to Juju. Some handle PXE bare metal, some don’t. Etc. There usually are free / no pay versions of the above that you can use self-supported, but then you’ll also need to coordinate your own upgrades and use community forums for q&a rather than being able to contractually have someone looking out for you and answering your questions. If you’d prefer to avoid lock-in, All of that plumbing would otherwise have to be configured and scripted yourself with your chosen toolchain plus the newer “k8s small tools” like Kubeadm, Kops, Kube-spray, etc. As the old saying goes, open source is only free (as in beer) if your time has no value.
- jcoffland 8y ago
  > It’s common to pay for things to make them easier to configure/manage. Yes but in the case of RedHat specifically those goals are not achieved.

dankohn1 8y ago

> The only easy and reliable ways to have a production grade Kubernetes cluster are to lock yourself into either a big player cloud provider, or an enterprise OS (Redhat/Ubuntu), or introduce a new layer on top of Kubernetes [1] I think you might have misunderstood that page. The standard and universal way to deploy Kubernetes on to either your own bare metal or any cloud provider is to use kubeadm. However, if you would like a simpler and more automated solution and/or one backed by a vendor, you are welcome to pick any of the hosted platforms, distributions, or installers. CNCF has certified 70 conformant solutions: https://www.cncf.io/certification/software-conformance/ https://www.cncf.io/certification/software-conformance/ > Even if I never moved out of the big 3 (AWS/Azure/GCloud), the migration process could be painful since their Kubernetes may introduce further lock-ins for logging, monitoring, and so on. If you choose open source solutions for logging and monitoring like Fluentd and Prometheus, then you can avoid locking into anyone's value added services and remain completely portable. If you decide to go with a vendor's solution, you may trade convenience for higher switching costs. [1]: https://kubernetes.io/docs/setup/pick-right-solution/ https://kubernetes.io/docs/setup/pick-right-solution/ Disclosure: I'm executive director of CNCF and run the conformance program.

pquerna 8y ago
The point is not about the minimum conformance, but rather the lock-in provided by the maximum configuration / extensions of each vendor. Take AWS EKS as an example. Their feature page[1] does mention conformance. Then it mentions 20 other non-conformance focused features that create an effective lock-in. k8s is becoming like OpenStack in this regards. You need to embrace a vendor version of k8s in order to have a functional cluster without a massive team. [1] - https://aws.amazon.com/eks/features/ https://aws.amazon.com/eks/features/
- devthane 8y ago
  This isn’t my experience at all. I as one person taught myself over the past couple years docker then Kubernetes and am now managing a small 3 node bare metal cluster on my own. But using rancher 2.0 has helped a bunch to ease me into it. Now I feel comfortable enough to start up a cluster on my own without it.

tyingq 8y ago

Digital Ocean's K8S offering is out of beta now: https://www.digitalocean.com/products/kubernetes/ https://www.digitalocean.com/products/kubernetes/

clvx 8y ago
It’s good, but the storage layer has some bugs. For example, if you create a pvc, then resize the volume according to their docs, the new size doesn’t reflect in k8s. Also you can create pv’s manually and it won’t show up in the dashboard.
- farslan 8y ago
  Hi, I'm the maintainer of the csi-digitalocean driver. Thanks for the feedback! Unfortunately "resizing" is still not supported by csi yet (the resizer sidecar is still in development), hence there is no way to provide the resize functionality to the customers. Once the `external-resizer` sidecar is finalized, it'll be part of the csi-digitalocean driver. When you say dashboard, which dashboard do you mean? Happy to look at it. Thanks again for the feedback.
hjacobs 8y ago
DO K8s is pretty neat, but last time I checked it did not have the metrics server (CPU/mem metrics, also "kubectl top") yet.
karakanb 8y ago
Migrated my very small cluster from GKE to DigitalOcean's K8s a few weeks ago. I was using 3 nodes on GKE with 1 core & 3.75GB RAM per node, and the cost was around 100 $ per month including load balancer for the cheapest region, `us-central1-a`. Now, on DigitalOcean, I have 3 nodes with 1 core & 2GB RAM per node. The cost is exactly 40$ including load balancer. I am a pretty basic user, I have started using k8s on this project as a learning and 100$ was too much for the learning price, but now on DO I get a similar cluster for less than half of GKE price and I feel like it is worth it, considering all the simplicity and observability of deployments. Also, DO allows me to select regions without any price difference, so I was able to select Amsterdam to get 10 times better latency from where I live. My setup is quite basic, my app with aroud 8-10 pods, + additional stuff such as cert-manager and prometheus. YMMV, but so far I am really happy with DO's offering, both in terms of performance, simplicity and performance. I am not a power user and definitely operate at no scale, but using DO in general is much simpler than using GCP with GKE.

DavidWoof 8y ago

I've had small-ish docker swarms in production for a couple of years as well, and I really don't understand why it doesn't seem to be popular at all. I feel like I need to move to K8S just because swarm seems to be going away, but I'm really not seeing the technical advantages at all. If someone could point me to an article explaining why k8s is so much better than swarm, I'd really appreciate it. Are the big advantages only at 100-node scales?

clofresh 8y ago
I'm also constantly surprised at how unpopular docker swarm is given that everyone already uses docker itself. Why do you think swarm is going away though? I love the idea of just using my docker compose file as my deployment config.
- hjacobs 8y ago
  I'm not familiar with Docker Swarm, but the Kubernetes API is the major strong point for me (not saying that it's perfect): it has the right abstractions (CronJob, StatefulSet, ..) and is extensible (Custom Resource Definitions). There are many ways to run containerized workloads (ECS, Mesos, Docker Swarm, ..), but the de-facto agreement on the Kubernetes API is a game changer: now we can start building things on top of it :-)
- jamougha 8y ago
  My company tried using Swarm 2-3 years ago and ran into the problem that it didn't actually work. Containers would just go missing from the network. Consequently we switched to Kubernetes. I imagine it does work now but it seems to be too late. I've recently started using Kompose to autogenerate Helm charts from docker compose files and I've found that pretty satisfactory.
- InTheArena 8y ago
  The second version of Swarm (the first version built into the stack) was really unstable. It got a bad reputation very quickly in one area that it could not afford to. That said, I was at a conference, talking with a CIO of a K8s related company, back during the initial days of that startup ecosystem. I was told off the record that Google and Redhat were offering an "marketing budget" for companies that would come on board the Kubernetes ecosystem. The person rightly stated that K8s was going to roll everyone else because of it. Can't validate that - it's hearsay, but I definitely think that it was a big factor

romeisendcoming 8y ago

Or you could just learn how to manage infrastructure the old fashioned way, which was never broken for small business and mid-sized enterprise environments. The only time you need the complexity and overhead of something like kubernetes is when you are truly large or when you have caught the in-fashion disease. It's quite simple for a 20 year SA to stand up a highly integrated environment with modular monitoring, directory service, virtualization and hybrid cloud options for all services in a week. Why don't you hire one of these for the job instead of recipe/containering yourself into 'doesn't work, I dunno' posts.

Tehchops 8y ago
>Why don't you hire one of these for the job instead of recipe/containering yourself into 'doesn't work, I dunno' posts. Because every.single.one of these "integrated environments" I've ever come across was an objective mess, poorly documented and littered with tech-debt. It was clear the "20-year SA" had forced 20-year old administration abstractions and ideas on top of modern infrastructure and application concerns. It was cheaper/better/easier to throw it out and rebuild on something like k8s than to make any attempting at "scaling" the existing solution. You're simply trading the "in-fashion" disease for the "I'm a 20-year Linuxbeard I know best and no one tells me different" disease.
- romeisendcoming 8y ago
  There is some of that, certainly, but you can misapply any solution and perform poorly using any tool or personality. For your information I find the new breed and k8s FUD to be more of a reaction rather than an educated, seamless and practical systems approach.
manigandham 8y ago
Why not hire an SA? Because K8S is free and runs well from small to large and is available on every cloud where the IT infrastructure already is. Why is it better to spend money to rebuild a fraction of K8S with a patchwork of infrastructure put together by a single person?

jordanbeiber 8y ago

The kubeadm api for ”phases” going beta in 1.12 -> 1.13 has actually made rolling k8s clusters (almost) a breeze. It used to be really clunky, but these days all you need is a simple bash script or ansible play (or whatever you’re comfortable with) to get going. But yeah, no unix philosphy vibes from k8s as a whole...

resouer 8y ago

Kubernetes does not provide functionalities like logging & monitoring. The way how this works is totally a bunch of open source solutions like Prometheus & Fluentd. Actually, I barely saw any Kubernetes cloud provider provides meaningful service which can lock myself in, they are basically managed Kubernetes clusters with their cloud services as plugins. You can verify this by comparing GKE/AKE/EKS, you'll find they are almost same thing.