18 ms·

Teaching Kubernetes

223 points by ahawkins 10y ago

amq 10y ago
The first thing I'd like to be taught is where to use Kubernetes and where to avoid it (many people tend to think it's the best practice to force everything into containers, and ideally, Kubernetes).
- ben_jones 10y ago
  I say this only partially in jest but, I hope when presented with such choices I always pick the one that allows me the most hours of peaceful sleep at night. This usually gets me half way there during decision making.
  - StavrosK 10y ago
    Why even partially in jest? Isn't the whole point of ops "sleeplessness avoidance"?
    - ben_jones 10y ago
      Honestly ops is about making/saving money for the company. Like an HR employee can choose the right thing, an ops engineer can also choose "sleeplessness avoidance", but it doesn't always work out that way.
      - StavrosK 10y ago
        That's true, but "sleeplessness avoidance" saves lots more money for small-scale companies. Downtime and man-hours are much more expensive than getting more reliable hosting, unless you're a very large company.
- voiceclonr 10y ago
  I tend to agree with you. Often, I see some of these stack decisions are forced by decision makers who want their design to look modern.
  - FatalBaboon 10y ago
    Which is not a bad call, looking modern helps recruit. It's great that technology x lets you build something in 1 month, but if it takes 6 months to find someone willing to do it, then you could have written it faster in almost any other technology.
  - ownagefool 10y ago
    Sometimes it's just because of experience though. If you're simply bootstrapping a single app on a single vm, then fine, have at it. * As soon as you go down the route of making that reproducable, it can as easily be mostly done by a Dockerfile than a bash or Ansible (or any other configuration management) script. * Then you want failover or have it active on another server? Okay, you can just run you ansible against, maybe you put it in cloud-init and an autoscaling group, or maybe you can have something like a kubernetes framework take care of that. * You want health checking? Sure, have it configuring nagios or something similar, or you can have a ELB check an endpoint, or you can have kubernetes do it? * Want some storage? Let me add a PVC, or I can play about with managing EBSs and other block storage myself. Once you start digging a bit deeper, realising that many of the things your apps will probably want you either need to build yourself, or will lock you into AWS, going to google and clicking the GKE button doesn't seem a terrible prospect. There are other ways to do things, but once you've learnt this way, you can reuse it almost anywhere. Our industry is really faddish so I don't blame you for being skeptical, but the second you start hosting 3-4 apps, I'd rather have just bootstrapped kubernetes (or went to a managed service provider) and have all the primitives avaiable to me.
- ekidd 10y ago
  My rule of thumb is "keep it as simple as possible", and I say this as somebody who maintains a complex, containerized project using both Amazon's ECS and Kubernetes. For example: 1. If you can use a simple deployment system based on 'git push' (such as Heroku), you should just go ahead and do that until you outgrow it. This is easy to test, deploy and maintain. 2. If your project is tricky to build, go ahead and set up a Dockerfile with a correctly configured environment. You can still deploy this using Heroku, Google Container Engine, Amazon ECS or many other cloud providers. Choose something simple. This is harder than something like Heroku, but it's mostly a one-time learning cost. 3. If you need multiple containers, then set up a continuous integration system that automatically builds and tests your containers on every 'git push', and which allows you to deploy a service with a single click. You'll also want a separate staging environment. At this point, you're probably going to need to devote at least 10 hours/week of engineering time to keeping everything working. System-wide integration testing will get more complex. Again, you can stay on a managed cloud provider like Amazon's ECS or Google Container Engine for a long time before you need to set up and manage your own Kubernetes cluster. But if you find that you're writing lots of scripts to manage the containers on your clusters, it's probably time to look at a standard solution like Kubernetes. TL;dr: Stay with the easiest deployment solutions, and only add complexity when you need it.
- user5994461 10y ago
  Use for: Stateless applications (mostly web applications). Ban for: Databases.
  - derekperkins 10y ago
    I know that banning databases from k8s is the popular point of view now, but Google has been running databases in containers for nearly a decade. YouTube runs on top of http://vitess.io/ http://vitess.io/, which is a cloud native approach to MySQL. Certainly approach with caution, but there's no reason for a blanket dismissal of dbs in containers.
    - smarterclayton 10y ago
      I think it's even simpler: 1. Know what it takes to run a database (including storage, backup, upgrade, lifecycle, failure modes) 2. Know how a containerized cluster manager manages process, storage, lifecycle, and failure modes If you know both of those, running databases on Kubernetes (can't speak for swarm or mesos) is not hard or surprising, and you can get the benefits of both. If you don't know both of those in detail, you don't have any business running databases in containers. The intersection of folks who know both is still small. And the risk of problems when you understand only one is still high.
      - bogomipz 10y ago
        >"If you know both of those, running databases on Kubernetes (can't speak for swarm or mesos) is not hard or surprising," Are you speaking from experience when you say it is not hard? Could you elaborate on what databases your are currently running on Kubernetes and how they are configured? Also are these production? If I know number 1 and number 2 does that mean that I automatically understand all of the of the potential failure modes I might experience from combining 1 and 2? I certainly wouldn't think so.
        smarterclayton 10y ago
        I'm one of the engineers on OpenShift, and there have been different production databases (sql and nosql alike) running on OpenShift in very large companies for almost 2 years now, as well as many databases in staging and test configurations. Your point about 1/2 is fair, I was trying to convey that Kube follows certain rules w.r.t. process termination, storage, and safety that can be relied on when you internalize them. What's lacking today is the single doc that walks people through the tradeoffs and is easily approachable (although the stateful set docs do a pretty good job of it). In addition, we've made increasing effort at ensuring that behavior is predictable (why StatefulSets exist, and the changes in 1.5 to ensure terminating pods remain even if the node goes down). Storage continues to be the most important part of stateful apps in general. On AWS/gce/azure you get safe semantics for fencing storage (as long as you don't bend the rules). On metal you'll need a lot more care - the variety of NAS storage comes with lots of tradeoffs, and safe use assumes a level of sophistication that I wouldn't expect unless folks have made an investment in storage infrastructure. I expect that to continue to improve, with things like Ceph and Glusters direct integration, VMWare storage, and NetApp / other serious NFS integration. And it's always possible to treat nodes like pets on the cloud and leverage their local storage if you have good backups - at scale that can be fairly effective, but when doing one-off DBs using RDS and Aurora and others is hard to beat.
sandGorgon 10y ago
The first thing I do when I look at kubernetes tutorials is look at ingress or load-balancing... and usually I find it absent. It is not the fault of the author (and I commend them for taking the trouble), but Kubernetes is super complex for getting traffic inside the cluster. But here's a genuine appeal from someone who has suffered this before. In kubernetes, you have multiple types of ingresses - and you invariably need to have a combination of them to get anything done correctly. For example nginx-ingress will lose source-ip info and will not do L4 proxying very well. Which is why the unsaid point of all kubernetes deployments is "use ELB or GLB with proxy protocol and call it a day". I get why - K8s is trying to solve the most complex problems first and trickling it down. Which means for the time being, ingress is a very hard problem to solve in k8s. The second place where every one will get stuck is calico vs weave vs flannel vs whatever. Most tutorials say "use any one of them and let's move on". Unfortunately, that is not the case - choose your network plugins wisely. After spending a lot of time in k8s, I setup a Docker Swarm cluster in about 3 hours. It does not have some of the more complex usecases supported but it does three things super well - networking, ingress and secrets ... and for most beginners, that's all they need. P.S. i have the exact same stack running on kubernetes and docker swarm Docker Compose yml form 3.1 is brilliant - other than my docker files, i need a 30 line yml file to deploy my whole stack across 3 machines. Kubernetes on the other hand needs atleast 12 yml files ("deployments" and "services"). I'm not counting another 4 yml files for statefulsets and persistentvolumeclaim since there is nothing equivalent in docker swarm
- predakanga 10y ago
  Could you provide some more detail about your statement that "nginx-ingress will lose source-ip info"? I'm currently working towards a k8s-on-metal system using nginx for ingress and was led to believe that I could use either X-Forwarded-For or proxy_protocol to preserve the originating IP address. Is the traffic bounced through an SNAT rule on it's way to the ingress controller or similar?
  - sandGorgon 10y ago
    you should google for proxy-protocol. so here's the thing - nginx cannot INJECT proxy protocol. I'm not an expert in nginx.. but atleast the versions used in k8s couldnt. It can read/pass-through proxy protocol headers. So if you have haproxy in front of nginx.. it will work fine (or ELB/GLB). Which is why there's another beta ingress - the "keepalived-vip" repo in k8s ingress. However, not a lot of people seem to be using haproxy primarily because haproxy cannot have a zero-downtime deploy. A couple of weeks back on k8s slack, I was discussing using Github multibinder to solve that one problem and move ahead with haproxy ingress. protip: please join the #sig-onpremise channel on slack. There's not a lot of momentum for bare metal deploys otherwise. https://githubengineering.com/glb-part-2-haproxy-zero-downtime-zero-delay-reloads-with-multibinder/ https://githubengineering.com/glb-part-2-haproxy-zero-downti...
ricw 10y ago
This is nice and an interesting overview, but it still leaves me confused whether one should even use k8s. We currently use a docker-compose to set up and run multi-container services (django, celery & workers, nginx, cron/backup), yet I'm somewhat unconvinced of why we should invest in "upgrading" to kubernetes. My understanding is that it will make the system deal automatically with failing containers, though docker-compose already does that for us. Load balancing is another strength, though that is currently not an issue for us. It might automate certain container managing scripts, but that is less of an issue right now In other words, I currently don't see an actual benefit of k8s over even docker-compose (not docker-swarm, which I understand is the direct k8s alternative). Anyone able to elaborate?!
- why-el 10y ago
  I have been thinking about this myself lately, and always wondered whether K8s is missing a chance to position itself as the "resource manager" by default, regardless of what size your current load is at. In other words, change the terminology so that K8s appears to be a good choice even if you are just starting (this is the marketing part). Granted it will seem too complex for most apps, but how about a community effort, perhaps backed by a company, that essentially wraps most stacks (say Rails and PG, Django and Mysql, perhaps a caching layer, nginx, and so on) in a K8s module so that devs can start coding and be oblivious to changing loads as their apps grow? In this scenario, K8s will just feel natural, and will be something you just have to learn as part of your journey (Git was like this for a lot of developers, but I concede the analogy is not that great). Even if most devs reject this as being too complex, the few who pick it will be an invaluable feedback source.
  - gtirloni 10y ago
    Maybe you have Helm in mind? https://github.com/kubernetes/helm https://github.com/kubernetes/helm
    - why-el 10y ago
      Yeah that looks very promising.
raesene9 10y ago
One thing that I think is very much still missing from the Kubernetes Documentation space is hardening guidelines. There's a lot of moving parts in there and some of the defaults for common install methods like kubeadm might be a bit of a surprise to people (e.g. the kubelet port being default open and allowing someone to take complete control of the cluster without authentication (https://raesene.github.io/blog/2016/10/08/Kubernetes-From-Container-To-Cluster/) https://raesene.github.io/blog/2016/10/08/Kubernetes-From-Co...) Ideally something which broke out the various components and had guidelines for possible security options would be a great addition, I think.
- clebio 10y ago
  Link in parenthesis got broken: https://raesene.github.io/blog/2016/10/08/Kubernetes-From-Container-To-Cluster/ https://raesene.github.io/blog/2016/10/08/Kubernetes-From-Co...
atombender 10y ago
For me, the weakest point of Kubernetes at the moment is the developer story. Many shops don't have dedicated ops engineers, and would need to adapt/abstract K8s in order to make it user friendly enough for developers to use. For example, to start: * Where do you store the YAML manifests, and how do you maintain them? Do you put them in the same git repo as your app, and if so, how do you deal with the fact that the YAML files are going to be different for production, staging, testing/QA, etc.? (For example, ingresses will use other host names. Configs are likely to be rather different altogether.) * Or if you centralize them in a single git repo, you have to make sure that you always pull the newest version, and that your workflow includes diffing against the currently deployed version, and so on. * How do you protect configs/secrets, if they're in git and available to all? * Dependency tracking: If app A needs app B, you want to encapsulate that dependency in your workflow. * Continuous delivery: How do you take care of these concerns in relation to the CD system (e.g. Drone, Jenkins)? * And of course, you'll want to be able to develop locally. If you run Minikube, how do interact with it, YAML-wise? There's Helm, but Helm is essentially just a "templates in a central repo" manager. It doesn't solve the configuration issue: You still need to provide values to the charts. It doesn't sit right to have the YAML files in the project's git repo. Config follows environment, not project. For our current, non-Kubernetes production setup, we have a tool called Monkey that allows people do things like "monkey deploy staging:myapp -r mybranch" to deploy a branch, or "monkey sql prod:myapp" to get a PostgreSQL shell against the production database for that app, with lots of nice commands to work with the cluster. They also work directly with the Vagrant VM that developers run locally: "monkey deploy dev:myapp --dev" actually deploys to the VM, pointing the directory to the user's own project (on the host machine), so that the files are the same. Combined with a React/Redux app, they get hot code reloading and so on without having to do anything special. This shields developers from the complexities and realities of a Linux server in a very nice, convenient way. We want to provide the same on top of Kubernetes. Right now I'm debating whether to just go for "simple and stupid" and have a central repo, and then build a small toolset to wrap kubectl for devs. By running Minikube locally, devs would be able to use the same tool locally. The command-line tool could do things like always pull the repo to ensure you're working with the newest files. The main challenge there is organizing the YAML files in a clean way. Do we use templates to reduce boilerplate? Or do we rely on just copy-pasting stuff? It's not quite clear to me. Helm seems nice, but it's another moving part to add to the mix. I'm leaning towards a simpler approach. I've looked into Deis Workflow, but it's actually a full-blown PaaS, and seems a bit much.
dustinmoris 10y ago
Kubernetes is really cool. The only bad thing I can say about Kubernetes is that it has "uber" in its name.

223 points by ahawkins 10y ago

amq 10y ago

The first thing I'd like to be taught is where to use Kubernetes and where to avoid it (many people tend to think it's the best practice to force everything into containers, and ideally, Kubernetes).

ben_jones 10y ago
I say this only partially in jest but, I hope when presented with such choices I always pick the one that allows me the most hours of peaceful sleep at night. This usually gets me half way there during decision making.
- StavrosK 10y ago
  Why even partially in jest? Isn't the whole point of ops "sleeplessness avoidance"?
  - ben_jones 10y ago
    Honestly ops is about making/saving money for the company. Like an HR employee can choose the right thing, an ops engineer can also choose "sleeplessness avoidance", but it doesn't always work out that way.
    - StavrosK 10y ago
      That's true, but "sleeplessness avoidance" saves lots more money for small-scale companies. Downtime and man-hours are much more expensive than getting more reliable hosting, unless you're a very large company.
voiceclonr 10y ago
I tend to agree with you. Often, I see some of these stack decisions are forced by decision makers who want their design to look modern.
- FatalBaboon 10y ago
  Which is not a bad call, looking modern helps recruit. It's great that technology x lets you build something in 1 month, but if it takes 6 months to find someone willing to do it, then you could have written it faster in almost any other technology.
- ownagefool 10y ago
  Sometimes it's just because of experience though. If you're simply bootstrapping a single app on a single vm, then fine, have at it. * As soon as you go down the route of making that reproducable, it can as easily be mostly done by a Dockerfile than a bash or Ansible (or any other configuration management) script. * Then you want failover or have it active on another server? Okay, you can just run you ansible against, maybe you put it in cloud-init and an autoscaling group, or maybe you can have something like a kubernetes framework take care of that. * You want health checking? Sure, have it configuring nagios or something similar, or you can have a ELB check an endpoint, or you can have kubernetes do it? * Want some storage? Let me add a PVC, or I can play about with managing EBSs and other block storage myself. Once you start digging a bit deeper, realising that many of the things your apps will probably want you either need to build yourself, or will lock you into AWS, going to google and clicking the GKE button doesn't seem a terrible prospect. There are other ways to do things, but once you've learnt this way, you can reuse it almost anywhere. Our industry is really faddish so I don't blame you for being skeptical, but the second you start hosting 3-4 apps, I'd rather have just bootstrapped kubernetes (or went to a managed service provider) and have all the primitives avaiable to me.
ekidd 10y ago
My rule of thumb is "keep it as simple as possible", and I say this as somebody who maintains a complex, containerized project using both Amazon's ECS and Kubernetes. For example: 1. If you can use a simple deployment system based on 'git push' (such as Heroku), you should just go ahead and do that until you outgrow it. This is easy to test, deploy and maintain. 2. If your project is tricky to build, go ahead and set up a Dockerfile with a correctly configured environment. You can still deploy this using Heroku, Google Container Engine, Amazon ECS or many other cloud providers. Choose something simple. This is harder than something like Heroku, but it's mostly a one-time learning cost. 3. If you need multiple containers, then set up a continuous integration system that automatically builds and tests your containers on every 'git push', and which allows you to deploy a service with a single click. You'll also want a separate staging environment. At this point, you're probably going to need to devote at least 10 hours/week of engineering time to keeping everything working. System-wide integration testing will get more complex. Again, you can stay on a managed cloud provider like Amazon's ECS or Google Container Engine for a long time before you need to set up and manage your own Kubernetes cluster. But if you find that you're writing lots of scripts to manage the containers on your clusters, it's probably time to look at a standard solution like Kubernetes. TL;dr: Stay with the easiest deployment solutions, and only add complexity when you need it.
user5994461 10y ago
Use for: Stateless applications (mostly web applications). Ban for: Databases.
- derekperkins 10y ago
  I know that banning databases from k8s is the popular point of view now, but Google has been running databases in containers for nearly a decade. YouTube runs on top of http://vitess.io/ http://vitess.io/, which is a cloud native approach to MySQL. Certainly approach with caution, but there's no reason for a blanket dismissal of dbs in containers.
  - smarterclayton 10y ago
    I think it's even simpler: 1. Know what it takes to run a database (including storage, backup, upgrade, lifecycle, failure modes) 2. Know how a containerized cluster manager manages process, storage, lifecycle, and failure modes If you know both of those, running databases on Kubernetes (can't speak for swarm or mesos) is not hard or surprising, and you can get the benefits of both. If you don't know both of those in detail, you don't have any business running databases in containers. The intersection of folks who know both is still small. And the risk of problems when you understand only one is still high.
    - bogomipz 10y ago
      >"If you know both of those, running databases on Kubernetes (can't speak for swarm or mesos) is not hard or surprising," Are you speaking from experience when you say it is not hard? Could you elaborate on what databases your are currently running on Kubernetes and how they are configured? Also are these production? If I know number 1 and number 2 does that mean that I automatically understand all of the of the potential failure modes I might experience from combining 1 and 2? I certainly wouldn't think so.
      - smarterclayton 10y ago
        I'm one of the engineers on OpenShift, and there have been different production databases (sql and nosql alike) running on OpenShift in very large companies for almost 2 years now, as well as many databases in staging and test configurations. Your point about 1/2 is fair, I was trying to convey that Kube follows certain rules w.r.t. process termination, storage, and safety that can be relied on when you internalize them. What's lacking today is the single doc that walks people through the tradeoffs and is easily approachable (although the stateful set docs do a pretty good job of it). In addition, we've made increasing effort at ensuring that behavior is predictable (why StatefulSets exist, and the changes in 1.5 to ensure terminating pods remain even if the node goes down). Storage continues to be the most important part of stateful apps in general. On AWS/gce/azure you get safe semantics for fencing storage (as long as you don't bend the rules). On metal you'll need a lot more care - the variety of NAS storage comes with lots of tradeoffs, and safe use assumes a level of sophistication that I wouldn't expect unless folks have made an investment in storage infrastructure. I expect that to continue to improve, with things like Ceph and Glusters direct integration, VMWare storage, and NetApp / other serious NFS integration. And it's always possible to treat nodes like pets on the cloud and leverage their local storage if you have good backups - at scale that can be fairly effective, but when doing one-off DBs using RDS and Aurora and others is hard to beat.

sandGorgon 10y ago

The first thing I do when I look at kubernetes tutorials is look at ingress or load-balancing... and usually I find it absent. It is not the fault of the author (and I commend them for taking the trouble), but Kubernetes is super complex for getting traffic inside the cluster. But here's a genuine appeal from someone who has suffered this before. In kubernetes, you have multiple types of ingresses - and you invariably need to have a combination of them to get anything done correctly. For example nginx-ingress will lose source-ip info and will not do L4 proxying very well. Which is why the unsaid point of all kubernetes deployments is "use ELB or GLB with proxy protocol and call it a day". I get why - K8s is trying to solve the most complex problems first and trickling it down. Which means for the time being, ingress is a very hard problem to solve in k8s. The second place where every one will get stuck is calico vs weave vs flannel vs whatever. Most tutorials say "use any one of them and let's move on". Unfortunately, that is not the case - choose your network plugins wisely. After spending a lot of time in k8s, I setup a Docker Swarm cluster in about 3 hours. It does not have some of the more complex usecases supported but it does three things super well - networking, ingress and secrets ... and for most beginners, that's all they need. P.S. i have the exact same stack running on kubernetes and docker swarm Docker Compose yml form 3.1 is brilliant - other than my docker files, i need a 30 line yml file to deploy my whole stack across 3 machines. Kubernetes on the other hand needs atleast 12 yml files ("deployments" and "services"). I'm not counting another 4 yml files for statefulsets and persistentvolumeclaim since there is nothing equivalent in docker swarm

predakanga 10y ago
Could you provide some more detail about your statement that "nginx-ingress will lose source-ip info"? I'm currently working towards a k8s-on-metal system using nginx for ingress and was led to believe that I could use either X-Forwarded-For or proxy_protocol to preserve the originating IP address. Is the traffic bounced through an SNAT rule on it's way to the ingress controller or similar?
- sandGorgon 10y ago
  you should google for proxy-protocol. so here's the thing - nginx cannot INJECT proxy protocol. I'm not an expert in nginx.. but atleast the versions used in k8s couldnt. It can read/pass-through proxy protocol headers. So if you have haproxy in front of nginx.. it will work fine (or ELB/GLB). Which is why there's another beta ingress - the "keepalived-vip" repo in k8s ingress. However, not a lot of people seem to be using haproxy primarily because haproxy cannot have a zero-downtime deploy. A couple of weeks back on k8s slack, I was discussing using Github multibinder to solve that one problem and move ahead with haproxy ingress. protip: please join the #sig-onpremise channel on slack. There's not a lot of momentum for bare metal deploys otherwise. https://githubengineering.com/glb-part-2-haproxy-zero-downtime-zero-delay-reloads-with-multibinder/ https://githubengineering.com/glb-part-2-haproxy-zero-downti...

ricw 10y ago

This is nice and an interesting overview, but it still leaves me confused whether one should even use k8s. We currently use a docker-compose to set up and run multi-container services (django, celery & workers, nginx, cron/backup), yet I'm somewhat unconvinced of why we should invest in "upgrading" to kubernetes. My understanding is that it will make the system deal automatically with failing containers, though docker-compose already does that for us. Load balancing is another strength, though that is currently not an issue for us. It might automate certain container managing scripts, but that is less of an issue right now In other words, I currently don't see an actual benefit of k8s over even docker-compose (not docker-swarm, which I understand is the direct k8s alternative). Anyone able to elaborate?!

why-el 10y ago
I have been thinking about this myself lately, and always wondered whether K8s is missing a chance to position itself as the "resource manager" by default, regardless of what size your current load is at. In other words, change the terminology so that K8s appears to be a good choice even if you are just starting (this is the marketing part). Granted it will seem too complex for most apps, but how about a community effort, perhaps backed by a company, that essentially wraps most stacks (say Rails and PG, Django and Mysql, perhaps a caching layer, nginx, and so on) in a K8s module so that devs can start coding and be oblivious to changing loads as their apps grow? In this scenario, K8s will just feel natural, and will be something you just have to learn as part of your journey (Git was like this for a lot of developers, but I concede the analogy is not that great). Even if most devs reject this as being too complex, the few who pick it will be an invaluable feedback source.
- gtirloni 10y ago
  Maybe you have Helm in mind? https://github.com/kubernetes/helm https://github.com/kubernetes/helm
  - why-el 10y ago
    Yeah that looks very promising.

raesene9 10y ago

One thing that I think is very much still missing from the Kubernetes Documentation space is hardening guidelines. There's a lot of moving parts in there and some of the defaults for common install methods like kubeadm might be a bit of a surprise to people (e.g. the kubelet port being default open and allowing someone to take complete control of the cluster without authentication (https://raesene.github.io/blog/2016/10/08/Kubernetes-From-Container-To-Cluster/) https://raesene.github.io/blog/2016/10/08/Kubernetes-From-Co...) Ideally something which broke out the various components and had guidelines for possible security options would be a great addition, I think.

clebio 10y ago
Link in parenthesis got broken: https://raesene.github.io/blog/2016/10/08/Kubernetes-From-Container-To-Cluster/ https://raesene.github.io/blog/2016/10/08/Kubernetes-From-Co...

atombender 10y ago

For me, the weakest point of Kubernetes at the moment is the developer story. Many shops don't have dedicated ops engineers, and would need to adapt/abstract K8s in order to make it user friendly enough for developers to use. For example, to start: * Where do you store the YAML manifests, and how do you maintain them? Do you put them in the same git repo as your app, and if so, how do you deal with the fact that the YAML files are going to be different for production, staging, testing/QA, etc.? (For example, ingresses will use other host names. Configs are likely to be rather different altogether.) * Or if you centralize them in a single git repo, you have to make sure that you always pull the newest version, and that your workflow includes diffing against the currently deployed version, and so on. * How do you protect configs/secrets, if they're in git and available to all? * Dependency tracking: If app A needs app B, you want to encapsulate that dependency in your workflow. * Continuous delivery: How do you take care of these concerns in relation to the CD system (e.g. Drone, Jenkins)? * And of course, you'll want to be able to develop locally. If you run Minikube, how do interact with it, YAML-wise? There's Helm, but Helm is essentially just a "templates in a central repo" manager. It doesn't solve the configuration issue: You still need to provide values to the charts. It doesn't sit right to have the YAML files in the project's git repo. Config follows environment, not project. For our current, non-Kubernetes production setup, we have a tool called Monkey that allows people do things like "monkey deploy staging:myapp -r mybranch" to deploy a branch, or "monkey sql prod:myapp" to get a PostgreSQL shell against the production database for that app, with lots of nice commands to work with the cluster. They also work directly with the Vagrant VM that developers run locally: "monkey deploy dev:myapp --dev" actually deploys to the VM, pointing the directory to the user's own project (on the host machine), so that the files are the same. Combined with a React/Redux app, they get hot code reloading and so on without having to do anything special. This shields developers from the complexities and realities of a Linux server in a very nice, convenient way. We want to provide the same on top of Kubernetes. Right now I'm debating whether to just go for "simple and stupid" and have a central repo, and then build a small toolset to wrap kubectl for devs. By running Minikube locally, devs would be able to use the same tool locally. The command-line tool could do things like always pull the repo to ensure you're working with the newest files. The main challenge there is organizing the YAML files in a clean way. Do we use templates to reduce boilerplate? Or do we rely on just copy-pasting stuff? It's not quite clear to me. Helm seems nice, but it's another moving part to add to the mix. I'm leaning towards a simpler approach. I've looked into Deis Workflow, but it's actually a full-blown PaaS, and seems a bit much.

dustinmoris 10y ago

Kubernetes is really cool. The only bad thing I can say about Kubernetes is that it has "uber" in its name.