3 ms·

For me, the weakest point of Kubernetes at the moment is the developer story. Many shops don't have dedicated ops engineers, and would need to adapt/abstract K8

For me, the weakest point of Kubernetes at the moment is the developer story. Many shops don't have dedicated ops engineers, and would need to adapt/abstract K8s in order to make it user friendly enough for developers to use. For example, to start: * Where do you store the YAML manifests, and how do you maintain them? Do you put them in the same git repo as your app, and if so, how do you deal with the fact that the YAML files are going to be different for production, staging, testing/QA, etc.? (For example, ingresses will use other host names. Configs are likely to be rather different altogether.) * Or if you centralize them in a single git repo, you have to make sure that you always pull the newest version, and that your workflow includes diffing against the currently deployed version, and so on. * How do you protect configs/secrets, if they're in git and available to all? * Dependency tracking: If app A needs app B, you want to encapsulate that dependency in your workflow. * Continuous delivery: How do you take care of these concerns in relation to the CD system (e.g. Drone, Jenkins)? * And of course, you'll want to be able to develop locally. If you run Minikube, how do interact with it, YAML-wise? There's Helm, but Helm is essentially just a "templates in a central repo" manager. It doesn't solve the configuration issue: You still need to provide values to the charts. It doesn't sit right to have the YAML files in the project's git repo. Config follows environment, not project. For our current, non-Kubernetes production setup, we have a tool called Monkey that allows people do things like "monkey deploy staging:myapp -r mybranch" to deploy a branch, or "monkey sql prod:myapp" to get a PostgreSQL shell against the production database for that app, with lots of nice commands to work with the cluster. They also work directly with the Vagrant VM that developers run locally: "monkey deploy dev:myapp --dev" actually deploys to the VM, pointing the directory to the user's own project (on the host machine), so that the files are the same. Combined with a React/Redux app, they get hot code reloading and so on without having to do anything special. This shields developers from the complexities and realities of a Linux server in a very nice, convenient way. We want to provide the same on top of Kubernetes. Right now I'm debating whether to just go for "simple and stupid" and have a central repo, and then build a small toolset to wrap kubectl for devs. By running Minikube locally, devs would be able to use the same tool locally. The command-line tool could do things like always pull the repo to ensure you're working with the newest files. The main challenge there is organizing the YAML files in a clean way. Do we use templates to reduce boilerplate? Or do we rely on just copy-pasting stuff? It's not quite clear to me. Helm seems nice, but it's another moving part to add to the mix. I'm leaning towards a simpler approach. I've looked into Deis Workflow, but it's actually a full-blown PaaS, and seems a bit much.

maktouch 10y ago
I've been running K8S for the past 6 months. So far, so good. Helm helped us a lot in our workflow. A lot of trial and errors, but to answer your points, this is what we're doing so far: > * Where do you store the YAML manifests, and how do you maintain them? Do you put them in the same git repo as your app, and if so, how do you deal with the fact that the YAML files are going to be different for production, staging, testing/QA, etc.? (For example, ingresses will use other host names. Configs are likely to be rather different altogether.) We store the manifests in the same git repo as the app. Docker images and YAML files are all the same, what's different are the values, so we use helm and override the values per environment. The values are split 4 ways: - default sensible values (values.yaml). For example, feature flags. - datacenter specific (gcp.yaml, aws.yaml). For example, REDIS_HOST and MYSQL_HOST. - environment values (env-production.yaml or env-staging.yaml). For example, the ingress values and its certs, how many replicas for each service - secrets.yaml (not stored in git, but generated on the fly, will explain more later). For example, MYSQL_PASSWORD I mean, if you're doing your services right, they should all have the same docker images, only the config changes. Helm has a pretty good templating and config helpers. > * Or if you centralize them in a single git repo, you have to make sure that you always pull the newest version, and that your workflow includes diffing against the currently deployed version, and so on. Helm takes care of this. When you upgrade a deployment, it looks like it's doing a diff. What's cool though, is that it seems like it does a diff on everything except replicas, which is exactly what we needed. > * How do you protect configs/secrets, if they're in git and available to all? I don't mind having the config in git, I do when it comes to secret. We store secrets as a gitlab variable, and our CI dynamically creates a "secrets.yaml" before deploying. I'm still not really happy about this though, I think a better way would be to use Vault, but it adds some complexities that I'm not really to deal with yet. > * Dependency tracking: If app A needs app B, you want to encapsulate that dependency in your workflow. I'm not too sure I understand what you mean by this one. We, thankfully, merged all our git repos into 1 monolithic repository. I remember at first reading about the big boys (FB, MS, Google) having a single repository and thought they were crazy. Then we added more and more services and suddenly, I just realized that having a monorepo makes is a lot easier for developers and ops. AppA~1.0 needs AppB~2.5. We used to have a crazy graph of dependency like that. Not anymore. Now every services has the same version -- we're simply using git hashes. AppA~4c20d needs AppB-4c20d. Every push to the repo rebuilds the docker images and tags them like this: AppA:master-4c20d and AppB:master-4c20d. And yes, we build for all branches and all commits. So for deploying, it's super easy. I just deploy everything at once, at all time. If I need to rollback, I can rollback to a specific version on all services. If I need to test in a real environment a specific version, all I do is add --set BRANCH=fix-bug-branch,COMMIT=4c20d > * Continuous delivery: How do you take care of these concerns in relation to the CD system (e.g. Drone, Jenkins)? There's no concerns here since every code push starts our pipeline, and the pipeline rebuilds all docker images, and runs the tests between each other. It's pretty dope. > * And of course, you'll want to be able to develop locally. If you run Minikube, how do interact with it, YAML-wise? We have a local bare-metal server running a single-node k8s, but to be honest, we've never had a dev needing to develop on K8S specifically. Each dev that builds a service is also tasked with building the dockerfile that goes with it, and with assistance, they build the YAML file that goes with it. It's pretty straightforward. Our gitlab auto deploys to dev, staging and review environment, so if there's an invalid YAML, the pipeline will catch it. > There's Helm, but Helm is essentially just a "templates in a central repo" manager. It doesn't solve the configuration issue: You still need to provide values to the charts. I think it does but not directly. It solves it because it allows you to use templates and override config in a sane way. Here's an extract of our .gitlab-ci file for deploying to production - echo "Deploying to Kubernetes Env=production Branch=$CI_BUILD_REF_NAME Version=$CI_BUILD_REF" - "echo MYSQL_PASSWORD: $MYSQL_PASSWORD_PRODUCTION > tmp.yaml" - "echo MYSQL_USERNAME: $MYSQL_USERNAME_PRODUCTION >> tmp.yaml" - ... more secrets echo config here - helm upgrade master ./manifests/App -f ./manifests/gcp.yaml -f ./manifests/env-production.yaml -f tmp.yaml --set BRANCH=master,COMMIT=$CI_BUILD_REF > Right now I'm debating whether to just go for "simple and stupid" and have a central repo, and then build a small toolset to wrap kubectl for devs [...] We started with building our own toolset to wrap kubectl. It worked. Then we found Helm. I remember at first looking at it and thought it wasn't useful for us.. now it looks pretty good. We heavily use templates to reduce boilerplate. ----- I'm curious about your tool called Monkey. Is it a frontend for chef or ansible? Also, I can pretty much ask all the same questions you've asked but geared towards your tool (where do you store manifests, how do you protect config/secrets, how do do you handle dependency, continuous delivery, etc etc).

maktouch 10y ago

I've been running K8S for the past 6 months. So far, so good. Helm helped us a lot in our workflow. A lot of trial and errors, but to answer your points, this is what we're doing so far: > * Where do you store the YAML manifests, and how do you maintain them? Do you put them in the same git repo as your app, and if so, how do you deal with the fact that the YAML files are going to be different for production, staging, testing/QA, etc.? (For example, ingresses will use other host names. Configs are likely to be rather different altogether.) We store the manifests in the same git repo as the app. Docker images and YAML files are all the same, what's different are the values, so we use helm and override the values per environment. The values are split 4 ways: - default sensible values (values.yaml). For example, feature flags. - datacenter specific (gcp.yaml, aws.yaml). For example, REDIS_HOST and MYSQL_HOST. - environment values (env-production.yaml or env-staging.yaml). For example, the ingress values and its certs, how many replicas for each service - secrets.yaml (not stored in git, but generated on the fly, will explain more later). For example, MYSQL_PASSWORD I mean, if you're doing your services right, they should all have the same docker images, only the config changes. Helm has a pretty good templating and config helpers. > * Or if you centralize them in a single git repo, you have to make sure that you always pull the newest version, and that your workflow includes diffing against the currently deployed version, and so on. Helm takes care of this. When you upgrade a deployment, it looks like it's doing a diff. What's cool though, is that it seems like it does a diff on everything except replicas, which is exactly what we needed. > * How do you protect configs/secrets, if they're in git and available to all? I don't mind having the config in git, I do when it comes to secret. We store secrets as a gitlab variable, and our CI dynamically creates a "secrets.yaml" before deploying. I'm still not really happy about this though, I think a better way would be to use Vault, but it adds some complexities that I'm not really to deal with yet. > * Dependency tracking: If app A needs app B, you want to encapsulate that dependency in your workflow. I'm not too sure I understand what you mean by this one. We, thankfully, merged all our git repos into 1 monolithic repository. I remember at first reading about the big boys (FB, MS, Google) having a single repository and thought they were crazy. Then we added more and more services and suddenly, I just realized that having a monorepo makes is a lot easier for developers and ops. AppA~1.0 needs AppB~2.5. We used to have a crazy graph of dependency like that. Not anymore. Now every services has the same version -- we're simply using git hashes. AppA~4c20d needs AppB-4c20d. Every push to the repo rebuilds the docker images and tags them like this: AppA:master-4c20d and AppB:master-4c20d. And yes, we build for all branches and all commits. So for deploying, it's super easy. I just deploy everything at once, at all time. If I need to rollback, I can rollback to a specific version on all services. If I need to test in a real environment a specific version, all I do is add --set BRANCH=fix-bug-branch,COMMIT=4c20d > * Continuous delivery: How do you take care of these concerns in relation to the CD system (e.g. Drone, Jenkins)? There's no concerns here since every code push starts our pipeline, and the pipeline rebuilds all docker images, and runs the tests between each other. It's pretty dope. > * And of course, you'll want to be able to develop locally. If you run Minikube, how do interact with it, YAML-wise? We have a local bare-metal server running a single-node k8s, but to be honest, we've never had a dev needing to develop on K8S specifically. Each dev that builds a service is also tasked with building the dockerfile that goes with it, and with assistance, they build the YAML file that goes with it. It's pretty straightforward. Our gitlab auto deploys to dev, staging and review environment, so if there's an invalid YAML, the pipeline will catch it. > There's Helm, but Helm is essentially just a "templates in a central repo" manager. It doesn't solve the configuration issue: You still need to provide values to the charts. I think it does but not directly. It solves it because it allows you to use templates and override config in a sane way. Here's an extract of our .gitlab-ci file for deploying to production - echo "Deploying to Kubernetes Env=production Branch=$CI_BUILD_REF_NAME Version=$CI_BUILD_REF" - "echo MYSQL_PASSWORD: $MYSQL_PASSWORD_PRODUCTION > tmp.yaml" - "echo MYSQL_USERNAME: $MYSQL_USERNAME_PRODUCTION >> tmp.yaml" - ... more secrets echo config here - helm upgrade master ./manifests/App -f ./manifests/gcp.yaml -f ./manifests/env-production.yaml -f tmp.yaml --set BRANCH=master,COMMIT=$CI_BUILD_REF > Right now I'm debating whether to just go for "simple and stupid" and have a central repo, and then build a small toolset to wrap kubectl for devs [...] We started with building our own toolset to wrap kubectl. It worked. Then we found Helm. I remember at first looking at it and thought it wasn't useful for us.. now it looks pretty good. We heavily use templates to reduce boilerplate. ----- I'm curious about your tool called Monkey. Is it a frontend for chef or ansible? Also, I can pretty much ask all the same questions you've asked but geared towards your tool (where do you store manifests, how do you protect config/secrets, how do do you handle dependency, continuous delivery, etc etc).