8 ms·

Show HN: Gitkube – Deploy to Kubernetes Using Git Push

dev1n 9y ago
Google released a tool to help with continuous deployments called skaffold [1]. Haven't used it myself (just using bitbucket pipelines) but looks like it's a relevant tool to this. Nice. [1]: https://github.com/GoogleCloudPlatform/skaffold https://github.com/GoogleCloudPlatform/skaffold
deepaksahoo 9y ago
This is awesome! Thanks for open-sourcing :)
bonesss 9y ago
Wow! This is highly relevant to my troubles this morning, and solves a highly relevant pain-point. I would not be surprised to find something like this mainlined into k8s over time... Nicely done :)
chrissoundz 9y ago
Why not just `git push`, `docker build`, `kubectl set image`?
- tirumaraiselvan 9y ago
  Because why not just `git push` :)
- ecthiender 9y ago
  I'm not the author but I would imagine - this allows more room for automation. You should be able to setup a CI/CD system around it with webhooks from your git repo.
- lincolnq 9y ago
  Lots of reasons: - A local Docker build might have issues due to your local computer's environment that cause mysterious or hard-to-replicate problems. - You might not have permission to push new images to your organization's Docker repo from your computer. - You might be on slow internet, or your docker images large enough, such that pushing on the cloud is 10-100x faster than pushing locally. (this is my pain point) - You need to remember, and use, whatever Docker image tagging scheme your organization requires.
  - arianvanp 9y ago
    > A local Docker build might have issues due to your local computer's environment that cause mysterious or hard-to-replicate problems. This always comes and bites us back isn't it? The whole promise of Docker was that we would be freed form these pains... But yet here we stand :( Wondering what kind of issues you're running into. The first one that comes to mind for me is Kernel version mismatches. Though those shouldn't always cause that much trouble due to the "Dont break userspace mantra" that Linus has
    - Promarged 9y ago
      > This always comes and bites us back isn't it? The whole promise of Docker was that we would be freed form these pains... But yet here we stand :( Exactly. Using a separate build host makes builds more independent but on the other hand you'll be running images that the dev did not check. By the way, are docker image identifiers derived from the actual contents? (Are they reproducible?) This would suggest they are not: > Docker images are non-reproducible: each "layer" identifier is a random hex string (and not cryptographic hash of the layer content), Source: https://blog.bazel.build/2015/07/28/docker_build.html https://blog.bazel.build/2015/07/28/docker_build.html
      - dlor 9y ago
        That's a bit out of date - today layer IDs are for the most part sha256's of the contents of the layer tar.gz. However, it's still very difficult to generate the same layer ID twice. Timestamps permeate the layer contents itself in the form of file mtimes. The final image metadata itself also contains several timestamps. Docker has a local cache of layers that helps simulate reproducibility. But if you clear that cache or use a different build machine, you will have a very hard time ever generating the same layer ID again.
pcnix 9y ago
This certainly does seem to be a fairly easy way to deploy. I'm assuming you'd integrate standard ci/cd tools in via webhooks? Or even pre-push scripts?
- tirumaraiselvan 9y ago
  Yes, that is definitely in the roadmap and should be simple to do. Checkout this issue https://github.com/hasura/gitkube/issues/20 https://github.com/hasura/gitkube/issues/20
lima 9y ago
Red Hat's OpenShift[1] Kubernetes distribution can do this too - you even get a webhook for GitHub copy and paste. Kubernetes is a toolbox, not a product. There's so much you have to set up yourself (build system, registry, security, logging stack, metrics...). That's fine if you're in the business of selling Kubernetes itself, but otherwise, most teams should use something like OpenShift and not re-invent the wheel. [1]: https://github.com/openshift/origin https://github.com/openshift/origin
- tirumaraiselvan 9y ago
  This is supposed to be very light weight and independent of any distribution or vendor.
  - tango12 9y ago
    (I'm from hasura.io) In fact, this is the component that powers a portion of the "git-ops" on the hasura.io platform. Like one of the commenters said, we also do some pre-push magic right now (esp. for client-side kubernetes templating) for a more complete and opionionated CI/CD workflow.
- merb 9y ago
  well the problem with openshift on redhat/centos/atomic/whatever is that you still need to regulary update your cluster os's. however when you setup a bootkubed/self-hosted kubeadm cluster you can basically run on coreos and only need to update your self-hosted control plane and kubelet/(etcd, if not self hosted, too) this makes it way, way easier to have an up-to-date os + kube than any other solution (kubelet and etcd can be updated via etcd-wrapper and kubelet-wrapper, so this is even more insane). BUT i think that red hat pushes for an easier upgrade path anyway, that's why they basically bought coreos. for CoreUpdate + Tectonic + MatchBox, so I think that in the end Redhat will have a similar simplier upgrade mechanism, but only time will tell. What would in fact be awesome if CoreUpdate could also update Kubernetes Kubelet's and Etcd and than it would be a no-ops os if something broke you just boot a new system and delete the broken node. And yes kubectl is the ssh/ftp of k8s. https://www.youtube.com/watch?v=07jq-5VbBVQ https://www.youtube.com/watch?v=07jq-5VbBVQ
paulsmal 9y ago
I really like when deployment tools rely on git hooks and doesn't require any other installations in dev environment.
lifeeth_ 9y ago
I was thinking of hasura.io while reading the page only to find this is a hasura project - Keep up the great work
- tirumaraiselvan 9y ago
  Haha :) This is the first of a few projects open-sourced out of the Hasura platform
vegancap 9y ago
This is a beauty! Excellent work.
anirudhmurali 9y ago
Sexy stuff!
- dominotw 9y ago
  Hyping ur own stuff on HN?
praseodym 9y ago
We’ve been using a similar solution for a while now: pushes to GitHub trigger Travis CI to build a Docker images. These get pushed to Docker Hub or Quay, which triggers a webhook to Keel.sh[1] running in our Kubernetes cluster. Keel then updates the Kubernetes deployment to perform a rolling update of the application. 1: https://keel.sh https://keel.sh
- deforciant 9y ago
  Thanks for the shout-out! Keel author here. I have created Keel exactly for these workflows where something (Travis CI, Docker cloud, Google Cloud Builder - my favourite due to the pricing and proper speed) builds an image and then we would want to trigger a deployment update. I was running workloads on GKE and I found that while it's easy to get images being built, there was no straightforward way to automate deployment updates. My goal was to tag a release on Github and get a new version deployed in my environment. After searching for similar tools and couldn't find anything lightweight (spinnaker or jenkins were way too big!) I ended up building Keel. Here's an example if anyone has this problem https://keel.sh/v1/examples/enabling-auto-updates.html https://keel.sh/v1/examples/enabling-auto-updates.html.
  - cheez 9y ago
    What is it we can achieve with Keel that we can't with Terraform?
    - alexnewman 9y ago
      Or by awk for that matter. Sorry to be silly but terraform solves a completely different problem, which is what happens when you have disparate cloud resources you want to manage. For instance cloud network resources. Using it to manage containers is... Not it's use case.
  - antoncohen 9y ago
    Keel looks nice, thanks for making it! Does Keel include any ability to rollback to a previously deployed version? How to you recommend handling that use case? Keel looks similar to Weave Flux (https://github.com/weaveworks/flux https://github.com/weaveworks/flux). Have you looked at Flux, are you able to compare them? At first glance it looks like Keel is probably easier to setup, but Flux is more feature rich (edits k8s .yaml files in git with updated image tags, has CLI that supports listing deployments and images, can release an old image).
linkmotif 9y ago
I use https://github.com/dminkovsky/kube-cloud-build https://github.com/dminkovsky/kube-cloud-build for this, which uses Google Cloud Container Builder to perform builds. It’s a really nice workflow!
dlor 9y ago
I wonder how this tool handles performing the docker build itself. It looks like it runs a controller inside the cluster, but by default apps running inside kubernetes don't get access to the docker daemon. There are a few existing workarounds, but none of them are terribly appealing. See this longstanding issue for some more context: https://github.com/kubernetes/kubernetes/issues/1806 https://github.com/kubernetes/kubernetes/issues/1806
- tirumaraiselvan 9y ago
  Docker sock is being mounted from the host [1]. This could be a concern if you are running multi-tenant apps etc. One (slightly hacky) workaround is to have a Persistent Volume mount the hostPath and restrict hostPath mounting in the pod through PSPs. [1] https://github.com/hasura/gitkube/blob/7d0e7f25b811d48089b788593fabf7dd092b0687/manifests/gitkube-setup.yaml#L93 https://github.com/hasura/gitkube/blob/7d0e7f25b811d48089b78...
asadlionpk 9y ago
This looks exactly what I had been looking for these days. None of existing solutions fit my use case. Thanks for making this, I will give it a try soon.
zackify 9y ago
I’ve been using drone.io to do this for a month now. Loving it: - git push starts the process - build and deploy images to google cloud with the plugins/gcr tool - set the image in kubernetes in the next step using the google cloud sdk image I run unit tests as a docker build step. If they fail, it won’t deploy. If the readiness probe fails after setting it in kubernetes, it won’t switch over!
vijaykodam 9y ago
Good work from Team Hasura! Also liked the HDFS on k8s tutorial posted sometime back.
minieggs 9y ago
Interesting, and real nice! I built something similar for my raspberry pi cluster with docker swarm (albeit no testing and tailored to my use case). Kicking myself for not making "production ready" and posting here now.
truesy 9y ago
Close to opening up a service that does this for pull request branches, from GitHub to AWS ECS/Fargate. https://conjure.sh https://conjure.sh Glad to see this kind of workflow gaining momentum.
_5meq 9y ago
Keel does a really good job of this, good to see people are starting to have options with their k8 deployments! My two cents: So far my deployments have been pretty painless, so changing over to something like Gitkube / keel isn't my top priority currently. Configuration management / Creating my kubernetes resources ( deployments / services / ingress / etc. ) in a repeatable and elegant way is my biggest pain point by far. # Helm I've been very unhappy with Helm. Helm is insanely verbose. Go ahead and take a look at the kubernetes/charts repo and look at the various stable charts. Note the huge amount of copy-pasta code hanging out. Too much of my life has been spent wrangling YAML templates that are rendered using the under-featured gotmpl library. Also, helm's client library support is less than stellar. Also, schlepping my configurations around is a pain. In Helm, it's a crazy amount of code to implement a `ingress -> service -> deployment` pattern that is standard for 99% of kubernetes resources. You have to write the same things over and over, which means that making changes takes forever and is brittle. I believe this is due to gotmpl being a poor templating engine for this use case. It doesn't have enough features to allow one to develop decent abstractions over declaring k8 resources The helm tiller doesn't expose a restful API or proper tooling to allow other resources to interact with it effectively, making automated deployments a chore. My only option is to call out to the shell and try and be devensive. Finally, Helm doesn't really do that great a job of validating my resources are going to be valid. `helm install --dry-run` will tell you everything is great, and then break in the middle of an actual installation, leaving a half-configured mess in it's wake. # KSonnet KSonnet looks like an attractive option, it's certainly less verbose. But the documentation isn't there yet, and if you peek under the hood there is a MOUNTAIN of ksonnet code waiting to be read and understood. I belive it's going through some churn currently, so features and impovements have been slow to appear. Also KSonnet doesn't have enough example implementations to hit the ground running when trying it out on my clusters. Targeting individual contexts in Ksonnet provides some nice additional safety, and some very simple implementations using google KMS or something similar could be a really special way to safely store my secret configs at rest, similar to ansible-vault. # Terraform Terraform has this NAILED in the VM space. I declare my environment: `terraform plan` tells me what is going to happen, `terraform apply` checks to ensure I've not protected any of my resources with a flag, and applies the changes it listed if not. I'd love to write my k8 resources in terraform, but it doesn't currently support modern k8 resources, and I don't LOVE how terraform handles storing my tfstate files either way.
pmarreck 9y ago
I deploy via git-push to Travis CI which then runs my test suites and if they pass, continues on with the deployment to the (virtual) hardware server via https://www.gigalixir.com/ https://www.gigalixir.com/ (it's an Elixir Phoenix web app). A properly set up CI is awesome and a huge timesaver.
lewq 9y ago
Looks a lot like http://github.com/weaveworks/flux http://github.com/weaveworks/flux which we're happy users of at dotmesh.com. Glad to see gitops getting some traction! https://www.weave.works/blog/gitops-operations-by-pull-request https://www.weave.works/blog/gitops-operations-by-pull-reque...

Google released a tool to help with continuous deployments called skaffold [1]. Haven't used it myself (just using bitbucket pipelines) but looks like it's a relevant tool to this. Nice. [1]: https://github.com/GoogleCloudPlatform/skaffold https://github.com/GoogleCloudPlatform/skaffold

deepaksahoo 9y ago

This is awesome! Thanks for open-sourcing :)

bonesss 9y ago

Wow! This is highly relevant to my troubles this morning, and solves a highly relevant pain-point. I would not be surprised to find something like this mainlined into k8s over time... Nicely done :)

chrissoundz 9y ago

Why not just `git push`, `docker build`, `kubectl set image`?

tirumaraiselvan 9y ago
Because why not just `git push` :)
ecthiender 9y ago
I'm not the author but I would imagine - this allows more room for automation. You should be able to setup a CI/CD system around it with webhooks from your git repo.
lincolnq 9y ago
Lots of reasons: - A local Docker build might have issues due to your local computer's environment that cause mysterious or hard-to-replicate problems. - You might not have permission to push new images to your organization's Docker repo from your computer. - You might be on slow internet, or your docker images large enough, such that pushing on the cloud is 10-100x faster than pushing locally. (this is my pain point) - You need to remember, and use, whatever Docker image tagging scheme your organization requires.
- arianvanp 9y ago
  > A local Docker build might have issues due to your local computer's environment that cause mysterious or hard-to-replicate problems. This always comes and bites us back isn't it? The whole promise of Docker was that we would be freed form these pains... But yet here we stand :( Wondering what kind of issues you're running into. The first one that comes to mind for me is Kernel version mismatches. Though those shouldn't always cause that much trouble due to the "Dont break userspace mantra" that Linus has
  - Promarged 9y ago
    > This always comes and bites us back isn't it? The whole promise of Docker was that we would be freed form these pains... But yet here we stand :( Exactly. Using a separate build host makes builds more independent but on the other hand you'll be running images that the dev did not check. By the way, are docker image identifiers derived from the actual contents? (Are they reproducible?) This would suggest they are not: > Docker images are non-reproducible: each "layer" identifier is a random hex string (and not cryptographic hash of the layer content), Source: https://blog.bazel.build/2015/07/28/docker_build.html https://blog.bazel.build/2015/07/28/docker_build.html
    - dlor 9y ago
      That's a bit out of date - today layer IDs are for the most part sha256's of the contents of the layer tar.gz. However, it's still very difficult to generate the same layer ID twice. Timestamps permeate the layer contents itself in the form of file mtimes. The final image metadata itself also contains several timestamps. Docker has a local cache of layers that helps simulate reproducibility. But if you clear that cache or use a different build machine, you will have a very hard time ever generating the same layer ID again.

pcnix 9y ago

This certainly does seem to be a fairly easy way to deploy. I'm assuming you'd integrate standard ci/cd tools in via webhooks? Or even pre-push scripts?

tirumaraiselvan 9y ago
Yes, that is definitely in the roadmap and should be simple to do. Checkout this issue https://github.com/hasura/gitkube/issues/20 https://github.com/hasura/gitkube/issues/20

lima 9y ago

Red Hat's OpenShift[1] Kubernetes distribution can do this too - you even get a webhook for GitHub copy and paste. Kubernetes is a toolbox, not a product. There's so much you have to set up yourself (build system, registry, security, logging stack, metrics...). That's fine if you're in the business of selling Kubernetes itself, but otherwise, most teams should use something like OpenShift and not re-invent the wheel. [1]: https://github.com/openshift/origin https://github.com/openshift/origin

tirumaraiselvan 9y ago
This is supposed to be very light weight and independent of any distribution or vendor.
- tango12 9y ago
  (I'm from hasura.io) In fact, this is the component that powers a portion of the "git-ops" on the hasura.io platform. Like one of the commenters said, we also do some pre-push magic right now (esp. for client-side kubernetes templating) for a more complete and opionionated CI/CD workflow.
merb 9y ago
well the problem with openshift on redhat/centos/atomic/whatever is that you still need to regulary update your cluster os's. however when you setup a bootkubed/self-hosted kubeadm cluster you can basically run on coreos and only need to update your self-hosted control plane and kubelet/(etcd, if not self hosted, too) this makes it way, way easier to have an up-to-date os + kube than any other solution (kubelet and etcd can be updated via etcd-wrapper and kubelet-wrapper, so this is even more insane). BUT i think that red hat pushes for an easier upgrade path anyway, that's why they basically bought coreos. for CoreUpdate + Tectonic + MatchBox, so I think that in the end Redhat will have a similar simplier upgrade mechanism, but only time will tell. What would in fact be awesome if CoreUpdate could also update Kubernetes Kubelet's and Etcd and than it would be a no-ops os if something broke you just boot a new system and delete the broken node. And yes kubectl is the ssh/ftp of k8s. https://www.youtube.com/watch?v=07jq-5VbBVQ https://www.youtube.com/watch?v=07jq-5VbBVQ

paulsmal 9y ago

I really like when deployment tools rely on git hooks and doesn't require any other installations in dev environment.

lifeeth_ 9y ago

I was thinking of hasura.io while reading the page only to find this is a hasura project - Keep up the great work

tirumaraiselvan 9y ago
Haha :) This is the first of a few projects open-sourced out of the Hasura platform

vegancap 9y ago

This is a beauty! Excellent work.

anirudhmurali 9y ago

Sexy stuff!

dominotw 9y ago
Hyping ur own stuff on HN?

praseodym 9y ago

We’ve been using a similar solution for a while now: pushes to GitHub trigger Travis CI to build a Docker images. These get pushed to Docker Hub or Quay, which triggers a webhook to Keel.sh[1] running in our Kubernetes cluster. Keel then updates the Kubernetes deployment to perform a rolling update of the application. 1: https://keel.sh https://keel.sh

deforciant 9y ago
Thanks for the shout-out! Keel author here. I have created Keel exactly for these workflows where something (Travis CI, Docker cloud, Google Cloud Builder - my favourite due to the pricing and proper speed) builds an image and then we would want to trigger a deployment update. I was running workloads on GKE and I found that while it's easy to get images being built, there was no straightforward way to automate deployment updates. My goal was to tag a release on Github and get a new version deployed in my environment. After searching for similar tools and couldn't find anything lightweight (spinnaker or jenkins were way too big!) I ended up building Keel. Here's an example if anyone has this problem https://keel.sh/v1/examples/enabling-auto-updates.html https://keel.sh/v1/examples/enabling-auto-updates.html.
- cheez 9y ago
  What is it we can achieve with Keel that we can't with Terraform?
  - alexnewman 9y ago
    Or by awk for that matter. Sorry to be silly but terraform solves a completely different problem, which is what happens when you have disparate cloud resources you want to manage. For instance cloud network resources. Using it to manage containers is... Not it's use case.
- antoncohen 9y ago
  Keel looks nice, thanks for making it! Does Keel include any ability to rollback to a previously deployed version? How to you recommend handling that use case? Keel looks similar to Weave Flux (https://github.com/weaveworks/flux https://github.com/weaveworks/flux). Have you looked at Flux, are you able to compare them? At first glance it looks like Keel is probably easier to setup, but Flux is more feature rich (edits k8s .yaml files in git with updated image tags, has CLI that supports listing deployments and images, can release an old image).

linkmotif 9y ago

I use https://github.com/dminkovsky/kube-cloud-build https://github.com/dminkovsky/kube-cloud-build for this, which uses Google Cloud Container Builder to perform builds. It’s a really nice workflow!

dlor 9y ago

I wonder how this tool handles performing the docker build itself. It looks like it runs a controller inside the cluster, but by default apps running inside kubernetes don't get access to the docker daemon. There are a few existing workarounds, but none of them are terribly appealing. See this longstanding issue for some more context: https://github.com/kubernetes/kubernetes/issues/1806 https://github.com/kubernetes/kubernetes/issues/1806

tirumaraiselvan 9y ago
Docker sock is being mounted from the host [1]. This could be a concern if you are running multi-tenant apps etc. One (slightly hacky) workaround is to have a Persistent Volume mount the hostPath and restrict hostPath mounting in the pod through PSPs. [1] https://github.com/hasura/gitkube/blob/7d0e7f25b811d48089b788593fabf7dd092b0687/manifests/gitkube-setup.yaml#L93 https://github.com/hasura/gitkube/blob/7d0e7f25b811d48089b78...

asadlionpk 9y ago

This looks exactly what I had been looking for these days. None of existing solutions fit my use case. Thanks for making this, I will give it a try soon.

zackify 9y ago

I’ve been using drone.io to do this for a month now. Loving it: - git push starts the process - build and deploy images to google cloud with the plugins/gcr tool - set the image in kubernetes in the next step using the google cloud sdk image I run unit tests as a docker build step. If they fail, it won’t deploy. If the readiness probe fails after setting it in kubernetes, it won’t switch over!

vijaykodam 9y ago

Good work from Team Hasura! Also liked the HDFS on k8s tutorial posted sometime back.

minieggs 9y ago

Interesting, and real nice! I built something similar for my raspberry pi cluster with docker swarm (albeit no testing and tailored to my use case). Kicking myself for not making "production ready" and posting here now.

truesy 9y ago

Close to opening up a service that does this for pull request branches, from GitHub to AWS ECS/Fargate. https://conjure.sh https://conjure.sh Glad to see this kind of workflow gaining momentum.

_5meq 9y ago

Keel does a really good job of this, good to see people are starting to have options with their k8 deployments! My two cents: So far my deployments have been pretty painless, so changing over to something like Gitkube / keel isn't my top priority currently. Configuration management / Creating my kubernetes resources ( deployments / services / ingress / etc. ) in a repeatable and elegant way is my biggest pain point by far. # Helm I've been very unhappy with Helm. Helm is insanely verbose. Go ahead and take a look at the kubernetes/charts repo and look at the various stable charts. Note the huge amount of copy-pasta code hanging out. Too much of my life has been spent wrangling YAML templates that are rendered using the under-featured gotmpl library. Also, helm's client library support is less than stellar. Also, schlepping my configurations around is a pain. In Helm, it's a crazy amount of code to implement a `ingress -> service -> deployment` pattern that is standard for 99% of kubernetes resources. You have to write the same things over and over, which means that making changes takes forever and is brittle. I believe this is due to gotmpl being a poor templating engine for this use case. It doesn't have enough features to allow one to develop decent abstractions over declaring k8 resources The helm tiller doesn't expose a restful API or proper tooling to allow other resources to interact with it effectively, making automated deployments a chore. My only option is to call out to the shell and try and be devensive. Finally, Helm doesn't really do that great a job of validating my resources are going to be valid. `helm install --dry-run` will tell you everything is great, and then break in the middle of an actual installation, leaving a half-configured mess in it's wake. # KSonnet KSonnet looks like an attractive option, it's certainly less verbose. But the documentation isn't there yet, and if you peek under the hood there is a MOUNTAIN of ksonnet code waiting to be read and understood. I belive it's going through some churn currently, so features and impovements have been slow to appear. Also KSonnet doesn't have enough example implementations to hit the ground running when trying it out on my clusters. Targeting individual contexts in Ksonnet provides some nice additional safety, and some very simple implementations using google KMS or something similar could be a really special way to safely store my secret configs at rest, similar to ansible-vault. # Terraform Terraform has this NAILED in the VM space. I declare my environment: `terraform plan` tells me what is going to happen, `terraform apply` checks to ensure I've not protected any of my resources with a flag, and applies the changes it listed if not. I'd love to write my k8 resources in terraform, but it doesn't currently support modern k8 resources, and I don't LOVE how terraform handles storing my tfstate files either way.

pmarreck 9y ago

I deploy via git-push to Travis CI which then runs my test suites and if they pass, continues on with the deployment to the (virtual) hardware server via https://www.gigalixir.com/ https://www.gigalixir.com/ (it's an Elixir Phoenix web app). A properly set up CI is awesome and a huge timesaver.

lewq 9y ago

Looks a lot like http://github.com/weaveworks/flux http://github.com/weaveworks/flux which we're happy users of at dotmesh.com. Glad to see gitops getting some traction! https://www.weave.works/blog/gitops-operations-by-pull-request https://www.weave.works/blog/gitops-operations-by-pull-reque...