9 ms·

Tools to Run Kubernetes Locally

140 points by EntICOnc 5y ago

whalesalad 5y ago
Running kube is the most trivial part, especially with some of these tools like k3s/k0s. Hard part is deployments. Managing Kube's raw yaml files is tedious. What tools are y'all using there? I am having a tough time with Helm, but I feel like that is the standard answer these days. Back in ~2016 at FarmLogs we built and operated our own (Clojure!) deployment system that was backed by a Github repository. Developers worked with a DSL that was really just a slimmed down version of Kube yaml. This eliminated lots of boilerplate thanks to our company-wide conventions for building microservices. The master branch reflected the intended state of the cluster. It was easy to reason about. I have not found a modern/open-source tool that is similar, but I have to imagine it exists.
- marcc 5y ago
  I agree that "apps" and deploying to the cluster is the hard part. Helm is definitely the most popular choice, but there are others. kpt: https://github.com/GoogleContainerTools/kpt https://github.com/GoogleContainerTools/kpt kapp: https://carvel.dev/kapp/ https://carvel.dev/kapp/ cnab: https://cnab.io https://cnab.io There are many (many) more. But these are somewhat popular recently.
  - whalesalad 5y ago
    Thank you for the links. I had not heard of cnab until now but that looks really interesting. I will look into all of these.
  - ithkuil 5y ago
    I made a variation on the theme of kpt, you may find interesting: https://github.com/mkmik/knot8 https://github.com/mkmik/knot8 (Rendered manpage at https://knot8.io/ https://knot8.io/, if you like that exposition format)
- q3k 5y ago
  > Hard part is deployments. Managing Kube's raw yaml files is tedious. What tools are y'all using there? I am having a tough time with Helm, but I feel like that is the standard answer these days. kubecfg (with jsonnet) or CUE. Gives you actual composability and a powerful programming language for reducing boilerplate. Oh, and no text templating. With kubecfg deploying/updating is: kubecfg update foo.jsonnet No need to first build any Helm images or for any scaffolding. Just get `kubecfg` on your PATH and you're done. Version things the same way you version the rest of your code. Build your own abstractions that solve your particular problems as you go, instead of relying on someone's idea on how your deployments should work. Perfect for building orga-wide microservice deployment boilerplate.
- dharmab 5y ago
  For some more complex deployments (i.e. not simple batch jobs or web services), we've used the Kubernetes API directly from programming languages like Python or Go. Often as a controller running within the cluster. That allows us the full data structures toolset and freedom of a programming language. It reminds me of the old article "Beating the Averages": http://www.paulgraham.com/avg.html http://www.paulgraham.com/avg.html
  - whalesalad 5y ago
    Afraid to admit I had not even considered that idea, good thinking.
  - q3k 5y ago
    Yeah, just writing some code that talks to the Kubernetes API really ought to be more common. It's not a great API, but it certainly beats writing bash scripts that shell out to kubectl and yq. In the same vein, I wish people knew that YAML isn't even really a native Kubernetes thing, it's just one of the many ways you can serialize k8s objects...
    - dharmab 5y ago
      In my experience, engineers with a formal education in Data Structures and Algorithms tend "get it" in a way that others don't- that the underlying data structure is a set of maps/structs and YAML, JSON, etc. are just text formats.
    - sascha_sl 5y ago
      > writing bash scripts that shell out to kubectl and yq We did this for a while, from a container rendering out our stack in Go, when server-side apply was not a reality yet. The main issue is that lifecycle management of resources beyond writing code for a single deployment is hard to do with the typed clients, and the dynamic client used to be even more of a mess than it is now. There's a lot of implicit magic happening, baked into kubectl in a non-reusable way you just couldn't replicate before 1.16.
      - 5y ago
- oweiler 5y ago
  I've found kustomize to be really nice. Much less powerful than Helm, easy to learn, good enough for most tasks. https://github.com/kubernetes-sigs/kustomize https://github.com/kubernetes-sigs/kustomize
- mbushey 5y ago
  Helm is horrible. Please use Kustomize. Life will get better, it did for me. I also use Sops (with goabout/kustomize-sopssecretgenerator) and ArgoCD for a very slick well integrated system. The CRDs in ArgoCD are so well done that it feels like native K8s CD.
  - booleanbetrayal 5y ago
    Big fan of ArgoCD here. The development team is also very responsive with good processes in place, from my experience in the repo.
  - mfer 5y ago
    Kustomize and Helm take different routes. Kustomize expects the user to know Kubernetes resources and their schemas. Helm and charts target the chart creator being being an expert in Kubernetes while the chart consumer (the Helm user) doesn't need to be an expert. They can have a simple interface to deploy an application. One of the issues for those running applications in Kubernetes is the complexity. Kubernetes is complex. So much so that people like Kelsey Hightower have said Kubernetes is a platform for creating platforms. For most people to learn Kubernetes will slow down their velocity which is a negative. Most people need a simpler system on top of Kubernetes to make them effective.
    - sascha_sl 5y ago
      Helm itself is pretty awful through. For instance, when people upgraded to 1.16, where Kubernetes retired apps/v1beta1, Kubernetes automatically transparently converts these types. Helm, having once created these resources as v1beta1 will completely refuse to touch the deployment. Without some specialized extra tooling that people created in the wake of this, resolving this requires deleting the entire namespace and reinstalling the chart. Issue on helm repo closed as wontfix.
      - mfer 5y ago
        This still comes down to assumptions. If someone was using raw kubernetes manifests instead of Helm and charts they would need to know the schemas and details of those manifests. Then, when 1.16 came out they would need to update to the new apiVersion and modify their manifests for any differences in schema. This requires someone who knows and understands k8s resources. This IS NOT a typical app dev or person who wants to run apps in k8s. I've been in numerous circles of app folks who have complained about this expectation from the k8s community. On the main Helm project they have the stance that what's in a chart is up to the charts authors. Just like what's in a debian package is what's up to the debian package authors. That those folks should do a good job maintaining them. That's why it was a wontfix issue. What we see as an underlying issue with k8s is the experience that app devs and people who want to run apps have to deal with. It's currently like expecting an app dev to write in assembly and know when assembly codes are deprecated or changed. It's in the pre-high level language phase. This comparison to assembly is one that's come out of the k8s community itself.
- jrsdav 5y ago
  Have you heard of Flux? It's not a manifest abstraction tool per se (like Helm or Kustomize), but automates the management of your manifests using GitOps. It sounds like it might be what you're looking for. https://fluxcd.io/docs/concepts/ https://fluxcd.io/docs/concepts/
- gray_-_wolf 5y ago
  We are using tanka [0] and are failry happy with it. The jsonnet language allows nice layer of abstraction while still allowing to directly generate kubernetes resources when necessary. We wrote in-house library to map (very) simplified .jsonnet file to kuberenetes deployment; after all, 90% deployements are mostly same and differs in minor details only. Seems to work reasonably well. 0: https://github.com/grafana/tanka https://github.com/grafana/tanka
- luxurytent 5y ago
  This is how I feel too. At my workplace, we can easily spin up a kube cluster locally but our Helm charts aren't transportable for local development. They can be modified, torn apart, extracted from, but we don't have a seamless and elegant way to say "Hey, this thing in production, that's running via Helm/k8s, I want the same thing locally ... with these environment variables adjusted" I don't think that's a difficult thing to accomplish and there is appetite from others I work with to do so, but there's _effort_ behind it. Maybe there are smarter ways and we just don't know it yet.
- birdyrooster 5y ago
  jsonnet, works great and is composable however you like
- stormbrew 5y ago
  > Back in ~2016 at FarmLogs we built and operated our own (Clojure!) deployment system that was backed by a Github repository. Developers worked with a DSL that was really just a slimmed down version of Kube yaml. This eliminated lots of boilerplate thanks to our company-wide conventions for building microservices. The master branch reflected the intended state of the cluster. It was easy to reason about. The lack of this is what frustrates me so much about all of these things. Writing some bundle of yaml but just telling the management software "here go run this 'app'" with no state or version management of it makes me feel like I'm running a stack like it's a windows desktop. Just random shit everywhere and I don't know what any of it is. Maybe you're keeping it in a git repo but is what's running live what's in the git repo? Who the heck knows. All feels like a huge step backwards to me. Let me use version control as a source of truth, not just a spot to throw things in.
  - topspin 5y ago
    "Maybe you're keeping it in a git repo but is what's running live what's in the git repo? Who the heck knows." See Argo CD.
- mushishi 5y ago
  In previous project, we had Kotlin DSL based definitions, and even though I only read it, it seemed to make a lot of sense: static typing, and reuse of code e.g., between QA and production environment definitions. I think it may have been this: https://github.com/fkorotkov/k8s-kotlin-dsl https://github.com/fkorotkov/k8s-kotlin-dsl
rckrd 5y ago
One interesting fact is that minikube actually used to be very similar to k3s. (I used to be a maintainer of minikube) We called it localkube, and it was an all-in-one binary with features stripped out that ran the kubelet, apiserver, kube-proxy, etcd, and other components as Go routines. Eventually it had to be removed - upstream k8s was moving too fast, and there were fundamental issues that couldn't be fixed without large patches (TPRs, the predecessor to CRDs were one). The problem with this approach is that it requires a ton of maintenance and there can be subtle bugs that come up. The other distinction not found on this list is that only Docker for Desktop and minikube are cross-platform solutions. Running Kubernetes and docker on Linux is somewhat trivial, but managing a lightweight virtual machine and adding that extra layer of abstraction is what makes it really difficult.
- gizdan 5y ago
  Out of curiousity, how come k3s has been able to keep up? Or is my assumption incorrect?
  - rckrd 5y ago
    (1) Maturity of Kubernetes APIs. Fewer upstream changes. (2) It's one of the main products of a venture-backed company. At Google, at the time, it was tough to get engineers on products like minikube.
- SomaticPirate 5y ago
  Having worked around k3s quite a bit, one aspect I miss from minikube is docker image loading. This hurts for using k3s as a local dev cluster vs minikube & kind.
  - comma_at 5y ago
    In k3d this is already available. https://k3d.io/usage/commands/k3d_image_import/ https://k3d.io/usage/commands/k3d_image_import/
- 0xbadcafebee 5y ago
sul_tasto 5y ago
I’m surprised docker desktop wasn’t mentioned: https://www.docker.com/products/docker-desktop https://www.docker.com/products/docker-desktop
- jayd16 5y ago
  Yeah, and a simple docker compose for local dev does the job pretty easily.
singhrac 5y ago
I’ve recently become interested in learning Kubernetes for some work I’m doing. But I’m not a sysops engineer or even really a backend person, so I kind of want to pick the right horse to back and learn some simple tools so that I can get builds onto GCP or AWS. Mostly I want to make sure there is a StackOverflow post for whatever problem I will encounter. What do people think is the best ‘stack’ to invest in? I’m also curious about experiences with CapRover and/or FastAPI.
- mplewis 5y ago
  Use a language you know and like, then use its popular web framework. FastAPI is just fine. Package your apps into containers. Then run them somewhere. DigitalOcean Apps, AWS Lambda, and Google Cloud Run are all fine. So is Kubernetes.
- mwaitjmp 5y ago
  Terraform to create and manage the cluster and then helm to deploy any applications. Those two tools have been around for a while and you’ll find plenty of documentation and other people looking for help online.
atombender 5y ago
I found that the biggest blocker to running Kubernetes locally is that it's very resource-hungry, especially on a Mac with Linux virtualization. This also true about Docker on Mac, but Kubernetes is worse. I have not tried k3s or any of the other attempts to produce a stripped-down version, so I'm curious if they fix this problem. But the fact remains that the Kubernetes daemons are heavily based on polling; even a Kubernetes instance that does nothing will be using a lot of CPU. Right now, the tool of my dreams does not run anything locally at all, but transparently maintains a small remote Kubernetes cluster in the cloud of my choice. The benefit here is that I could get all the tools at my disposal, such as public load balancers, ability to effortlessly scale up my nodes when needed, etc.
mfer 5y ago
A couple things I notice about this list are the use of terminal and the reliance on available Linux or Docker. For example, minikube (by default) and k3d both look for Docker Desktop to be installed. We've been working on something taking a different direction that makes it a simple app to run. It's called Rancher Desktop (https://rancherdesktop.io/ https://rancherdesktop.io/). It uses k3s under the hood. The goal is to make running k8s locally as easy as running a normal desktop app. Note, as the creator of Rancher Desktop I'm entirely biased and this is a bit of a shameless plug.
silasb 5y ago
Off topic: I really dig `fly` from `fly.io` is there any model that is similar to that. Akin to hereoku but for k8s?
- loloquwowndueo 5y ago
  Do you mean flyctl? “fly” is the CLI tool from concourse-Ci.org.
  - silasb 5y ago
    Yes, `flyctl` from https://fly.io/docs/getting-started/login-to-fly/ https://fly.io/docs/getting-started/login-to-fly/
nickjj 5y ago
I tried pretty much all of them and settled with kind. It takes 1 command and 1 minute to spin up a 3 node cluster from scratch, everything works the same as you would expect on a real managed Kubernetes cluster and it took a few lines of YAML to config the cluster so that the ingress works in such a way where you can write your manifests exactly how you would on a managed provider. It's also super easy to get your local Docker images into the cluster. Did anyone else come to the same conclusion? Maybe I'm missing something but you can't really ask for much more than the above.
- 8bitbuddhist 5y ago
  Agreed, Kind is perfect for running disposable, reproducible multi-node clusters that are essentially upstream K8s.
- stonecharioteer 5y ago
  First time I've heard of Kind. I run k3s and minikube. How is Kind different?
  - nickjj 5y ago
    I had lots of trouble with minikube. Often times it failed to create a cluster with low level networking issues related to hyper-v. It was really flaky. I could delete and create a cluster and it felt like half the time it worked, half the time it failed. I was using the Docker back-end too. For k3s, it mainly comes down to wanting to use k8s not k3s. I'd like my local cluster to be as close as possible to what's running in a real managed k8s cluster on a major cloud provider. This might come off as naive but I want to use k8s in dev / local testing because I'm using k8s in production. I also found kind's documentation much better than k3s for wanting to accomplish the task of setting up a cluster and using an ingress. It also had easy to find and follow docs for WSL 2 to configure extra port mappings in the cluster config so I could access external services (managed with an nginx ingress) over a defined port. Basically the dev experience has been flawless.
oxplot 5y ago
If you're on linux, `kind` is the way to go
orsenthil 5y ago
> tools to run kubernetes locally And do what? This is the most important question.
- bo0tzz 5y ago
  Development environments?
revskill 5y ago
K8S is fine until u want to scale a service which uses local storage. Is there any cheapest cloud service to provide such kind of plan ?
- cyberpunk 5y ago
  Don’t do that…
- dilyevsky 5y ago
  Equinix Metal or Vultr
thiht 5y ago
Honest question: what do you need a local Kubernetes for? It’s an orchestrator, surely you don’t need that in local? Don’t you have a way to mock your services? Or simple compose files for local integration? I have worked with Marathon and Kubernetes in the last few years and never had the need to deploy them locally, it seems strange to even need that
- ibeckermayer 5y ago
  Our team builds a custom controller for a CRD for which it’s useful to be able to run and develop locally, rather than have each developer on the team run their own cluster in the cloud
- marwatk 5y ago
  If you're using k8s in prod and compose locally are you really testing integration?
  - GauntletWizard 5y ago
    No, I'm testing integration in my staging environment, which is also running kubernetes. Local development is not for testing integration. The interface and surfaces should be well defined enough that you don't worry about the difference between development and integration, but when you do have issues they should fall along the lines of "Differences easily spotted on the edge between compose and k8s" and if you can handle those, you're good
- dilyevsky 5y ago
  Dev environment. Mocks are a bad way to test certain things such as db interactions and generally can be a pain
- chii 5y ago
  > Don’t you have a way to mock your services? Or simple compose files for local integration? now your dev env is very different from how production works. Problems that show up in production won't easily be reproducible in dev. And if you are testing in this dev env, you aren't really testing, so implicitly turn to testing in production!

140 points by EntICOnc 5y ago

Running kube is the most trivial part, especially with some of these tools like k3s/k0s. Hard part is deployments. Managing Kube's raw yaml files is tedious. What tools are y'all using there? I am having a tough time with Helm, but I feel like that is the standard answer these days. Back in ~2016 at FarmLogs we built and operated our own (Clojure!) deployment system that was backed by a Github repository. Developers worked with a DSL that was really just a slimmed down version of Kube yaml. This eliminated lots of boilerplate thanks to our company-wide conventions for building microservices. The master branch reflected the intended state of the cluster. It was easy to reason about. I have not found a modern/open-source tool that is similar, but I have to imagine it exists.

marcc 5y ago
I agree that "apps" and deploying to the cluster is the hard part. Helm is definitely the most popular choice, but there are others. kpt: https://github.com/GoogleContainerTools/kpt https://github.com/GoogleContainerTools/kpt kapp: https://carvel.dev/kapp/ https://carvel.dev/kapp/ cnab: https://cnab.io https://cnab.io There are many (many) more. But these are somewhat popular recently.
- whalesalad 5y ago
  Thank you for the links. I had not heard of cnab until now but that looks really interesting. I will look into all of these.
- ithkuil 5y ago
  I made a variation on the theme of kpt, you may find interesting: https://github.com/mkmik/knot8 https://github.com/mkmik/knot8 (Rendered manpage at https://knot8.io/ https://knot8.io/, if you like that exposition format)
q3k 5y ago
> Hard part is deployments. Managing Kube's raw yaml files is tedious. What tools are y'all using there? I am having a tough time with Helm, but I feel like that is the standard answer these days. kubecfg (with jsonnet) or CUE. Gives you actual composability and a powerful programming language for reducing boilerplate. Oh, and no text templating. With kubecfg deploying/updating is: kubecfg update foo.jsonnet No need to first build any Helm images or for any scaffolding. Just get `kubecfg` on your PATH and you're done. Version things the same way you version the rest of your code. Build your own abstractions that solve your particular problems as you go, instead of relying on someone's idea on how your deployments should work. Perfect for building orga-wide microservice deployment boilerplate.
dharmab 5y ago
For some more complex deployments (i.e. not simple batch jobs or web services), we've used the Kubernetes API directly from programming languages like Python or Go. Often as a controller running within the cluster. That allows us the full data structures toolset and freedom of a programming language. It reminds me of the old article "Beating the Averages": http://www.paulgraham.com/avg.html http://www.paulgraham.com/avg.html
- whalesalad 5y ago
  Afraid to admit I had not even considered that idea, good thinking.
- q3k 5y ago
  Yeah, just writing some code that talks to the Kubernetes API really ought to be more common. It's not a great API, but it certainly beats writing bash scripts that shell out to kubectl and yq. In the same vein, I wish people knew that YAML isn't even really a native Kubernetes thing, it's just one of the many ways you can serialize k8s objects...
  - dharmab 5y ago
    In my experience, engineers with a formal education in Data Structures and Algorithms tend "get it" in a way that others don't- that the underlying data structure is a set of maps/structs and YAML, JSON, etc. are just text formats.
  - sascha_sl 5y ago
    > writing bash scripts that shell out to kubectl and yq We did this for a while, from a container rendering out our stack in Go, when server-side apply was not a reality yet. The main issue is that lifecycle management of resources beyond writing code for a single deployment is hard to do with the typed clients, and the dynamic client used to be even more of a mess than it is now. There's a lot of implicit magic happening, baked into kubectl in a non-reusable way you just couldn't replicate before 1.16.
    - 5y ago
oweiler 5y ago
I've found kustomize to be really nice. Much less powerful than Helm, easy to learn, good enough for most tasks. https://github.com/kubernetes-sigs/kustomize https://github.com/kubernetes-sigs/kustomize
mbushey 5y ago
Helm is horrible. Please use Kustomize. Life will get better, it did for me. I also use Sops (with goabout/kustomize-sopssecretgenerator) and ArgoCD for a very slick well integrated system. The CRDs in ArgoCD are so well done that it feels like native K8s CD.
- booleanbetrayal 5y ago
  Big fan of ArgoCD here. The development team is also very responsive with good processes in place, from my experience in the repo.
- mfer 5y ago
  Kustomize and Helm take different routes. Kustomize expects the user to know Kubernetes resources and their schemas. Helm and charts target the chart creator being being an expert in Kubernetes while the chart consumer (the Helm user) doesn't need to be an expert. They can have a simple interface to deploy an application. One of the issues for those running applications in Kubernetes is the complexity. Kubernetes is complex. So much so that people like Kelsey Hightower have said Kubernetes is a platform for creating platforms. For most people to learn Kubernetes will slow down their velocity which is a negative. Most people need a simpler system on top of Kubernetes to make them effective.
  - sascha_sl 5y ago
    Helm itself is pretty awful through. For instance, when people upgraded to 1.16, where Kubernetes retired apps/v1beta1, Kubernetes automatically transparently converts these types. Helm, having once created these resources as v1beta1 will completely refuse to touch the deployment. Without some specialized extra tooling that people created in the wake of this, resolving this requires deleting the entire namespace and reinstalling the chart. Issue on helm repo closed as wontfix.
    - mfer 5y ago
      This still comes down to assumptions. If someone was using raw kubernetes manifests instead of Helm and charts they would need to know the schemas and details of those manifests. Then, when 1.16 came out they would need to update to the new apiVersion and modify their manifests for any differences in schema. This requires someone who knows and understands k8s resources. This IS NOT a typical app dev or person who wants to run apps in k8s. I've been in numerous circles of app folks who have complained about this expectation from the k8s community. On the main Helm project they have the stance that what's in a chart is up to the charts authors. Just like what's in a debian package is what's up to the debian package authors. That those folks should do a good job maintaining them. That's why it was a wontfix issue. What we see as an underlying issue with k8s is the experience that app devs and people who want to run apps have to deal with. It's currently like expecting an app dev to write in assembly and know when assembly codes are deprecated or changed. It's in the pre-high level language phase. This comparison to assembly is one that's come out of the k8s community itself.
jrsdav 5y ago
Have you heard of Flux? It's not a manifest abstraction tool per se (like Helm or Kustomize), but automates the management of your manifests using GitOps. It sounds like it might be what you're looking for. https://fluxcd.io/docs/concepts/ https://fluxcd.io/docs/concepts/
gray_-_wolf 5y ago
We are using tanka [0] and are failry happy with it. The jsonnet language allows nice layer of abstraction while still allowing to directly generate kubernetes resources when necessary. We wrote in-house library to map (very) simplified .jsonnet file to kuberenetes deployment; after all, 90% deployements are mostly same and differs in minor details only. Seems to work reasonably well. 0: https://github.com/grafana/tanka https://github.com/grafana/tanka
luxurytent 5y ago
This is how I feel too. At my workplace, we can easily spin up a kube cluster locally but our Helm charts aren't transportable for local development. They can be modified, torn apart, extracted from, but we don't have a seamless and elegant way to say "Hey, this thing in production, that's running via Helm/k8s, I want the same thing locally ... with these environment variables adjusted" I don't think that's a difficult thing to accomplish and there is appetite from others I work with to do so, but there's _effort_ behind it. Maybe there are smarter ways and we just don't know it yet.
birdyrooster 5y ago
jsonnet, works great and is composable however you like
stormbrew 5y ago
> Back in ~2016 at FarmLogs we built and operated our own (Clojure!) deployment system that was backed by a Github repository. Developers worked with a DSL that was really just a slimmed down version of Kube yaml. This eliminated lots of boilerplate thanks to our company-wide conventions for building microservices. The master branch reflected the intended state of the cluster. It was easy to reason about. The lack of this is what frustrates me so much about all of these things. Writing some bundle of yaml but just telling the management software "here go run this 'app'" with no state or version management of it makes me feel like I'm running a stack like it's a windows desktop. Just random shit everywhere and I don't know what any of it is. Maybe you're keeping it in a git repo but is what's running live what's in the git repo? Who the heck knows. All feels like a huge step backwards to me. Let me use version control as a source of truth, not just a spot to throw things in.
- topspin 5y ago
  "Maybe you're keeping it in a git repo but is what's running live what's in the git repo? Who the heck knows." See Argo CD.
mushishi 5y ago
In previous project, we had Kotlin DSL based definitions, and even though I only read it, it seemed to make a lot of sense: static typing, and reuse of code e.g., between QA and production environment definitions. I think it may have been this: https://github.com/fkorotkov/k8s-kotlin-dsl https://github.com/fkorotkov/k8s-kotlin-dsl

rckrd 5y ago

One interesting fact is that minikube actually used to be very similar to k3s. (I used to be a maintainer of minikube) We called it localkube, and it was an all-in-one binary with features stripped out that ran the kubelet, apiserver, kube-proxy, etcd, and other components as Go routines. Eventually it had to be removed - upstream k8s was moving too fast, and there were fundamental issues that couldn't be fixed without large patches (TPRs, the predecessor to CRDs were one). The problem with this approach is that it requires a ton of maintenance and there can be subtle bugs that come up. The other distinction not found on this list is that only Docker for Desktop and minikube are cross-platform solutions. Running Kubernetes and docker on Linux is somewhat trivial, but managing a lightweight virtual machine and adding that extra layer of abstraction is what makes it really difficult.

gizdan 5y ago
Out of curiousity, how come k3s has been able to keep up? Or is my assumption incorrect?
- rckrd 5y ago
  (1) Maturity of Kubernetes APIs. Fewer upstream changes. (2) It's one of the main products of a venture-backed company. At Google, at the time, it was tough to get engineers on products like minikube.
SomaticPirate 5y ago
Having worked around k3s quite a bit, one aspect I miss from minikube is docker image loading. This hurts for using k3s as a local dev cluster vs minikube & kind.
- comma_at 5y ago
  In k3d this is already available. https://k3d.io/usage/commands/k3d_image_import/ https://k3d.io/usage/commands/k3d_image_import/
0xbadcafebee 5y ago

sul_tasto 5y ago

I’m surprised docker desktop wasn’t mentioned: https://www.docker.com/products/docker-desktop https://www.docker.com/products/docker-desktop

jayd16 5y ago
Yeah, and a simple docker compose for local dev does the job pretty easily.

singhrac 5y ago

I’ve recently become interested in learning Kubernetes for some work I’m doing. But I’m not a sysops engineer or even really a backend person, so I kind of want to pick the right horse to back and learn some simple tools so that I can get builds onto GCP or AWS. Mostly I want to make sure there is a StackOverflow post for whatever problem I will encounter. What do people think is the best ‘stack’ to invest in? I’m also curious about experiences with CapRover and/or FastAPI.

mplewis 5y ago
Use a language you know and like, then use its popular web framework. FastAPI is just fine. Package your apps into containers. Then run them somewhere. DigitalOcean Apps, AWS Lambda, and Google Cloud Run are all fine. So is Kubernetes.
mwaitjmp 5y ago
Terraform to create and manage the cluster and then helm to deploy any applications. Those two tools have been around for a while and you’ll find plenty of documentation and other people looking for help online.

atombender 5y ago

I found that the biggest blocker to running Kubernetes locally is that it's very resource-hungry, especially on a Mac with Linux virtualization. This also true about Docker on Mac, but Kubernetes is worse. I have not tried k3s or any of the other attempts to produce a stripped-down version, so I'm curious if they fix this problem. But the fact remains that the Kubernetes daemons are heavily based on polling; even a Kubernetes instance that does nothing will be using a lot of CPU. Right now, the tool of my dreams does not run anything locally at all, but transparently maintains a small remote Kubernetes cluster in the cloud of my choice. The benefit here is that I could get all the tools at my disposal, such as public load balancers, ability to effortlessly scale up my nodes when needed, etc.

mfer 5y ago

A couple things I notice about this list are the use of terminal and the reliance on available Linux or Docker. For example, minikube (by default) and k3d both look for Docker Desktop to be installed. We've been working on something taking a different direction that makes it a simple app to run. It's called Rancher Desktop (https://rancherdesktop.io/ https://rancherdesktop.io/). It uses k3s under the hood. The goal is to make running k8s locally as easy as running a normal desktop app. Note, as the creator of Rancher Desktop I'm entirely biased and this is a bit of a shameless plug.

silasb 5y ago

Off topic: I really dig `fly` from `fly.io` is there any model that is similar to that. Akin to hereoku but for k8s?

loloquwowndueo 5y ago
Do you mean flyctl? “fly” is the CLI tool from concourse-Ci.org.
- silasb 5y ago
  Yes, `flyctl` from https://fly.io/docs/getting-started/login-to-fly/ https://fly.io/docs/getting-started/login-to-fly/

nickjj 5y ago

I tried pretty much all of them and settled with kind. It takes 1 command and 1 minute to spin up a 3 node cluster from scratch, everything works the same as you would expect on a real managed Kubernetes cluster and it took a few lines of YAML to config the cluster so that the ingress works in such a way where you can write your manifests exactly how you would on a managed provider. It's also super easy to get your local Docker images into the cluster. Did anyone else come to the same conclusion? Maybe I'm missing something but you can't really ask for much more than the above.

8bitbuddhist 5y ago
Agreed, Kind is perfect for running disposable, reproducible multi-node clusters that are essentially upstream K8s.
stonecharioteer 5y ago
First time I've heard of Kind. I run k3s and minikube. How is Kind different?
- nickjj 5y ago
  I had lots of trouble with minikube. Often times it failed to create a cluster with low level networking issues related to hyper-v. It was really flaky. I could delete and create a cluster and it felt like half the time it worked, half the time it failed. I was using the Docker back-end too. For k3s, it mainly comes down to wanting to use k8s not k3s. I'd like my local cluster to be as close as possible to what's running in a real managed k8s cluster on a major cloud provider. This might come off as naive but I want to use k8s in dev / local testing because I'm using k8s in production. I also found kind's documentation much better than k3s for wanting to accomplish the task of setting up a cluster and using an ingress. It also had easy to find and follow docs for WSL 2 to configure extra port mappings in the cluster config so I could access external services (managed with an nginx ingress) over a defined port. Basically the dev experience has been flawless.

oxplot 5y ago

If you're on linux, `kind` is the way to go

orsenthil 5y ago

> tools to run kubernetes locally And do what? This is the most important question.

bo0tzz 5y ago
Development environments?

revskill 5y ago

K8S is fine until u want to scale a service which uses local storage. Is there any cheapest cloud service to provide such kind of plan ?

cyberpunk 5y ago
Don’t do that…
dilyevsky 5y ago
Equinix Metal or Vultr

thiht 5y ago

Honest question: what do you need a local Kubernetes for? It’s an orchestrator, surely you don’t need that in local? Don’t you have a way to mock your services? Or simple compose files for local integration? I have worked with Marathon and Kubernetes in the last few years and never had the need to deploy them locally, it seems strange to even need that

ibeckermayer 5y ago
Our team builds a custom controller for a CRD for which it’s useful to be able to run and develop locally, rather than have each developer on the team run their own cluster in the cloud
marwatk 5y ago
If you're using k8s in prod and compose locally are you really testing integration?
- GauntletWizard 5y ago
  No, I'm testing integration in my staging environment, which is also running kubernetes. Local development is not for testing integration. The interface and surfaces should be well defined enough that you don't worry about the difference between development and integration, but when you do have issues they should fall along the lines of "Differences easily spotted on the edge between compose and k8s" and if you can handle those, you're good
dilyevsky 5y ago
Dev environment. Mocks are a bad way to test certain things such as db interactions and generally can be a pain
chii 5y ago
> Don’t you have a way to mock your services? Or simple compose files for local integration? now your dev env is very different from how production works. Problems that show up in production won't easily be reproducible in dev. And if you are testing in this dev env, you aren't really testing, so implicitly turn to testing in production!