4 ms·

Not really on-topic but does anyone have a "fake" Kubernetes? More than just an API server that serves canned responses, but something that actually runs the c

Not really on-topic but does anyone have a "fake" Kubernetes? More than just an API server that serves canned responses, but something that actually runs the control loops but doesn't run any containers. For example, I could configure it in advance like: containers: image: foo/bar:1.2.3 produce_logs: "hello this is fake foo/bar" Then kubectl apply a manifest like: apiVersion: apps/v1 kind: Deployment metadata: name: foobar spec: replicas: 1 template: spec: containers: - name: foobar image: foo/bar:1.2.3 Then run a command like "kubectl get pods": NAME READY STATUS RESTARTS AGE foobar-abcd123456-f0ob4 2/2 Running 0 1s Or even "kubectl get logs foobar-abcd123456-f0ob4": hello this is fake foo/bar! Basically, this would let you test things that interact with kubernetes, without having to have a cluster. This thing would just be a library that you link against your tests, or a program that you start up next to your tests, and it gets you most of the way there. It's better than a fake apiserver you write yourself, where you "know" that the service controller looks at pods that match a selector, then produces endpoints objects that contain that IP address; you could write this kind of low-overhead test without having to know those things (or get it wrong, and code your app to a k8s spec that doesn't exist except in your own mind). That's what I'd really like.

whacker 5y ago
Not quite fake: but this might be close enough? https://github.com/kubernetes-sigs/kind https://github.com/kubernetes-sigs/kind Alternatively you could create a CRI plugin that does what you want. https://github.com/kubernetes/kubernetes/blob/242a97307b34076d5d8f5bbeb154fa4d97c9ef1d/docs/devel/container-runtime-interface.md https://github.com/kubernetes/kubernetes/blob/242a97307b3407...
- jrockway 5y ago
  Yeah, kind is a good compromise. It's more than I want and it does take some time to startup, but it's pretty lightweight and very accurate. Because everything is running locally you can use tools like `perf` to keep an eye on things you're running on k8s, right from your usual workstation. I like it a lot. The CRI plugin sounds like probably the right thing to do. Everything is real up until actually running stuff.
GauntletWizard 5y ago
Kubernetes actually ships with a pretty decent "Fake" infrastructure, but I'll admit (having used it) pretending at control loops is still a PITA. https://pkg.go.dev/k8s.io/client-go/kubernetes/fake https://pkg.go.dev/k8s.io/client-go/kubernetes/fake
remram 5y ago
It's easy enough to mock the images in there to do this via a real container, e.g.: containers: image: busybox args: ["echo", "hello this is fake foo/bar"] and run this on kind
gentele 5y ago
You could deploy the vcluster pod without the syncer. That would give you a k3s cluster without a scheduler. You can create deployments, pods, services etc. but no containers would actually be started. Of course, that means you still need one host cluster though for running the half-baked vclusters on.
Nullabillity 5y ago
You should be able to get close to that with "real" Kubernetes, by only running etcd, kube-apiserver, and kube-controller-manager (without any kubelets, or kube-proxies), which should get you most of the way there (all Pods will just get stuck Pending, but most other stuff should work). Of course, that means that you're still running etcd and managing multiple servers, but at least you don't have to give it any global system access. If you're testing something that depends on pod lifecycles then you should be able to get most of the way by updating the status yourself from your tests. Fake logs may be a bit trickier, but I'd be fairly wary of anything trying to machine-parse logs anyway (unless you're testing a log aggregator, I suppose..).
- moondev 5y ago
  You can also achieve the above very easily with kubeadm which is the upstream kubernetes bootstrapper. All it takes: kubeadm init
- ithkuil 5y ago
  You'd just then need to implement a fake kubelet that pretends it ran something and produced the fake logs.
moondev 5y ago
As others mentioned, just use kind to create a real cluster on demand whenever you need one one. kind create cluster 30 seconds later you will have a fresh cluster to do whatever you want. kind can even do multi-node HA clusters complete with a haproxy container for load balancing the kube-apiserver. Tearing things down is super simple too via "kind delete cluster"

whacker 5y ago

Not quite fake: but this might be close enough? https://github.com/kubernetes-sigs/kind https://github.com/kubernetes-sigs/kind Alternatively you could create a CRI plugin that does what you want. https://github.com/kubernetes/kubernetes/blob/242a97307b34076d5d8f5bbeb154fa4d97c9ef1d/docs/devel/container-runtime-interface.md https://github.com/kubernetes/kubernetes/blob/242a97307b3407...

jrockway 5y ago
Yeah, kind is a good compromise. It's more than I want and it does take some time to startup, but it's pretty lightweight and very accurate. Because everything is running locally you can use tools like `perf` to keep an eye on things you're running on k8s, right from your usual workstation. I like it a lot. The CRI plugin sounds like probably the right thing to do. Everything is real up until actually running stuff.

GauntletWizard 5y ago

Kubernetes actually ships with a pretty decent "Fake" infrastructure, but I'll admit (having used it) pretending at control loops is still a PITA. https://pkg.go.dev/k8s.io/client-go/kubernetes/fake https://pkg.go.dev/k8s.io/client-go/kubernetes/fake

remram 5y ago

It's easy enough to mock the images in there to do this via a real container, e.g.: containers: image: busybox args: ["echo", "hello this is fake foo/bar"] and run this on kind

gentele 5y ago

You could deploy the vcluster pod without the syncer. That would give you a k3s cluster without a scheduler. You can create deployments, pods, services etc. but no containers would actually be started. Of course, that means you still need one host cluster though for running the half-baked vclusters on.

Nullabillity 5y ago

You should be able to get close to that with "real" Kubernetes, by only running etcd, kube-apiserver, and kube-controller-manager (without any kubelets, or kube-proxies), which should get you most of the way there (all Pods will just get stuck Pending, but most other stuff should work). Of course, that means that you're still running etcd and managing multiple servers, but at least you don't have to give it any global system access. If you're testing something that depends on pod lifecycles then you should be able to get most of the way by updating the status yourself from your tests. Fake logs may be a bit trickier, but I'd be fairly wary of anything trying to machine-parse logs anyway (unless you're testing a log aggregator, I suppose..).

moondev 5y ago
You can also achieve the above very easily with kubeadm which is the upstream kubernetes bootstrapper. All it takes: kubeadm init
ithkuil 5y ago
You'd just then need to implement a fake kubelet that pretends it ran something and produced the fake logs.

moondev 5y ago

As others mentioned, just use kind to create a real cluster on demand whenever you need one one. kind create cluster 30 seconds later you will have a fresh cluster to do whatever you want. kind can even do multi-node HA clusters complete with a haproxy container for load balancing the kube-apiserver. Tearing things down is super simple too via "kind delete cluster"