5 ms·

That's what I've been saying all along. For me, when I use something I need to understand it. I'm usually a quick learner (3rd iteration is where I start making

That's what I've been saying all along. For me, when I use something I need to understand it. I'm usually a quick learner (3rd iteration is where I start making sense of a complex topic). I beat my head against k8s for 2 weeks, every day, I wanted to setup a 3 node control pane on my own, and it was just too hard. I gave up. Also, I don't have many resources at my disposal, k8s requires a separate storage cluster. All-together I would've required 9 servers, 3 control pane, 3 app or pods, 3 storage. Way too expensive. And then think further, all this has to be updated sometime. I don't have the brain and time resources to do that. So, if I want k8s, I need to pay for expensive cloud. Or, I can remain with my 1 server with backups and build systems on demand. Yes, I hear the booksmarties scream "but but failsafety". 1 incubator server, specialized systems when something takes off. Some companies can afford to pay for expensive cloud, more power to them. I can't. I'm busy solving problems at hand, not dealing with additional problems of a system I don't fully understand. Yes, k8s is a nice comfortable way, if you can afford it. But it's expensive and wasteful of resources. And we do have a climate crisis.

mjburgess 3y ago
It's better to pay the cost of abstraction when you've good evidence of its necessity
ncrmro 3y ago
K3s/k9s can be ran on a single server and gives you the declarative benefits and the entire helm ecosystem. Everyone assumes you have to run multi node.
thinkmassive 3y ago
> All-together I would've required 9 servers, 3 control pane, 3 app or pods, 3 storage. … Yes, k8s is a nice comfortable way, if you can afford it. But it’s expensive and wasteful of resources. And we do have a climate crisis. You can run each of those roles on the same machine, achieving high availability from 3 total. For learning purposes you can run this all in VMs on a single machine. There are plenty of good reasons not to try kubernetes when it doesn’t fit your use case, but the overhead it imposes is mostly in time and expertise. Claiming it’s more costly in compute resources, and especially contributing climate crisis, is reaching.
hosh 3y ago
Someone is building a systemd based kublet replacement. Why? Because Kubernetes has some great ideas on self-healing and resiliency. But if you are small, and can work with a single instance for availability, then probably should stick with systemd. You could also use podman to get a small bit of the kubernetes ideas for working with containers. The next level of scale wouldn’t be the full k8s but more like k0s, or k3s.
- AlexITC 3y ago
  > Someone is building a systemd based kublet replacement. I'm working on a product that uses something like this internally, basically, we provide multiple preview-environments in the same big server, these are managed by systemd. This is the first time I get to think that this may be a project by itself, ping me if you have any ideas or want to get into a discussion on this topic.
  - hosh 3y ago
    I forgot who built it, but it isn’t a strict replacement. One of the rationale for doing so was identifying daemonsets as an anti-pattern.
tarruda 3y ago
With "kind" (kubernetes in docker) setting up a one node cluster is as easy as typing "kind create cluster"
- vbezhenar 3y ago
  Setting up a one node cluster is as easy as typing kubeadm init. You don't need anything fancy for that.
hujun 3y ago
there are a few k8s distributions works on single server like microk8s, kind, and they are not difficult to setup on a single server.
vbezhenar 3y ago
You don't need 3 control planes. If your control plane will be down for some time, your cluster will be OK. Even if your control plane will be down for prolonged time, your small cluster likely will survive. 3 control planes is good, but if you can't afford it, it's not necessary. Just make sure to backup etcd and keys for your control plane and learn how to recover it in case of hardware failure. Kubernetes does not require storage cluster. If you need it, you can run it. But it's not required. Please think about your arguments when comparing kubernetes with any other solution. If you don't need storage cluster with any other solution, then you don't need it with kubernetes. Kubernetes at its minimum is 3GB RAM and 0.5 CPU for one control plane. That's all you need. Everything extra is not necessary. People often try to build some kind of super-resilient cluster with kubernetes, which is suitable for huge company with hundreds of servers, while comparing it with one host docker compose. If your task is simple, don't make it more complex than necessary. Kubernetes can be simple or it can be complex, it depends on you. What you do need for kubernetes is knowledge and experience. I spent several months tinkering with cluster before I dared to deploy something valuable there. Although I used kubeadm, with cloud offering few things are simpler.
- funcDropShadow 3y ago
  > If you don't need storage cluster with any other solution, then you don't need it with kubernetes. That is only half of the truth. If your application needs a database, and which application doesn't, you have to store its data somewhere. So you either need a storage cluster, or you open the can of warms called local volumes and (anti-)affinity rules. For smaller application deployments, hosting a separate kubernetes cluster is extremely expensive in terms of time spend to debug infrastructure issues. k8s introduces orders of magnitudes of additional complexity to networking, dns, storage, routing, logging, etc. All in the name of making these exact same things easier for the developer. This can be a good separation of work, if you are large enough to afford a dedicated k8s maintenance team. But often it is not.
  - figassis 3y ago
    Keep your db outside the cluster. Use a managed db or just host it on its own server. You should be doing that even if not using k8s anyway. Don’t share memory and cpu with your db. Regarding storage, try to build stateless apps that only need temporary storage that don’t need to outlive a request (like uploads). For long lived unchanging assets, you can try including in the container (ie. defaults.json, countries.json, samples.json, etc). For assets that need to be fresh, try to have the app generate at startup (eg. Email vírus definitions) or via init container. I think this will cover most use cases. This won’t work with apps that depend on shared storage like Wordpress, but Wordpress is horrible and I can’t support you running it :-)
erinaceousjones 3y ago
We went through these loops where I work due to the "you have to use k8s now" thing. Your control plane and worker node can be the same machine. Rancher + Rancher Kubernetes Engine made it a lot easier for us to manage as maintaining that is simply "run this docker container" and it deploys all the k8s gubbins for you. And k3s is a lot more lightweight on resources than k8s. Stuff like certificate rotation and kubernetes version upgrades are handled for us, it's reduced to clicking buttons on UI. Storage can be local, it's just not / wasn't well explained in the documentation. PersistentVolume[Claim]s can use directories on the host, those PVs/PVCs can be shared between many pods and you can mount subdirectories into each pod using `subPath`. You can get it all running on a laptop using a couple hundred MB of memory, all in all. So, not going to be ideal for running on a Raspberry Pi, but on par with a lightweight linux desktop environment in terms of resource usage. I hate the complexity sometimes, but then I still end up having the same learning curves with things like SystemD when I use them anyway - every "run a bunch of stuff on my machine and restart it if it breaks" system seems to be rife with edge cases and behaviours which you have to dig deep into the documentation to see - and we don't _want_ to RTFM if we don't expressly _have_ to :-) ^ Although, okay, SystemD doesn't require you to have random HTTPS authenticated endpoints to talk to it and TLS certificates which expire every year and consists of _one_ init process rather than like 7 interdependent docker containers which can restart at any time and rely on container infrastructure to be running and argh. Okay yeah the flashbacks are happening and I agree with everything you've said again.

mjburgess 3y ago

It's better to pay the cost of abstraction when you've good evidence of its necessity

ncrmro 3y ago

K3s/k9s can be ran on a single server and gives you the declarative benefits and the entire helm ecosystem. Everyone assumes you have to run multi node.

thinkmassive 3y ago

> All-together I would've required 9 servers, 3 control pane, 3 app or pods, 3 storage. … Yes, k8s is a nice comfortable way, if you can afford it. But it’s expensive and wasteful of resources. And we do have a climate crisis. You can run each of those roles on the same machine, achieving high availability from 3 total. For learning purposes you can run this all in VMs on a single machine. There are plenty of good reasons not to try kubernetes when it doesn’t fit your use case, but the overhead it imposes is mostly in time and expertise. Claiming it’s more costly in compute resources, and especially contributing climate crisis, is reaching.

hosh 3y ago

Someone is building a systemd based kublet replacement. Why? Because Kubernetes has some great ideas on self-healing and resiliency. But if you are small, and can work with a single instance for availability, then probably should stick with systemd. You could also use podman to get a small bit of the kubernetes ideas for working with containers. The next level of scale wouldn’t be the full k8s but more like k0s, or k3s.

AlexITC 3y ago
> Someone is building a systemd based kublet replacement. I'm working on a product that uses something like this internally, basically, we provide multiple preview-environments in the same big server, these are managed by systemd. This is the first time I get to think that this may be a project by itself, ping me if you have any ideas or want to get into a discussion on this topic.
- hosh 3y ago
  I forgot who built it, but it isn’t a strict replacement. One of the rationale for doing so was identifying daemonsets as an anti-pattern.

tarruda 3y ago

With "kind" (kubernetes in docker) setting up a one node cluster is as easy as typing "kind create cluster"

vbezhenar 3y ago
Setting up a one node cluster is as easy as typing kubeadm init. You don't need anything fancy for that.

hujun 3y ago

there are a few k8s distributions works on single server like microk8s, kind, and they are not difficult to setup on a single server.

vbezhenar 3y ago

You don't need 3 control planes. If your control plane will be down for some time, your cluster will be OK. Even if your control plane will be down for prolonged time, your small cluster likely will survive. 3 control planes is good, but if you can't afford it, it's not necessary. Just make sure to backup etcd and keys for your control plane and learn how to recover it in case of hardware failure. Kubernetes does not require storage cluster. If you need it, you can run it. But it's not required. Please think about your arguments when comparing kubernetes with any other solution. If you don't need storage cluster with any other solution, then you don't need it with kubernetes. Kubernetes at its minimum is 3GB RAM and 0.5 CPU for one control plane. That's all you need. Everything extra is not necessary. People often try to build some kind of super-resilient cluster with kubernetes, which is suitable for huge company with hundreds of servers, while comparing it with one host docker compose. If your task is simple, don't make it more complex than necessary. Kubernetes can be simple or it can be complex, it depends on you. What you do need for kubernetes is knowledge and experience. I spent several months tinkering with cluster before I dared to deploy something valuable there. Although I used kubeadm, with cloud offering few things are simpler.

funcDropShadow 3y ago
> If you don't need storage cluster with any other solution, then you don't need it with kubernetes. That is only half of the truth. If your application needs a database, and which application doesn't, you have to store its data somewhere. So you either need a storage cluster, or you open the can of warms called local volumes and (anti-)affinity rules. For smaller application deployments, hosting a separate kubernetes cluster is extremely expensive in terms of time spend to debug infrastructure issues. k8s introduces orders of magnitudes of additional complexity to networking, dns, storage, routing, logging, etc. All in the name of making these exact same things easier for the developer. This can be a good separation of work, if you are large enough to afford a dedicated k8s maintenance team. But often it is not.
- figassis 3y ago
  Keep your db outside the cluster. Use a managed db or just host it on its own server. You should be doing that even if not using k8s anyway. Don’t share memory and cpu with your db. Regarding storage, try to build stateless apps that only need temporary storage that don’t need to outlive a request (like uploads). For long lived unchanging assets, you can try including in the container (ie. defaults.json, countries.json, samples.json, etc). For assets that need to be fresh, try to have the app generate at startup (eg. Email vírus definitions) or via init container. I think this will cover most use cases. This won’t work with apps that depend on shared storage like Wordpress, but Wordpress is horrible and I can’t support you running it :-)

erinaceousjones 3y ago

We went through these loops where I work due to the "you have to use k8s now" thing. Your control plane and worker node can be the same machine. Rancher + Rancher Kubernetes Engine made it a lot easier for us to manage as maintaining that is simply "run this docker container" and it deploys all the k8s gubbins for you. And k3s is a lot more lightweight on resources than k8s. Stuff like certificate rotation and kubernetes version upgrades are handled for us, it's reduced to clicking buttons on UI. Storage can be local, it's just not / wasn't well explained in the documentation. PersistentVolume[Claim]s can use directories on the host, those PVs/PVCs can be shared between many pods and you can mount subdirectories into each pod using `subPath`. You can get it all running on a laptop using a couple hundred MB of memory, all in all. So, not going to be ideal for running on a Raspberry Pi, but on par with a lightweight linux desktop environment in terms of resource usage. I hate the complexity sometimes, but then I still end up having the same learning curves with things like SystemD when I use them anyway - every "run a bunch of stuff on my machine and restart it if it breaks" system seems to be rife with edge cases and behaviours which you have to dig deep into the documentation to see - and we don't _want_ to RTFM if we don't expressly _have_ to :-) ^ Although, okay, SystemD doesn't require you to have random HTTPS authenticated endpoints to talk to it and TLS certificates which expire every year and consists of _one_ init process rather than like 7 interdependent docker containers which can restart at any time and rely on container infrastructure to be running and argh. Okay yeah the flashbacks are happening and I agree with everything you've said again.