3 ms·

I find the basic Linux stuff is learning by doing and reading docs, using search and asking around. Docker has docs and also searchable examples and blogs. Bu

I find the basic Linux stuff is learning by doing and reading docs, using search and asking around. Docker has docs and also searchable examples and blogs. But k8s is an IMHO artificially fenced off area. No matter where I ask, I receive no response or am met with arrogance and elitism. I can't even get the simplest thing answered, which is "what is the minimum required setup for k8s, if not using a hosted solution? I plan to use 3cp 3w nodes and utilize all resources available. Do I NEED external storage, do I NEED an external load balancer or can I use DNS-LB?" Official k8s forum - zero response. Reddit k8s - zero response and downvotes Home operations Discord - zero response It's like this little well kept secret only select few have access to. The documentation is also not clear on that subject, because I believe the big companies like Google and Amazon want to sell you their k8s offerings, which are super expensive, over 2k per month for a 5 node cluster are you kidding me

d3Xt3r 2y ago
That's what I've been seeing in the industry as well, and it's honestly pretty disgusting. We were evaluating this one particular I*M product for infrastructure management, which needed to be installed on top of this other I*M platform, which specifically required to be run on top of a particular version of RHEL and a particular version of OpenShift (no other k8s flavor would do), and they wanted it as dedicated instance on a dedicated physical server (so we couldn't use our existing OCP instance). The overal system requirements to run the damn thing (which was ultimately just a Ruby app) was enormous, and the bill quoted was astronomical. Customising, upgrading and maintaining it would've been a massive PITA too. Thankfully our C-suits had a rare lightbulb moment and the whole project was canned. But man, what a massive waste of time and effort it was, we spent hundreds of hours doing the discovery, design and coding for it. Heck, I even ended up having to learn Ruby just so we could customise the app, because I*M were incapable of delivering a decent product. We then decided to go for another white labeled product from a competing vendor - which was admittedly a lot better than that I*M crap, but during our consultation with them it was decided that we'd need to port all our existing code and tooling to their proprietary platform - a lengthy process that would take several hundreds of hours of effort, with a migration plan set over the course of an year. Personally, none of this makes any sense to me. I can understand why we wanted to go for a commercial solution, but with the amount of time, money, effort we'd be putting in, we could've just continued to develop our in-house product which leveraged existing opensource solutions like Ansible. The whole insurance for "getting hit by a bus" scenario and "vendor support" is so blown out of proportion these days. I just want to go back to the good ol' days where being a sysadmin meant you were the one in control.
moondev 2y ago
I can't even get the simplest thing answered, which is "what is the minimum required setup for k8s, if not using a hosted solution? Most mininal is to run a cluster inside a docker container with KIND. 2 cores and 4G of memory should allow you to run some small workloads I plan to use 3cp 3w nodes and utilize all resources available. Do I NEED external storage, No. Storage is optional AKA stateless workloads. do I NEED an external load balancer or can I use DNS-LB?" I'm assuming you want to bootstrap an upstream k8s cluster with kubeadm. In that case yes you need load balancer to sit in front of your control plane nodes. You can use a project like kube-vip to function as the load balancer without introducing an extra machine. If you want to use an extra machine you use something like haproxy. You can use DNS but it's not ideal for production without healthcheck - but for a lab go for it
- lakomen 2y ago
  Thanks, yes rent 3 small (well 4 dedicated cores, 8GB RAM, 256GB storage) nodes for CP and double the size for workers, which should be enough for a test cluster, well test and actual real life prod cluster, external of course. Costs around 80€/m. My current server costs are about the same price. I noticed when the master or 1st cp node goes down, the other 2 are not sufficient to keep the cluster running. I wonder why do I then even have (and am paying for) 3 CP nodes when failover isn't working. I had the most success with microk8s. I tried many other solutions, and I'm not going down the manual kubeadm path. I can't dedicate all my time into maintaining that cluster. I want to focus on developing my services, writing a configuration file, or rather with k8s that's a collection of files... Everything screams "don't do it". "It's too complicated, too time intensive, don't do it, you will regret it" I'm not even sure k8s will be able to recover the loss of 1 worker node. All my experiments with a 3 worker postgres cluster showed that if the node count is less than 3, it goes into an endless loop of trying to bring the instances up. I mean the most important thing is, how good is the solution when disaster happens? K8s tries to reallocate the resources on other nodes. But if there are less nodes than required by the helmet chart, that fails. So that means in conclusion, I need a fallback node, aka 4 worker nodes, and the cluster now costs about 97€/m. But that also means that I can't say with certainty which IPs are being used, if I scale the service up. I'm trying to be cost effective, my resources are limited. And I'd like to learn about this, but a course costs over 10k€ and takes 6 months. I also noticed, when I have 3 CP nodes, only the 1st node has high CPU and RAM usage, the other 2 are pretty much dead, except for the 10% resource usage of etcd, no matter if they're doing anything or not. Stateless workloads are a myth. Or rather you can't provide a service completely without state. The units can of course be stateless, but they will access stateful containers, like postgres, which also run on k8s. I don't think I need k8s for now and when I will, other solutions are more likely. The whole effort of maintaining and upgrading the cluster, and right now there's a transition to gateways instead of ingress controllers, so you need to bother with infra instead of solving actual problems. Thanks for feedback anyhow.

d3Xt3r 2y ago

That's what I've been seeing in the industry as well, and it's honestly pretty disgusting. We were evaluating this one particular I*M product for infrastructure management, which needed to be installed on top of this other I*M platform, which specifically required to be run on top of a particular version of RHEL and a particular version of OpenShift (no other k8s flavor would do), and they wanted it as dedicated instance on a dedicated physical server (so we couldn't use our existing OCP instance). The overal system requirements to run the damn thing (which was ultimately just a Ruby app) was enormous, and the bill quoted was astronomical. Customising, upgrading and maintaining it would've been a massive PITA too. Thankfully our C-suits had a rare lightbulb moment and the whole project was canned. But man, what a massive waste of time and effort it was, we spent hundreds of hours doing the discovery, design and coding for it. Heck, I even ended up having to learn Ruby just so we could customise the app, because I*M were incapable of delivering a decent product. We then decided to go for another white labeled product from a competing vendor - which was admittedly a lot better than that I*M crap, but during our consultation with them it was decided that we'd need to port all our existing code and tooling to their proprietary platform - a lengthy process that would take several hundreds of hours of effort, with a migration plan set over the course of an year. Personally, none of this makes any sense to me. I can understand why we wanted to go for a commercial solution, but with the amount of time, money, effort we'd be putting in, we could've just continued to develop our in-house product which leveraged existing opensource solutions like Ansible. The whole insurance for "getting hit by a bus" scenario and "vendor support" is so blown out of proportion these days. I just want to go back to the good ol' days where being a sysadmin meant you were the one in control.

moondev 2y ago

I can't even get the simplest thing answered, which is "what is the minimum required setup for k8s, if not using a hosted solution? Most mininal is to run a cluster inside a docker container with KIND. 2 cores and 4G of memory should allow you to run some small workloads I plan to use 3cp 3w nodes and utilize all resources available. Do I NEED external storage, No. Storage is optional AKA stateless workloads. do I NEED an external load balancer or can I use DNS-LB?" I'm assuming you want to bootstrap an upstream k8s cluster with kubeadm. In that case yes you need load balancer to sit in front of your control plane nodes. You can use a project like kube-vip to function as the load balancer without introducing an extra machine. If you want to use an extra machine you use something like haproxy. You can use DNS but it's not ideal for production without healthcheck - but for a lab go for it

lakomen 2y ago
Thanks, yes rent 3 small (well 4 dedicated cores, 8GB RAM, 256GB storage) nodes for CP and double the size for workers, which should be enough for a test cluster, well test and actual real life prod cluster, external of course. Costs around 80€/m. My current server costs are about the same price. I noticed when the master or 1st cp node goes down, the other 2 are not sufficient to keep the cluster running. I wonder why do I then even have (and am paying for) 3 CP nodes when failover isn't working. I had the most success with microk8s. I tried many other solutions, and I'm not going down the manual kubeadm path. I can't dedicate all my time into maintaining that cluster. I want to focus on developing my services, writing a configuration file, or rather with k8s that's a collection of files... Everything screams "don't do it". "It's too complicated, too time intensive, don't do it, you will regret it" I'm not even sure k8s will be able to recover the loss of 1 worker node. All my experiments with a 3 worker postgres cluster showed that if the node count is less than 3, it goes into an endless loop of trying to bring the instances up. I mean the most important thing is, how good is the solution when disaster happens? K8s tries to reallocate the resources on other nodes. But if there are less nodes than required by the helmet chart, that fails. So that means in conclusion, I need a fallback node, aka 4 worker nodes, and the cluster now costs about 97€/m. But that also means that I can't say with certainty which IPs are being used, if I scale the service up. I'm trying to be cost effective, my resources are limited. And I'd like to learn about this, but a course costs over 10k€ and takes 6 months. I also noticed, when I have 3 CP nodes, only the 1st node has high CPU and RAM usage, the other 2 are pretty much dead, except for the 10% resource usage of etcd, no matter if they're doing anything or not. Stateless workloads are a myth. Or rather you can't provide a service completely without state. The units can of course be stateless, but they will access stateful containers, like postgres, which also run on k8s. I don't think I need k8s for now and when I will, other solutions are more likely. The whole effort of maintaining and upgrading the cluster, and right now there's a transition to gateways instead of ingress controllers, so you need to bother with infra instead of solving actual problems. Thanks for feedback anyhow.