4 ms·

> Can small companies afford the complexities of k8s? Where exactly do you see this complexity in Kubernetes? I have a couple of Hetzner nodes running microk8

> Can small companies afford the complexities of k8s? Where exactly do you see this complexity in Kubernetes? I have a couple of Hetzner nodes running microk8s and I have a couple of web apps running in them. All it takes to deploy each app is putting together the kustomize script for the app and afterwards simple call to kubectl apply -k ${kustomize_dir}. I'm talking about specifying an ingress, deployments, services,... The basics. I even threw in a couple of secrets to pull Docker images from private container registries. And everything just runs. With blue-green deployments, deployment history, monitoring, etc. It's far more complicated to setup a CICD pipeline, and don't get me started on the god-awful mess that is CloudFormation or even CDK. Where exactly did you see all that complexity you're talking about?

ozim 4y ago
You apparently learned all about microk8s and spent time configuring it. I assume you did not get it right at the first time. It is like Usain Bolt comming over and asking what is so hard for you in running 100m in ~10 seconds when you never left the couch.
- tene 4y ago
  It's just like any other tech; you read the docs, try it out, do some troubleshooting, and then you know how to use the tool. I bet you could get microk8s running correctly on your first try. Give it a shot! Here's a doc: https://microk8s.io/docs/getting-started https://microk8s.io/docs/getting-started You could probably get some additional nodes in your cluster on your first try too: https://microk8s.io/docs/clustering https://microk8s.io/docs/clustering This isn't Usain Bolt. This is normal people doing normal work with normal technical tools, and then somehow people keep claiming they must be some kind of world-class genius to have done it. Try it for yourself before you claim that you'd have to be a world-class peak performer to have installed and configured some simple daemons on a few linux servers.
  - ozim 4y ago
    Of course Usain Bolt is an exaggeration. But you try it out, set stuff up from tutorial then do some troubleshooting put production data there and you get articles like these: https://medium.com/handy-tech/analysis-of-a-kubernetes-hack-backdooring-through-kubelet-823be5c3d67c https://medium.com/handy-tech/analysis-of-a-kubernetes-hack-... https://www.zdnet.com/article/a-hacker-has-wiped-defaced-more-than-15000-elasticsearch-servers/ https://www.zdnet.com/article/a-hacker-has-wiped-defaced-mor... https://thenewstack.io/armo-misconfiguration-is-number-1-kubernetes-security-risk/ https://thenewstack.io/armo-misconfiguration-is-number-1-kub...
    - tene 4y ago
      I'll try to take a different approach here than I did in my other recent reply to you. I hear that you're saying that there are possible configurations that are insecure, and that at least some care and attention needs to be invested to avoid problems. I agree. This isn't specific to Kubernetes, as you show in your second link. This can be a problem, and people have in fact suffered harm due to leaving their doors unlocked. On the other hand, most important security doors in most professional environments are not left unlocked and unmonitored. If you have already learned basic sysadmin fundamentals, then you have the skills needed to learn to deploy and use Kubernetes just as securely as any other network service. The way that you learn to apply your general sysadmin skills to Kubernetes is by practicing with it. You can supplement your practice with books and training if you really want to, but it's not necessary. If you happen to have other people who have already gone through this process as support, that can help quite a bit. If there are any other better ways that people learn things like this, I have yet to hear of them. If you don't already have those skills, then the way to build and develop them is exactly the same process. You try stuff out, read some docs, poke at things to see if you can break them. You can supplement this with classes if you like, but they're not necessary. Peers and mentors are great if you have them, but they're not necessary. What alternative do you have in mind? What about kubernetes specifically is so monstrously complex? People keep asserting this, but I learned it just fine like any other nontrivial software I've ever worked with professionally. My peers at work learned it just fine like any other software we've worked with. My friends and colleagues I keep in touch with from previous jobs have learned it fine. I don't really understand what kind of complexity bar you're trying to imply is just objectively too high to be reasonable? Yeah, it's got more moving parts than like Redis, because it does way more than Redis does. Sed is simpler than python, but that doesn't mean that you need to be Usain Bolt to learn python.
- arinlen 4y ago
  > You apparently learned all about microk8s and spent time configuring it. I assume you did not get it right at the first time. What? With Ubuntu, microk8s works pretty much right out of the box. The only thing you need to learn is how to install it with Snap. What are you talking about? > I assume you did not get it right at the first time. I did, not because I'm a rocket surgeon but because it is really really that simple. https://ubuntu.com/tutorials/install-a-local-kubernetes-with-microk8s https://ubuntu.com/tutorials/install-a-local-kubernetes-with... What exactly leads people like you to complain harshly about how hard it is something you never even tried before? You're literally wasting far more time complaining in a random online forum about how hard a technology is than what it takes to not only learn the basics but also get it up and running.
  - ozim 4y ago
    You also had a pentest of your setup so you are perfectly sure you don't expose something to the internet that you are not supposed to? You also considered updates for Ubuntu and microk8s so you have strategy for updating your nodes with newer versions and security patches. I can follow a tutorial to set something up - but then there is always whole world of things that is never included in tutorials. Just like kubelet accepting unauthenticated requests by default: https://medium.com/handy-tech/analysis-of-a-kubernetes-hack-backdooring-through-kubelet-823be5c3d67c https://medium.com/handy-tech/analysis-of-a-kubernetes-hack-...
    - tene 4y ago
      If you're concerned that you've somehow accidentally exposed something to the internet that you didn't explicitly intend to expose to the internet, you can just do a trivial port scan. You just run nmap, look at the output, and you're done in like 30 seconds. What does this have to do with kubernetes? "Don't expose stuff to the internet that you don't intend for everyone across the planet to be able to access" applies exactly the same to literally everything you could run on your servers. This isn't remotely "Kubernetes is uniquely scary and complicated"; this is basic fundamental network security, and if you're not already handling this, then you need to go brush up on your basic networking fundamentals, not blame it somehow on kubernetes. Almost every network service I can think of defaults to accepting unauthenticated connections, or connections authenticated with some default credentials. This is the normal, expected, default situation with network services. If you make the decision to expose something to the entire world, in a professional context, it is your responsibility to know the specific reasons it is safe to do so. Are you really trying to argue that "Some rando decided to bareback the entire global internet with no firewall, on a personal home server, and didn't bother to type 'kubernetes secure configuration' into google, therefore Kubernetes is super hard and complicated and dangerous"? It's not like this is some obscure cryptic detail; it's explicitly called out in the documentation that any half-decent professional would read before deploying a production service: https://kubernetes.io/docs/tasks/administer-cluster/securing-a-cluster/ https://kubernetes.io/docs/tasks/administer-cluster/securing... Controlling access to the Kubelet Kubelets expose HTTPS endpoints which grant powerful control over the node and containers. By default Kubelets allow unauthenticated access to this API. Production clusters should enable Kubelet authentication and authorization. Consult the Kubelet authentication/authorization reference for more information. Yes, untrained amateurs sometimes do dumb stuff. Sometimes companies leave their S3 buckets open to the world. Sometimes people expose mysql to the internet with credentials they ship to users. Sometimes people expose unauthenticated Redis to the internet. This does not mean that these technologies are somehow fundamentally too complicated for mere mortals, it just means that it's dangerous to ask amateurs to do something in a professional context.
tene 4y ago
A lot of people in this job just really hate learning anything, and would rather spend way more time spread out over months and years than just investing some time and learning how to use something new. It seems like somehow some people get into this job by only learning tools that they can pick up without trouble over a weekend? The concepts you need to deploy stuff on kubernetes really aren't that complicated. It's just a bunch of yaml documents in extremely-well-documented schemas. If you want to run a service with N instances, you just write a deployment with `replicas: N`. There are a lot of details I'd choose slightly differently if I were designing my perfect ideal cluster orchestration system, but the whole point of open source is that everyone who wants to build a comprehensive cluster orchestration system can just get together and collectively build it once, so I don't have to design and own it all internally. It's got all the pieces to build exactly what I need to make good use of a ton of computers, in a simple, reliable, repeatable, consistent, standard way. It gives you trivial primitives to build HA fault-tolerant deployments. There are very few good excuses left to ever have any reason to page anyone over "One server had a hardware failure". It just baffles me that people can see this powerful, industrial-grade, comprehensive tool, and decide "Nah, that'll never be worth starting to learn".
mmcnl 4y ago
I half agree with you here. It is actually quite simple, but everything in Kubernetes is very explicit, which is good, but also intimidating. If you've never worked with Kubernetes before then it's a lot of added complexity without clear benefits.
- arinlen 4y ago
  > If you've never worked with Kubernetes before then it's a lot of added complexity without clear benefits. Unless you're someone who only had to work on a monolith deployed to a single box somewhere, Kubernetes adds zero complexity to the problem you're already dealing with. In fact, Kubernetes simplifies the whole problem of running stuff on a cluster. Network, security, deployment, observanility... That's all provided out of the box. And you can rollback whole deployments with a single command. Heck, even ssh-ing into a container, regardless of where it's running, became trivial. How is that harder than deploying stuff to boxes somewhere?
  - treis 4y ago
    >Unless you're someone who only had to work on a monolith deployed to a single box somewhere, I think the point is that 90%+ of websites are fine with a few monoliths behind a load balancer. That set up can handle low thousands of requests per second in Rails/Django/etc. Maybe low 10 thousands with a more performant language. And it's not just k8s. It's the whole microservice/SOA that comes with it. It ramps up the complexity of everything and is a constant time sink in my experience.

ozim 4y ago

You apparently learned all about microk8s and spent time configuring it. I assume you did not get it right at the first time. It is like Usain Bolt comming over and asking what is so hard for you in running 100m in ~10 seconds when you never left the couch.

tene 4y ago
It's just like any other tech; you read the docs, try it out, do some troubleshooting, and then you know how to use the tool. I bet you could get microk8s running correctly on your first try. Give it a shot! Here's a doc: https://microk8s.io/docs/getting-started https://microk8s.io/docs/getting-started You could probably get some additional nodes in your cluster on your first try too: https://microk8s.io/docs/clustering https://microk8s.io/docs/clustering This isn't Usain Bolt. This is normal people doing normal work with normal technical tools, and then somehow people keep claiming they must be some kind of world-class genius to have done it. Try it for yourself before you claim that you'd have to be a world-class peak performer to have installed and configured some simple daemons on a few linux servers.
- ozim 4y ago
  Of course Usain Bolt is an exaggeration. But you try it out, set stuff up from tutorial then do some troubleshooting put production data there and you get articles like these: https://medium.com/handy-tech/analysis-of-a-kubernetes-hack-backdooring-through-kubelet-823be5c3d67c https://medium.com/handy-tech/analysis-of-a-kubernetes-hack-... https://www.zdnet.com/article/a-hacker-has-wiped-defaced-more-than-15000-elasticsearch-servers/ https://www.zdnet.com/article/a-hacker-has-wiped-defaced-mor... https://thenewstack.io/armo-misconfiguration-is-number-1-kubernetes-security-risk/ https://thenewstack.io/armo-misconfiguration-is-number-1-kub...
  - tene 4y ago
    I'll try to take a different approach here than I did in my other recent reply to you. I hear that you're saying that there are possible configurations that are insecure, and that at least some care and attention needs to be invested to avoid problems. I agree. This isn't specific to Kubernetes, as you show in your second link. This can be a problem, and people have in fact suffered harm due to leaving their doors unlocked. On the other hand, most important security doors in most professional environments are not left unlocked and unmonitored. If you have already learned basic sysadmin fundamentals, then you have the skills needed to learn to deploy and use Kubernetes just as securely as any other network service. The way that you learn to apply your general sysadmin skills to Kubernetes is by practicing with it. You can supplement your practice with books and training if you really want to, but it's not necessary. If you happen to have other people who have already gone through this process as support, that can help quite a bit. If there are any other better ways that people learn things like this, I have yet to hear of them. If you don't already have those skills, then the way to build and develop them is exactly the same process. You try stuff out, read some docs, poke at things to see if you can break them. You can supplement this with classes if you like, but they're not necessary. Peers and mentors are great if you have them, but they're not necessary. What alternative do you have in mind? What about kubernetes specifically is so monstrously complex? People keep asserting this, but I learned it just fine like any other nontrivial software I've ever worked with professionally. My peers at work learned it just fine like any other software we've worked with. My friends and colleagues I keep in touch with from previous jobs have learned it fine. I don't really understand what kind of complexity bar you're trying to imply is just objectively too high to be reasonable? Yeah, it's got more moving parts than like Redis, because it does way more than Redis does. Sed is simpler than python, but that doesn't mean that you need to be Usain Bolt to learn python.
arinlen 4y ago
> You apparently learned all about microk8s and spent time configuring it. I assume you did not get it right at the first time. What? With Ubuntu, microk8s works pretty much right out of the box. The only thing you need to learn is how to install it with Snap. What are you talking about? > I assume you did not get it right at the first time. I did, not because I'm a rocket surgeon but because it is really really that simple. https://ubuntu.com/tutorials/install-a-local-kubernetes-with-microk8s https://ubuntu.com/tutorials/install-a-local-kubernetes-with... What exactly leads people like you to complain harshly about how hard it is something you never even tried before? You're literally wasting far more time complaining in a random online forum about how hard a technology is than what it takes to not only learn the basics but also get it up and running.
- ozim 4y ago
  You also had a pentest of your setup so you are perfectly sure you don't expose something to the internet that you are not supposed to? You also considered updates for Ubuntu and microk8s so you have strategy for updating your nodes with newer versions and security patches. I can follow a tutorial to set something up - but then there is always whole world of things that is never included in tutorials. Just like kubelet accepting unauthenticated requests by default: https://medium.com/handy-tech/analysis-of-a-kubernetes-hack-backdooring-through-kubelet-823be5c3d67c https://medium.com/handy-tech/analysis-of-a-kubernetes-hack-...
  - tene 4y ago
    If you're concerned that you've somehow accidentally exposed something to the internet that you didn't explicitly intend to expose to the internet, you can just do a trivial port scan. You just run nmap, look at the output, and you're done in like 30 seconds. What does this have to do with kubernetes? "Don't expose stuff to the internet that you don't intend for everyone across the planet to be able to access" applies exactly the same to literally everything you could run on your servers. This isn't remotely "Kubernetes is uniquely scary and complicated"; this is basic fundamental network security, and if you're not already handling this, then you need to go brush up on your basic networking fundamentals, not blame it somehow on kubernetes. Almost every network service I can think of defaults to accepting unauthenticated connections, or connections authenticated with some default credentials. This is the normal, expected, default situation with network services. If you make the decision to expose something to the entire world, in a professional context, it is your responsibility to know the specific reasons it is safe to do so. Are you really trying to argue that "Some rando decided to bareback the entire global internet with no firewall, on a personal home server, and didn't bother to type 'kubernetes secure configuration' into google, therefore Kubernetes is super hard and complicated and dangerous"? It's not like this is some obscure cryptic detail; it's explicitly called out in the documentation that any half-decent professional would read before deploying a production service: https://kubernetes.io/docs/tasks/administer-cluster/securing-a-cluster/ https://kubernetes.io/docs/tasks/administer-cluster/securing... Controlling access to the Kubelet Kubelets expose HTTPS endpoints which grant powerful control over the node and containers. By default Kubelets allow unauthenticated access to this API. Production clusters should enable Kubelet authentication and authorization. Consult the Kubelet authentication/authorization reference for more information. Yes, untrained amateurs sometimes do dumb stuff. Sometimes companies leave their S3 buckets open to the world. Sometimes people expose mysql to the internet with credentials they ship to users. Sometimes people expose unauthenticated Redis to the internet. This does not mean that these technologies are somehow fundamentally too complicated for mere mortals, it just means that it's dangerous to ask amateurs to do something in a professional context.

tene 4y ago

A lot of people in this job just really hate learning anything, and would rather spend way more time spread out over months and years than just investing some time and learning how to use something new. It seems like somehow some people get into this job by only learning tools that they can pick up without trouble over a weekend? The concepts you need to deploy stuff on kubernetes really aren't that complicated. It's just a bunch of yaml documents in extremely-well-documented schemas. If you want to run a service with N instances, you just write a deployment with `replicas: N`. There are a lot of details I'd choose slightly differently if I were designing my perfect ideal cluster orchestration system, but the whole point of open source is that everyone who wants to build a comprehensive cluster orchestration system can just get together and collectively build it once, so I don't have to design and own it all internally. It's got all the pieces to build exactly what I need to make good use of a ton of computers, in a simple, reliable, repeatable, consistent, standard way. It gives you trivial primitives to build HA fault-tolerant deployments. There are very few good excuses left to ever have any reason to page anyone over "One server had a hardware failure". It just baffles me that people can see this powerful, industrial-grade, comprehensive tool, and decide "Nah, that'll never be worth starting to learn".

mmcnl 4y ago

I half agree with you here. It is actually quite simple, but everything in Kubernetes is very explicit, which is good, but also intimidating. If you've never worked with Kubernetes before then it's a lot of added complexity without clear benefits.

arinlen 4y ago
> If you've never worked with Kubernetes before then it's a lot of added complexity without clear benefits. Unless you're someone who only had to work on a monolith deployed to a single box somewhere, Kubernetes adds zero complexity to the problem you're already dealing with. In fact, Kubernetes simplifies the whole problem of running stuff on a cluster. Network, security, deployment, observanility... That's all provided out of the box. And you can rollback whole deployments with a single command. Heck, even ssh-ing into a container, regardless of where it's running, became trivial. How is that harder than deploying stuff to boxes somewhere?
- treis 4y ago
  >Unless you're someone who only had to work on a monolith deployed to a single box somewhere, I think the point is that 90%+ of websites are fine with a few monoliths behind a load balancer. That set up can handle low thousands of requests per second in Rails/Django/etc. Maybe low 10 thousands with a more performant language. And it's not just k8s. It's the whole microservice/SOA that comes with it. It ramps up the complexity of everything and is a constant time sink in my experience.