8 ms·

I understand their rationale. We manage thousand Kubernetes clusters and end-users can find lots and lots of creative way to shoot themselves in the foot: - I

by fen4o 6y ago

I understand their rationale. We manage thousand Kubernetes clusters and end-users can find lots and lots of creative way to shoot themselves in the foot: - I can store anything in a secret? Let's have thousands of cat images. Etcd then stops working because we have over 2GB of funny cats in the key store. - I can run a root Pod? Lets mount the docker socket and start building images with it. Oh and by the way, I never clean those up and my Node simply fills up. Also I add some additional docker networks that break Pod to Pod networks. - Istio is nice - why we don't add automatic injection for Pods in all namespaces? Including kube-system? And then they brick kube-proxy and the cluster stops working. - I can use validating webhooks for better security? Lets watch on all resources. To keep it more secure lets set the failure policy of the webhook to Fail, so we never admit any modification without the apiserver to make a call to out webhook. Whats that? My single replica webhook has was evicted from the Pod (we didn't add any resource requests and limits) and now it cannot even be created or scheduled because kube-controller-manager and kube-scheduler cannot update their lease and they lost leadership and now are idling, effectively bricking the entire cluster. Google would reduce the pain points with this change, however they would still face countless other issues with Kubernetes.

raverbashing 6y ago
Oh yes secret management with kubectl is needlessly complicated Sure just put your secret data on a file then we'll use your file name as the key of the secret. Cronjobs sometimes have weird bugs as well. A lot of its complexity is due to the fact it's an evolving system, that's fine. But I see that some things end up way more complex or unreliable than it needs due to overengineering or use cases no one needs
el_seano 6y ago
These are some delightfully specific hypotheticals.
endymi0n 6y ago
I fully agree with your points and would sum them up as "Kubernetes has a steep learning curve, a (quite) large interface and ample opportunities to shoot yourself into the foot with it" (plus, they're very funny). However playing the devil's advocate here: If you actually took the steps of learning the basic abstractions, then for me it's really hard to see what you could still get rid of. If you actually go all-in and fit your application to the principles of Kubernetes-native applications (instead of the other way around), then it works nothing short to amazing. We're running 120 microservices in GKE and the difference to our custom-built setup before is night and day. I let my Infra team go surfing together for two weeks because without changes it flies mostly on autopilot. Let's not kid ourselves, distributed computing is _hard_ and Kubernetes is a testament to that. I'm not saying it can't be made more accessible by further standardization, but there are fundamental limits to how easy it can be made. Which by the way is leading to my only pet peeve with it: I feel most of the complexity of K8S comes from the fact that it got hyped as an enterprise product and then lots of features were built that support shoving your non cloud-native workload into Kubernetes even if it was never designed for it. If you don't do or need all of that, the amount of interface, complexity and footguns shrinks significantly. Maybe it's time to better pull them apart in the documentation.
- FlyingSnake 6y ago
  > If you actually took the steps of learning the basic abstractions, then for me it's really hard to see what you could still get rid of. This argument basically sums up to "Developers just need discipline, and stop blaming the tools". While this is a sound argument on paper, the intrinsic complexity of software systems make it hard to pin the blame on developers. BTW This is the same argument Uncle Bob makes which is not so popular with many mainstream developers. You're right about feature creep in k8s though.
  - endymi0n 6y ago
    I get what you're saying, but my point is a bit more nuanced: If your goal is to build highly reliable and available services to end users that are secure and scalable with a team of more than 10 engineers, eventually you will run into more than 50% of the concepts in Kubernetes anyway and end up re-inventing them. Scaling up and down, node draining, finding out whether services are healthy, RBAC, resource distribution, secrets management, service hardening, introspection capabilities, explicit declaration of dependencies and endpoints and many, many more. My point is: Sure, if your goal isn't that, it doesn't make sense to start out using Kubernetes. But if at least eventually that's what you need, imho it's way preferable to just learn and apply well proven abstractions instead of reinventing the wheel along the way and end up with a less maintainable, capable and standardized solution you won't find anyone for maintaining. If I hear about some of the comments here suggesting to "just spinning up docker-compose with Traefik in front" (disclaimer: I really like Traefik), then that reminds me of how some of the ops mess started that I historically had to care for.
    - FlyingSnake 6y ago
      Agreed, the truth usually lies somewhere in between and my point was we can't absolve the tools/ecosystems and put it on squarely on the devs. That definitely doesn't absolve teams and they need to do their homework before jumping on the bandwagon. K8s is great if you know what you're signing up for.
  - pojzon 6y ago
dilyevsky 6y ago
I once misconfigured iptables and locked myself out of our buildserver. Had to call lab support in a different country. Is Linux too complicated? Joyent famously took down their whole region by rebooting wrong nodes. It’s almost like running distributed networks of supercomputers at scale is hard or something...
- KronisLV 6y ago
  > Joyent famously took down their whole region by rebooting wrong nodes. In case anyone's interested, here's a pretty funny and educational talk by Bryan Cantrill about that particular incident: GOTO 2017 • Debugging Under Fire: Keep your Head when Systems have Lost their Mind https://www.youtube.com/watch?v=30jNsCVLpAE https://www.youtube.com/watch?v=30jNsCVLpAE
- regularfry 6y ago
  For two systems of equal functionality, the one that allows or encourages fewer footbullets is the better design. Not all complexity is essential.
  - dilyevsky 6y ago
    What kubernetes complexity is non-essential and can be replicated by simpler solution?
- tormeh 6y ago
  I once did an `apt autoremove` on a custom install of CentOS handed to my team. Apt uninstalled python (and a lot more), and apt depends on python to run, so that was a bummer. The easiest way out was to reinstall the OS.
runeks 6y ago
> I can store anything in a secret? Let's have thousands of cat images. Why would someone want to store non-secret information as a secret?
- young_unixer 6y ago
  I think it's just a cute way of saying "data". Like saying you're seeding Linux ISOs when you're actually seeding pirated movies on BitTorrent.
- endymi0n 6y ago
  "A common mistake that people make when trying to design something completely foolproof is to underestimate the ingenuity of complete fools" — Douglas Adams
- ForHackernews 6y ago
  What makes you think Kubernetes "secrets" are appropriate for storing secret information? They're not secure (not without adding a bunch of other nonsense on top of them).
- nanis 6y ago
  > Why would someone want to store non-secret information as a secret? Top reason given to me by developers: "I don't want to spend time thinking about the distinction."
- JeremyNT 6y ago
  Because mutable persistence in Kubernetes can be super annoying to manage and people might grasp for whatever lifeline they can find. If you have a managed object store or even relational database outside of k8s, the thought of storing arbitrary data in secrets probably doesn't come to mind. But if your enterprise spools up a cluster and tells you to use nfs PVCs with no other storage solution, suddenly you might start getting creative.
antpls 6y ago
There is a benefit : fixing each of these issues fixes them for everyone using K8S. One of the goal of K8S is normalization/standardization of a complex topic to better share knowledge
iooi 6y ago
> - I can run a root Pod? Lets mount the docker socket and start building images with it. Just in case you haven't figured out the proper way to do this, you should use docker:dind.
- edude03 6y ago
  kaniko is an even better way
  - sschueller 6y ago
    If it didn't have wierd issues when you start stacking in docker file.
    - edude03 6y ago
      like what? I'd like to hear your experiences with it

by fen4o 6y ago

raverbashing 6y ago

Oh yes secret management with kubectl is needlessly complicated Sure just put your secret data on a file then we'll use your file name as the key of the secret. Cronjobs sometimes have weird bugs as well. A lot of its complexity is due to the fact it's an evolving system, that's fine. But I see that some things end up way more complex or unreliable than it needs due to overengineering or use cases no one needs

el_seano 6y ago

These are some delightfully specific hypotheticals.

endymi0n 6y ago

I fully agree with your points and would sum them up as "Kubernetes has a steep learning curve, a (quite) large interface and ample opportunities to shoot yourself into the foot with it" (plus, they're very funny). However playing the devil's advocate here: If you actually took the steps of learning the basic abstractions, then for me it's really hard to see what you could still get rid of. If you actually go all-in and fit your application to the principles of Kubernetes-native applications (instead of the other way around), then it works nothing short to amazing. We're running 120 microservices in GKE and the difference to our custom-built setup before is night and day. I let my Infra team go surfing together for two weeks because without changes it flies mostly on autopilot. Let's not kid ourselves, distributed computing is _hard_ and Kubernetes is a testament to that. I'm not saying it can't be made more accessible by further standardization, but there are fundamental limits to how easy it can be made. Which by the way is leading to my only pet peeve with it: I feel most of the complexity of K8S comes from the fact that it got hyped as an enterprise product and then lots of features were built that support shoving your non cloud-native workload into Kubernetes even if it was never designed for it. If you don't do or need all of that, the amount of interface, complexity and footguns shrinks significantly. Maybe it's time to better pull them apart in the documentation.

FlyingSnake 6y ago
> If you actually took the steps of learning the basic abstractions, then for me it's really hard to see what you could still get rid of. This argument basically sums up to "Developers just need discipline, and stop blaming the tools". While this is a sound argument on paper, the intrinsic complexity of software systems make it hard to pin the blame on developers. BTW This is the same argument Uncle Bob makes which is not so popular with many mainstream developers. You're right about feature creep in k8s though.
- endymi0n 6y ago
  I get what you're saying, but my point is a bit more nuanced: If your goal is to build highly reliable and available services to end users that are secure and scalable with a team of more than 10 engineers, eventually you will run into more than 50% of the concepts in Kubernetes anyway and end up re-inventing them. Scaling up and down, node draining, finding out whether services are healthy, RBAC, resource distribution, secrets management, service hardening, introspection capabilities, explicit declaration of dependencies and endpoints and many, many more. My point is: Sure, if your goal isn't that, it doesn't make sense to start out using Kubernetes. But if at least eventually that's what you need, imho it's way preferable to just learn and apply well proven abstractions instead of reinventing the wheel along the way and end up with a less maintainable, capable and standardized solution you won't find anyone for maintaining. If I hear about some of the comments here suggesting to "just spinning up docker-compose with Traefik in front" (disclaimer: I really like Traefik), then that reminds me of how some of the ops mess started that I historically had to care for.
  - FlyingSnake 6y ago
    Agreed, the truth usually lies somewhere in between and my point was we can't absolve the tools/ecosystems and put it on squarely on the devs. That definitely doesn't absolve teams and they need to do their homework before jumping on the bandwagon. K8s is great if you know what you're signing up for.
- pojzon 6y ago

dilyevsky 6y ago

I once misconfigured iptables and locked myself out of our buildserver. Had to call lab support in a different country. Is Linux too complicated? Joyent famously took down their whole region by rebooting wrong nodes. It’s almost like running distributed networks of supercomputers at scale is hard or something...

KronisLV 6y ago
> Joyent famously took down their whole region by rebooting wrong nodes. In case anyone's interested, here's a pretty funny and educational talk by Bryan Cantrill about that particular incident: GOTO 2017 • Debugging Under Fire: Keep your Head when Systems have Lost their Mind https://www.youtube.com/watch?v=30jNsCVLpAE https://www.youtube.com/watch?v=30jNsCVLpAE
regularfry 6y ago
For two systems of equal functionality, the one that allows or encourages fewer footbullets is the better design. Not all complexity is essential.
- dilyevsky 6y ago
  What kubernetes complexity is non-essential and can be replicated by simpler solution?
tormeh 6y ago
I once did an `apt autoremove` on a custom install of CentOS handed to my team. Apt uninstalled python (and a lot more), and apt depends on python to run, so that was a bummer. The easiest way out was to reinstall the OS.

runeks 6y ago

> I can store anything in a secret? Let's have thousands of cat images. Why would someone want to store non-secret information as a secret?

young_unixer 6y ago
I think it's just a cute way of saying "data". Like saying you're seeding Linux ISOs when you're actually seeding pirated movies on BitTorrent.
endymi0n 6y ago
"A common mistake that people make when trying to design something completely foolproof is to underestimate the ingenuity of complete fools" — Douglas Adams
ForHackernews 6y ago
What makes you think Kubernetes "secrets" are appropriate for storing secret information? They're not secure (not without adding a bunch of other nonsense on top of them).
nanis 6y ago
> Why would someone want to store non-secret information as a secret? Top reason given to me by developers: "I don't want to spend time thinking about the distinction."
JeremyNT 6y ago
Because mutable persistence in Kubernetes can be super annoying to manage and people might grasp for whatever lifeline they can find. If you have a managed object store or even relational database outside of k8s, the thought of storing arbitrary data in secrets probably doesn't come to mind. But if your enterprise spools up a cluster and tells you to use nfs PVCs with no other storage solution, suddenly you might start getting creative.

antpls 6y ago

There is a benefit : fixing each of these issues fixes them for everyone using K8S. One of the goal of K8S is normalization/standardization of a complex topic to better share knowledge

iooi 6y ago

> - I can run a root Pod? Lets mount the docker socket and start building images with it. Just in case you haven't figured out the proper way to do this, you should use docker:dind.

edude03 6y ago
kaniko is an even better way
- sschueller 6y ago
  If it didn't have wierd issues when you start stacking in docker file.
  - edude03 6y ago
    like what? I'd like to hear your experiences with it