4 ms·

I keep seeing this opinion and I don't understand it. For various reasons, I recently transitioned from a dev role to running a 60+ node, 14+ PB bare metal clus

by remram 2y ago

I keep seeing this opinion and I don't understand it. For various reasons, I recently transitioned from a dev role to running a 60+ node, 14+ PB bare metal cluster. 3 years in, and the only thing ever giving me trouble is Ceph. Kubernetes is etcd, apiserver, and controllers. That's exactly as many components as your average MVC app. The control-loop thing is interesting, and there are a few "kinds" of resources to get used to, but why is it always presented as this insurmountable complexity? I ran into a VXLAN checksum offload kernel bug once, but otherwise this thing is just solid. Sure it's a lot of YAML but I don't understand the rep.

jimmaswell 2y ago
Yeah, I'm sure there are tricky details as in anything but the core idea doesn't sound that complicated to me. I've been looking into it a bit after seeing this fun video a while ago where a DOS BBS is ran on kubernetes. https://youtu.be/wLVHXn79l8M?si=U2FexAMKd3zQVA82 https://youtu.be/wLVHXn79l8M?si=U2FexAMKd3zQVA82
lolinder 2y ago
Because the root comment is mostly but not quite right: there are indeed a large subset of developers that aren't interested in thinking about infrastructure, but there are many subcategories of those people, and many of them aren't fly.io customers. A large number of people who are in that category aren't happy to let someone else handle their infra. They're not interested in infra in the sense that they don't believe it should be more complicated than "start process on Linux box and set up firewall and log rotation". For some applications these people are absolutely right, but they've persuaded themselves that that means it's the best way to handle all use cases, which makes them see Kubernetes as way more complex than is necessary, rather than as a roll-your-own ECS for those who would otherwise truly need a cloud provider.
- worldsayshi 2y ago
  Feels like swe engineers are talking past each other a lot about these topics. I assume everyone wants to be in control of their environment. But with so many ways to compose your infra that means a lot of different things for different people.
docandrew 2y ago
“etcd, apiserver, and controllers.” …and containerd and csi plugins and kubelet and cni plugins and kubectl and kube-proxy and ingresses and load balancers…
- igmor 2y ago
  These components are very different in complexity and scope. Let's be real: a seasoned developer is mostly familiar with load balancers and ingress controllers, so this will be mostly about naming and context. I agree though once you learn about k8s it becomes less mysterious but that also means the author hasn't pushed it to the limits. Outages in the control plane could be pretty nasty and it is easy to have them by creating an illusion everything is kind of free in k8s.
  - nicoburns 2y ago
    A really simple setup for many smaller organisations wouldn't have a load balancer at all.
    - lakomen 2y ago
      [dead]
    - darkwater 2y ago
      No load balancer means... entering one node only? Doing DNS RR over all the nodes? If you don't have a load balancer in front, why are you even using Kubernetes? Deploy a single VM and call it a day! I mean, in my homelab I do have Kubernetes and no LB in front, but it's a homelab for fun and learn K8s internals. But in a professional environment...
    - dilyevsky 2y ago
      No code at all even - just use excel
- zeroq 2y ago
  typical how to program an owl: step one: draw a circle step two: import the rest of the owl
figassis 2y ago
I use k8s, wouldn't call it simple, but there are ways to minimize the complexity of your setup. Mostly, what devs see as complexity is k8s packages a lot of system fundamentals, like networking, storage, name resolution, distributed architectures, etc, and if you mainly spent your career in a single lane, k8s becomes impossible to grasp. Not saying those devs are wrong, not everyone needs to be a networking pro. K8s is meant to be operated by some class of engineers, and used by another. Just like you have DBAs, sysadmins, etc, maybe your devops should have more system experience besides terraform.
spratzt 2y ago
Genuinely curious about what sort of business stores and processes 14 PB on a 60 node cluster.
- remram 2y ago
  Research institution. The department saw more need for storage than Kubernetes compute so that's what we're growing. Nowadays you can get storage machines with 1 PB in them.
- superq 2y ago
  Yeah, that's an interesting question, because it sounds like a ton of data vs not enough compute, but, aside from this all being in a SAN or large storage array: The larger Supermicro or Quanta storage servers can easily handle 36 HDD's each, or even more. So with just 16 of those with 36x24TB disks, that meets the ~14PB capacity mark, leaving 44 remaining nodes for other compute task, load balancing, NVME clusters, etc.
  - remram 2y ago
    We have boxes with up to 45 drives yes.
cullenking 2y ago
Hey fellow k8s+ceph on bare metaler! We only have a 13 machine rack and 350tb of raw storage. No major issues with ceph after 16.x and all nvme storage though.
chucky_z 2y ago
I consider a '60+ node' kubernetes cluster is very small. Kubernetes at that scale is genuinely excellent! At 6000, 60000, and 600000 nodes it becomes very different and goes from 'Hey, this is pretty great' to 'What have I done?' The maintenance costs of running more than a hundred clusters is incredibly nontrivial especially as a lot of folks end up taking something open-source and thinking they can definitely do a lot better (you can.... there's a lot of "but"s there though).
- fmbb 2y ago
  OK but the alternative if you think Kubernetes is too much magic when you want to operate hundreds of clusters with tens of thousands of nodes is? Some bash and Ansible and EC2? That is usually what Kubernetes haters suggest one does to simplify.
  - ukuina 2y ago
    HashiCorp Nomad?
  - zsoltkacsandi 2y ago
    Hashicorp Nomad, Docker Swarm, Apache Mesos, AWS ECS? I love that the Kubernetes lovers tend to forget that Kubernetes is just one tool, and they believe that the only possible alternative to this coolness is that sweaty sysadmins writing bash scripts in a dark room.
    - fmbb 2y ago
      I’m absolutely not a Kubernetes lover. Bash and Andible etc. is just a very common suggestion from haters. I thought Mesos was kinda dead nowadays, good to hear it’s still kicking. Last time I used it it the networking was a bit annoying, not able to provide virtual network interfaces but only ports. It seems like if you are going to operate these things, picking a solution with a huge community and in active development feels like the smart thing to do. Nomad is very nice to use from a developer perspective, and it’s nice to hear infrastructure people preferring it. From outside the reason people pick Kubernetes seems to be the level of control of infra and security teams want over things like networking and disk.
lenkite 2y ago
"Kubernetes is etcd, apiserver, and controllers....Sure it's a lot of YAML but I don't understand the rep." Sir, I upvoted you for your wonderful sense of humour.
deleted 2y ago
[deleted]

by remram 2y ago

jimmaswell 2y ago

Yeah, I'm sure there are tricky details as in anything but the core idea doesn't sound that complicated to me. I've been looking into it a bit after seeing this fun video a while ago where a DOS BBS is ran on kubernetes. https://youtu.be/wLVHXn79l8M?si=U2FexAMKd3zQVA82 https://youtu.be/wLVHXn79l8M?si=U2FexAMKd3zQVA82

lolinder 2y ago

Because the root comment is mostly but not quite right: there are indeed a large subset of developers that aren't interested in thinking about infrastructure, but there are many subcategories of those people, and many of them aren't fly.io customers. A large number of people who are in that category aren't happy to let someone else handle their infra. They're not interested in infra in the sense that they don't believe it should be more complicated than "start process on Linux box and set up firewall and log rotation". For some applications these people are absolutely right, but they've persuaded themselves that that means it's the best way to handle all use cases, which makes them see Kubernetes as way more complex than is necessary, rather than as a roll-your-own ECS for those who would otherwise truly need a cloud provider.

worldsayshi 2y ago
Feels like swe engineers are talking past each other a lot about these topics. I assume everyone wants to be in control of their environment. But with so many ways to compose your infra that means a lot of different things for different people.

docandrew 2y ago

“etcd, apiserver, and controllers.” …and containerd and csi plugins and kubelet and cni plugins and kubectl and kube-proxy and ingresses and load balancers…

igmor 2y ago
These components are very different in complexity and scope. Let's be real: a seasoned developer is mostly familiar with load balancers and ingress controllers, so this will be mostly about naming and context. I agree though once you learn about k8s it becomes less mysterious but that also means the author hasn't pushed it to the limits. Outages in the control plane could be pretty nasty and it is easy to have them by creating an illusion everything is kind of free in k8s.
- nicoburns 2y ago
  A really simple setup for many smaller organisations wouldn't have a load balancer at all.
  - lakomen 2y ago
    [dead]
  - darkwater 2y ago
    No load balancer means... entering one node only? Doing DNS RR over all the nodes? If you don't have a load balancer in front, why are you even using Kubernetes? Deploy a single VM and call it a day! I mean, in my homelab I do have Kubernetes and no LB in front, but it's a homelab for fun and learn K8s internals. But in a professional environment...
  - dilyevsky 2y ago
    No code at all even - just use excel
zeroq 2y ago
typical how to program an owl: step one: draw a circle step two: import the rest of the owl

figassis 2y ago

I use k8s, wouldn't call it simple, but there are ways to minimize the complexity of your setup. Mostly, what devs see as complexity is k8s packages a lot of system fundamentals, like networking, storage, name resolution, distributed architectures, etc, and if you mainly spent your career in a single lane, k8s becomes impossible to grasp. Not saying those devs are wrong, not everyone needs to be a networking pro. K8s is meant to be operated by some class of engineers, and used by another. Just like you have DBAs, sysadmins, etc, maybe your devops should have more system experience besides terraform.

spratzt 2y ago

Genuinely curious about what sort of business stores and processes 14 PB on a 60 node cluster.

remram 2y ago
Research institution. The department saw more need for storage than Kubernetes compute so that's what we're growing. Nowadays you can get storage machines with 1 PB in them.
superq 2y ago
Yeah, that's an interesting question, because it sounds like a ton of data vs not enough compute, but, aside from this all being in a SAN or large storage array: The larger Supermicro or Quanta storage servers can easily handle 36 HDD's each, or even more. So with just 16 of those with 36x24TB disks, that meets the ~14PB capacity mark, leaving 44 remaining nodes for other compute task, load balancing, NVME clusters, etc.
- remram 2y ago
  We have boxes with up to 45 drives yes.

cullenking 2y ago

Hey fellow k8s+ceph on bare metaler! We only have a 13 machine rack and 350tb of raw storage. No major issues with ceph after 16.x and all nvme storage though.

chucky_z 2y ago

I consider a '60+ node' kubernetes cluster is very small. Kubernetes at that scale is genuinely excellent! At 6000, 60000, and 600000 nodes it becomes very different and goes from 'Hey, this is pretty great' to 'What have I done?' The maintenance costs of running more than a hundred clusters is incredibly nontrivial especially as a lot of folks end up taking something open-source and thinking they can definitely do a lot better (you can.... there's a lot of "but"s there though).

fmbb 2y ago
OK but the alternative if you think Kubernetes is too much magic when you want to operate hundreds of clusters with tens of thousands of nodes is? Some bash and Ansible and EC2? That is usually what Kubernetes haters suggest one does to simplify.
- ukuina 2y ago
  HashiCorp Nomad?
- zsoltkacsandi 2y ago
  Hashicorp Nomad, Docker Swarm, Apache Mesos, AWS ECS? I love that the Kubernetes lovers tend to forget that Kubernetes is just one tool, and they believe that the only possible alternative to this coolness is that sweaty sysadmins writing bash scripts in a dark room.
  - fmbb 2y ago
    I’m absolutely not a Kubernetes lover. Bash and Andible etc. is just a very common suggestion from haters. I thought Mesos was kinda dead nowadays, good to hear it’s still kicking. Last time I used it it the networking was a bit annoying, not able to provide virtual network interfaces but only ports. It seems like if you are going to operate these things, picking a solution with a huge community and in active development feels like the smart thing to do. Nomad is very nice to use from a developer perspective, and it’s nice to hear infrastructure people preferring it. From outside the reason people pick Kubernetes seems to be the level of control of infra and security teams want over things like networking and disk.

lenkite 2y ago

"Kubernetes is etcd, apiserver, and controllers....Sure it's a lot of YAML but I don't understand the rep." Sir, I upvoted you for your wonderful sense of humour.

deleted 2y ago

[deleted]