4 ms·

And system calls and filesystems and sockets and LVM and... Sure at some point there are too many layers to count but I wouldn't say any of this is "Kubernetes

by remram 2y ago

And system calls and filesystems and sockets and LVM and... Sure at some point there are too many layers to count but I wouldn't say any of this is "Kubernetes". What people tend to be hung about is the difficulty of Kubernetes compared to `docker run` or `docker compose up`. That is what I am surprised about. I never had any issue with kubelet, or kube-proxy, or CSI plugins, or CNI plugins. That is after years of running a multi-tenant cluster in a research institution. I think about those about as much as I think about ext4, runc, or GRUB.

ffsm8 2y ago
But you just said that you had issues with ceph? How is that not a CSI problem? And CNI problems are extremely normal. Pretty much anyone that didn't just use weavenet and called it a day has had to spend quiet a bit of time to figure it out. If you already know networking by heart it's obviously going to be easier, but few devs do.
- freedomben 2y ago
  Very fair, although with managed services which are increasingly available, you don't typically need to think about CSI or CNI.
  - deleted 2y ago
    [deleted]
  - cuu508 2y ago
    Hence > Kubernetes is not the first thing that comes to mind when I think of "understanding where their code is running and what it's doing"...
    - remram 2y ago
      CSI and CNI do about as much magic as `docker volume` and `docker network`. People act like their web framework and SQL connection pooler and stuff are so simple, while Kubernetes is complex and totally inscrutable for mortals, and I don't get it. It has a couple of moving parts, but it is probably simpler overall than SystemD.
      - formerly_proven 2y ago
        I was genuinely surprised that k8s turned out to actually be pretty straightforward and very sensible after years of never having anything to do with it and just hearing about it on the net. Turns out opinions are just like after all. That being said, what people tend to build on top of that foundation is a somewhat different story.
        abustamam 2y ago
        I asked chatgpt the other day to explain to me Kubernetes. I still don't understand it. Can you share with me what clicked with you, or resources that helped you?
        amazingman 2y ago
        Controller in charge of a specific type of object watches a database table representing the object type. Database table represents the desired state of things. When entries to the table are CRUD-ed, that represents a change to the desired state of things. Controller interacts with the larger system to bring the state of things into alignment with the new desired state of things. "The larger system" is more controllers in charge of other object types, doing the same kind of work for its object types There is an API implemented for CRUD-ing each object type. The API specification (model) represents something important to developers, like a group of containers (Pod), a load balancer with VIP (Service), a network volume (PersistentVolume), and so on. Hand wave hand wave, Lego-style infrastructure. None of the above is exactly correct (e.g. the DB is actually a k/v store), but it should be conceptually correct.
        DrFalkyn 2y ago
        Is there only a single controller ? What happens if goes down? If multiple controllers, how do they coordinate ?
        chronid 2y ago
- p_l 2y ago
  Because CSI is just a way to connect a volume to a pod. Ceph is its own cluster of kettles filled with fishes
- remram 2y ago
  Never had a problem with the CSI plugin, I had problems with the Ceph cluster itself. No, I wouldn't call Ceph part of Kubernetes. You definitely can run Kubernetes without running Ceph or any storage system, and you already rely on a distributed storage system if you use the cloud whether you use Kubernetes or not. So I wouldn't count this as added complexity from Kubernetes.
  - ffsm8 2y ago
    I'm not sure I can agree with that interpretation. CSI is basically an interface that has to be implemented. If you discount issues like that, you can safely say that it's impossible to have any issues with CSI, because it's always going to be with one of it's implementation. That feels a little disingenuous, but maybe that's just me.
    - remram 2y ago
      So if you run Kubernetes in the cloud, you consider the entire cloud provider's block storage implementation to be part of Kubernetes too? For example you'd say AWS EBS is part of Kubernetes?
      - ffsm8 2y ago
        In the context of this discussion, which is about the complexity of the k8s stack: yes. Youre ultimately gonna have to use a storage of some form unless you're just a stateless service/keep the services with state out of k8s. That's why I'd include it, and the fact that you can use multiple storage backends, each with their own challenges and pitfalls makes k8s indeed quiet complex. You could argue that multinode PaaS is always going to be complex, and frankly- I'd agree with that. But that was kinda the original point. At least as far as I interpreted it: k8s is not simple and you most likely didn't need it either. But if you do need a distributed PaaS, then it's probably a good idea to use it. Doesn't change the fact that it's a complex system.
        remram 2y ago
ghaff 2y ago
Well, and the fact that in addition to Kubernetes itself, there are a gazillion adjacent products and options in the cloud-native space. Many/most of which a relatively simple setup may not need. But there's a lot of complexity. But then there's always always a lot of complexity and abstraction. Certainly, most software people don't need to know everything about what a CPU is doing at the lowest levels.

by remram 2y ago

ffsm8 2y ago

But you just said that you had issues with ceph? How is that not a CSI problem? And CNI problems are extremely normal. Pretty much anyone that didn't just use weavenet and called it a day has had to spend quiet a bit of time to figure it out. If you already know networking by heart it's obviously going to be easier, but few devs do.

freedomben 2y ago
Very fair, although with managed services which are increasingly available, you don't typically need to think about CSI or CNI.
- deleted 2y ago
  [deleted]
- cuu508 2y ago
  Hence > Kubernetes is not the first thing that comes to mind when I think of "understanding where their code is running and what it's doing"...
  - remram 2y ago
    CSI and CNI do about as much magic as `docker volume` and `docker network`. People act like their web framework and SQL connection pooler and stuff are so simple, while Kubernetes is complex and totally inscrutable for mortals, and I don't get it. It has a couple of moving parts, but it is probably simpler overall than SystemD.
    - formerly_proven 2y ago
      I was genuinely surprised that k8s turned out to actually be pretty straightforward and very sensible after years of never having anything to do with it and just hearing about it on the net. Turns out opinions are just like after all. That being said, what people tend to build on top of that foundation is a somewhat different story.
      - abustamam 2y ago
        I asked chatgpt the other day to explain to me Kubernetes. I still don't understand it. Can you share with me what clicked with you, or resources that helped you?
        amazingman 2y ago
        Controller in charge of a specific type of object watches a database table representing the object type. Database table represents the desired state of things. When entries to the table are CRUD-ed, that represents a change to the desired state of things. Controller interacts with the larger system to bring the state of things into alignment with the new desired state of things. "The larger system" is more controllers in charge of other object types, doing the same kind of work for its object types There is an API implemented for CRUD-ing each object type. The API specification (model) represents something important to developers, like a group of containers (Pod), a load balancer with VIP (Service), a network volume (PersistentVolume), and so on. Hand wave hand wave, Lego-style infrastructure. None of the above is exactly correct (e.g. the DB is actually a k/v store), but it should be conceptually correct.
        DrFalkyn 2y ago
        Is there only a single controller ? What happens if goes down? If multiple controllers, how do they coordinate ?
        chronid 2y ago
p_l 2y ago
Because CSI is just a way to connect a volume to a pod. Ceph is its own cluster of kettles filled with fishes
remram 2y ago
Never had a problem with the CSI plugin, I had problems with the Ceph cluster itself. No, I wouldn't call Ceph part of Kubernetes. You definitely can run Kubernetes without running Ceph or any storage system, and you already rely on a distributed storage system if you use the cloud whether you use Kubernetes or not. So I wouldn't count this as added complexity from Kubernetes.
- ffsm8 2y ago
  I'm not sure I can agree with that interpretation. CSI is basically an interface that has to be implemented. If you discount issues like that, you can safely say that it's impossible to have any issues with CSI, because it's always going to be with one of it's implementation. That feels a little disingenuous, but maybe that's just me.
  - remram 2y ago
    So if you run Kubernetes in the cloud, you consider the entire cloud provider's block storage implementation to be part of Kubernetes too? For example you'd say AWS EBS is part of Kubernetes?
    - ffsm8 2y ago
      In the context of this discussion, which is about the complexity of the k8s stack: yes. Youre ultimately gonna have to use a storage of some form unless you're just a stateless service/keep the services with state out of k8s. That's why I'd include it, and the fact that you can use multiple storage backends, each with their own challenges and pitfalls makes k8s indeed quiet complex. You could argue that multinode PaaS is always going to be complex, and frankly- I'd agree with that. But that was kinda the original point. At least as far as I interpreted it: k8s is not simple and you most likely didn't need it either. But if you do need a distributed PaaS, then it's probably a good idea to use it. Doesn't change the fact that it's a complex system.
      - remram 2y ago

ghaff 2y ago

Well, and the fact that in addition to Kubernetes itself, there are a gazillion adjacent products and options in the cloud-native space. Many/most of which a relatively simple setup may not need. But there's a lot of complexity. But then there's always always a lot of complexity and abstraction. Certainly, most software people don't need to know everything about what a CPU is doing at the lowest levels.