5 ms·

The wildest part is they’ll take those massive machines, shard them into tiny Kubernetes pods, and then engineer something that “scales horizontally” with the n

The wildest part is they’ll take those massive machines, shard them into tiny Kubernetes pods, and then engineer something that “scales horizontally” with the number of pods.

jesse__ 9mo ago
Yeah man, you're running on a multitasking OS. Just let the scheduler do the thing.
- mystraline 9mo ago
  Its all fun and games, until the control plane gets killed by the OOMkiller. Naturally, that detaches all your containers. And theres no seamless reattach for control plane restart.
  - dgxyz 9mo ago
    Or your CNI implementation is made of rolled up turds and you lose a node or two from the cluster control plane every day. (Large EKS cluster)
- zacmps 9mo ago
  Until you need to schedule GPUs or other heterogenous compute...
  - jesse__ 9mo ago
    Are you saying that running your application in a pile of containers somehow helps that problem ..? It's the same problem as CPU scheduling, we just don't have good schedulers yet.. Lots of people are working on it though
    - zacmps 9mo ago
      Not really? At the moment it's done by some user-land job scheduler. That could be something container based like k8s, something in-process like ray, or a workload manager like slurm.
- dgxyz 9mo ago
  Yeah this. As I explain many times to people, processes are the only virtualisation you need if you aren’t running a fucked up pile of shit. The problem we have is fucked up piles of shit not that we don’t have kubernetes and don’t have containers.
  - jesse__ 9mo ago
    Hahhah, yuuuup. I can maybe make a case for running in containers if you need some specific security properties but .. mostly I think the proliferation of 'fucked up piles of shit' is the problem.
  - jchw 9mo ago
    Containers are just processes plus some namespacing, nothing really stops you from running very huge tasks on Kubernetes nodes. I think the argument for containers and Kubernetes is pretty good owing to their operational advantages (OCI images for distributing software, distributed cron jobs in Kubernetes, observability tools like Falco, and so forth). So I totally understand why people preemptively choose Kubernetes before they are scaling to the point where having a distributed scheduler is strictly necessary. Hadoop, on the other hand, you're definitely paying a large upfront cost for scalability you very much might not need.
    - dgxyz 9mo ago
      Time to market and operational costs are much higher on kubernetes and containers from many years of actual experience. This is both in production and in development. It’s usually a bad engineering decision. If you’re doing a lift and shift, it’s definitely bad. If you’re starting greenfield it makes sense to pick technology stacks that don’t incur this crap. It only makes sense if you’re managing large amounts of large siloed bits of kit. I’ve not seen this other than at unnamed big tech companies. 99.9% of people are just burning money for a fashion show where everyone is wearing clown suits because someone said clown suits are good.
      - bartread 9mo ago
        Thanks. You’ve reassured me that I’m not going mad when I look at our project repo and seriously consider binning the Dockerfile and deploying direct to Ubuntu. The project is a Ruby on Rails app that talks to PostreSQL and a handful of third party services. It just seems unnecessary to include the complexity of containers.
ahartmetz 9mo ago
I think my brain hurts
andai 9mo ago
I had to re-read this a few times. I am sad now.
cyberpunk 9mo ago
To be fair each of those pods can have dedicated, separate external storage volumes which may actually help and it’s def easier than maintaining 200 iscsi or more whatever targets yourself
jayd16 9mo ago
I mean, a large part of the point is that you can run on separate physical machines, too.
pnt12 9mo ago
This is especially aggravating when the os inside the container and the language runtimes are much heavier than the process itself. I've seen arguments for nano services (I wouldn't even call them micros services), that completely ignored that part. Split a small service in n tiny services, such that you have 10(os, runtime, 0.5) rather than 2(os, runtime, x).
- SpaceNugget 9mo ago
  There is no os inside the container. That's a big part of the reason containerization is so popular as a replacement for heavier alternatives like full virtualization. I get that it's a bit confusing with base image names like "ubuntu" and "fedora", but that doesn't mean that there is a nested copy of ubuntu/fedora running for every container.

The wildest part is they’ll take those massive machines, shard them into tiny Kubernetes pods, and then engineer something that “scales horizontally” with the number of pods.

jesse__ 9mo ago

Yeah man, you're running on a multitasking OS. Just let the scheduler do the thing.

mystraline 9mo ago
Its all fun and games, until the control plane gets killed by the OOMkiller. Naturally, that detaches all your containers. And theres no seamless reattach for control plane restart.
- dgxyz 9mo ago
  Or your CNI implementation is made of rolled up turds and you lose a node or two from the cluster control plane every day. (Large EKS cluster)
zacmps 9mo ago
Until you need to schedule GPUs or other heterogenous compute...
- jesse__ 9mo ago
  Are you saying that running your application in a pile of containers somehow helps that problem ..? It's the same problem as CPU scheduling, we just don't have good schedulers yet.. Lots of people are working on it though
  - zacmps 9mo ago
    Not really? At the moment it's done by some user-land job scheduler. That could be something container based like k8s, something in-process like ray, or a workload manager like slurm.
dgxyz 9mo ago
Yeah this. As I explain many times to people, processes are the only virtualisation you need if you aren’t running a fucked up pile of shit. The problem we have is fucked up piles of shit not that we don’t have kubernetes and don’t have containers.
- jesse__ 9mo ago
  Hahhah, yuuuup. I can maybe make a case for running in containers if you need some specific security properties but .. mostly I think the proliferation of 'fucked up piles of shit' is the problem.
- jchw 9mo ago
  Containers are just processes plus some namespacing, nothing really stops you from running very huge tasks on Kubernetes nodes. I think the argument for containers and Kubernetes is pretty good owing to their operational advantages (OCI images for distributing software, distributed cron jobs in Kubernetes, observability tools like Falco, and so forth). So I totally understand why people preemptively choose Kubernetes before they are scaling to the point where having a distributed scheduler is strictly necessary. Hadoop, on the other hand, you're definitely paying a large upfront cost for scalability you very much might not need.
  - dgxyz 9mo ago
    Time to market and operational costs are much higher on kubernetes and containers from many years of actual experience. This is both in production and in development. It’s usually a bad engineering decision. If you’re doing a lift and shift, it’s definitely bad. If you’re starting greenfield it makes sense to pick technology stacks that don’t incur this crap. It only makes sense if you’re managing large amounts of large siloed bits of kit. I’ve not seen this other than at unnamed big tech companies. 99.9% of people are just burning money for a fashion show where everyone is wearing clown suits because someone said clown suits are good.
    - bartread 9mo ago
      Thanks. You’ve reassured me that I’m not going mad when I look at our project repo and seriously consider binning the Dockerfile and deploying direct to Ubuntu. The project is a Ruby on Rails app that talks to PostreSQL and a handful of third party services. It just seems unnecessary to include the complexity of containers.

ahartmetz 9mo ago

I think my brain hurts

andai 9mo ago

I had to re-read this a few times. I am sad now.

cyberpunk 9mo ago

To be fair each of those pods can have dedicated, separate external storage volumes which may actually help and it’s def easier than maintaining 200 iscsi or more whatever targets yourself

jayd16 9mo ago

I mean, a large part of the point is that you can run on separate physical machines, too.

pnt12 9mo ago

This is especially aggravating when the os inside the container and the language runtimes are much heavier than the process itself. I've seen arguments for nano services (I wouldn't even call them micros services), that completely ignored that part. Split a small service in n tiny services, such that you have 10(os, runtime, 0.5) rather than 2(os, runtime, x).

SpaceNugget 9mo ago
There is no os inside the container. That's a big part of the reason containerization is so popular as a replacement for heavier alternatives like full virtualization. I get that it's a bit confusing with base image names like "ubuntu" and "fedora", but that doesn't mean that there is a nested copy of ubuntu/fedora running for every container.