3 ms·

How does this handle multiple containers for a Pod? In a container runtime k8s, containers within a pod share the same network namespace (same localhost) and po

How does this handle multiple containers for a Pod? In a container runtime k8s, containers within a pod share the same network namespace (same localhost) and possibly pid namespace. The press release maps pods to machines, but provides no mapping of pod containers to a Fly.io concept. Are multiple containers allowed? Do they share the same network namespace? Is sharing PID namespace optional? Having multiple containers per pod is a core functionality of Kubernetes.

remram 3y ago
You can use mount namespaces, or even containers in your VM. Maybe that's how?
- kuhsaft 3y ago
  Fly.io claims it’s “just a VM”. But, Fly.io Machines are an abstraction of microVMs using Firecracker. Building upon that, the FKS implementation is an abstraction on top of Fly.io Machines. So what I’m asking is how, if even, does the FKS implementation support multiple containers for a pod? Using FKS, the abstraction is no longer a VM. It seems that Fly.io Machines support multiple processes for a single container, but not multiple containers per Machine [0]. This means one container image per Machine and thus no shared network namespace across multiple containers. [0] https://community.fly.io/t/multi-process-machines/8375 https://community.fly.io/t/multi-process-machines/8375
  - tptacek 3y ago
    You can run Docker on a Fly Machine, and run arbitrary numbers of containers inside of it. Or you can run lots of small Fly Machines. FKS is just one model for deploying things.
    - kuhsaft 3y ago
      Right, but what is the point of FKS then? It’s no longer Kubernetes if it doesn’t support a core behavior of Kubernetes. If you only support deploying single containers with single processes on FKS, then you might as well use flyctl. It’s a solvable issue of course. The virtual-kubelet implementation would need to create a Machine running a container runtime image that would then run a the pod containers to match the pod configuration. I think that some disclaimers of the limitations of FKS compared to standardized Kubernetes should be present and highly visible.
      - tptacek 3y ago
        The article discusses what you get by using K8s alongside Fly.io. If you want to bin-pack containers onto Fly Machines, you can of course just boot up your own K8s cluster here; that has always been an option.
        kuhsaft 3y ago
        It should discuss what you don’t get compared to the standard behaviors of Kubernetes.
        remram 3y ago
        The problem is not the need to bin-pack, the problem is completeness. Sidecars and multiple containers are used for logging, backups, etc. Not to mention that if you grab a manifest or chart for an app, it is going to have pods with multiple containers (whether that's strictly needed or not), and those won't work on fly.io. This is a critical feature available in every Kubernetes offering, and as such people rely on it. Trying to say that maybe you can do without is missing the forest for the trees. Like saying that your C compiler doesn't need arrays because it has pointers: sure, maybe, but now good luck compiling any existing code. Maybe don't call it a C compiler if no C program will work on it unmodified.
        tptacek 3y ago
        Again: Fly Machines are just Linux VMs, and you have root on them.
        remram 3y ago
        I'm discussing the capabilities of Fly Kubernetes, not Fly Machines. This is good news though, it means they might get there in the future.
      - jipperinbham 3y ago
        We're actively working on the ability to run multiple processes with different images because it's something people using our platform want and it just happens to also be something needed for us to make FKS a more standardized Kubernetes offering.
javaunsafe2019 3y ago
Why should you do this - sounds like an antipattern to me
- kuhsaft 3y ago
  It’s used widely in the Kubernetes world and is known as sidecars [0]. [0] https://kubernetes.io/blog/2023/08/25/native-sidecar-containers/ https://kubernetes.io/blog/2023/08/25/native-sidecar-contain...
  - javaunsafe2019 3y ago
    Widely? I lived in this helm, kubernetes, pulumi world the past 4 years and we followed the simple rule of one service/container per pod. Why add complexity where it’s not needed. Like running a dB a and a service in the same docker container - a no go for me and many.
    - qeternity 3y ago
      Nobody would suggest this. But what about a metrics scraper for your db pod? That’s where sidecars come in.
    - kuhsaft 3y ago
      The thing is you might not be adding the sidecars yourself. Kubernetes has a resource called MutatingAdmissionWebhook that allows mutations of the Pod object before creation. I think the most common use case is for service meshes. Your Deployment might not have any sidecars, but the service mesh controller will automatically add a network proxy container to your pod via the admission webhook. Another use case would be OpenTelemetry or some other observability service sidecar injection for auto-instrumentation.
      - javaunsafe2019 3y ago
        Ok that was something inward missing on my end. Thanks for all the explanations and sorry for me being wrong

remram 3y ago

You can use mount namespaces, or even containers in your VM. Maybe that's how?

kuhsaft 3y ago
Fly.io claims it’s “just a VM”. But, Fly.io Machines are an abstraction of microVMs using Firecracker. Building upon that, the FKS implementation is an abstraction on top of Fly.io Machines. So what I’m asking is how, if even, does the FKS implementation support multiple containers for a pod? Using FKS, the abstraction is no longer a VM. It seems that Fly.io Machines support multiple processes for a single container, but not multiple containers per Machine [0]. This means one container image per Machine and thus no shared network namespace across multiple containers. [0] https://community.fly.io/t/multi-process-machines/8375 https://community.fly.io/t/multi-process-machines/8375
- tptacek 3y ago
  You can run Docker on a Fly Machine, and run arbitrary numbers of containers inside of it. Or you can run lots of small Fly Machines. FKS is just one model for deploying things.
  - kuhsaft 3y ago
    Right, but what is the point of FKS then? It’s no longer Kubernetes if it doesn’t support a core behavior of Kubernetes. If you only support deploying single containers with single processes on FKS, then you might as well use flyctl. It’s a solvable issue of course. The virtual-kubelet implementation would need to create a Machine running a container runtime image that would then run a the pod containers to match the pod configuration. I think that some disclaimers of the limitations of FKS compared to standardized Kubernetes should be present and highly visible.
    - tptacek 3y ago
      The article discusses what you get by using K8s alongside Fly.io. If you want to bin-pack containers onto Fly Machines, you can of course just boot up your own K8s cluster here; that has always been an option.
      - kuhsaft 3y ago
        It should discuss what you don’t get compared to the standard behaviors of Kubernetes.
      - remram 3y ago
        The problem is not the need to bin-pack, the problem is completeness. Sidecars and multiple containers are used for logging, backups, etc. Not to mention that if you grab a manifest or chart for an app, it is going to have pods with multiple containers (whether that's strictly needed or not), and those won't work on fly.io. This is a critical feature available in every Kubernetes offering, and as such people rely on it. Trying to say that maybe you can do without is missing the forest for the trees. Like saying that your C compiler doesn't need arrays because it has pointers: sure, maybe, but now good luck compiling any existing code. Maybe don't call it a C compiler if no C program will work on it unmodified.
        tptacek 3y ago
        Again: Fly Machines are just Linux VMs, and you have root on them.
        remram 3y ago
        I'm discussing the capabilities of Fly Kubernetes, not Fly Machines. This is good news though, it means they might get there in the future.
    - jipperinbham 3y ago
      We're actively working on the ability to run multiple processes with different images because it's something people using our platform want and it just happens to also be something needed for us to make FKS a more standardized Kubernetes offering.

javaunsafe2019 3y ago

Why should you do this - sounds like an antipattern to me

kuhsaft 3y ago
It’s used widely in the Kubernetes world and is known as sidecars [0]. [0] https://kubernetes.io/blog/2023/08/25/native-sidecar-containers/ https://kubernetes.io/blog/2023/08/25/native-sidecar-contain...
- javaunsafe2019 3y ago
  Widely? I lived in this helm, kubernetes, pulumi world the past 4 years and we followed the simple rule of one service/container per pod. Why add complexity where it’s not needed. Like running a dB a and a service in the same docker container - a no go for me and many.
  - qeternity 3y ago
    Nobody would suggest this. But what about a metrics scraper for your db pod? That’s where sidecars come in.
  - kuhsaft 3y ago
    The thing is you might not be adding the sidecars yourself. Kubernetes has a resource called MutatingAdmissionWebhook that allows mutations of the Pod object before creation. I think the most common use case is for service meshes. Your Deployment might not have any sidecars, but the service mesh controller will automatically add a network proxy container to your pod via the admission webhook. Another use case would be OpenTelemetry or some other observability service sidecar injection for auto-instrumentation.
    - javaunsafe2019 3y ago
      Ok that was something inward missing on my end. Thanks for all the explanations and sorry for me being wrong