5 ms·

I don't think it's fizzling out, it's kind of becoming a basic standard. And if your use case is simple enough to not need Kubernetes, then doing it with Kubern

by jstx1 4y ago

I don't think it's fizzling out, it's kind of becoming a basic standard. And if your use case is simple enough to not need Kubernetes, then doing it with Kubernetes isn't that much more complicated either.

jiggawatts 4y ago
A standard in the Linux world. Windows doesn’t have “distributions”, so there’s less need for containers in that space. Microsoft uses Service Fabric internally for the Azure back-end, which covers most of the other use-cases of Kubernetes.
- Jenk 4y ago
  Service Fabric needs to die. It is an abomination on software. Why, why, *WHY* does each SF service need to have some terrible wrapper code that forces it to run in, and only in, SF? Oh ya.. vendor lock-in. No thanks. Yes, I'm aware that MS added support for docker containers in SF. That integration is dreadful. They did about as good a job of it as the WCF team did of shoe-horning "RESTFul Services" in. > Windows doesn’t have “distributions”, so there’s less need for containers in that space. I don't follow. What does distributions have to do with need for containers?
  - jiggawatts 4y ago
    > What does distributions have to do with need for containers? A lot (but not everything). It's important to recognise where the "stable API/ABI boundaries" are. In Linux, the stable boundary is the kernel. Userspace is unstable. There are many distributions, with many small and big differences. Different C runtimes, different folder structures, different configs, modules, etc... In Windows, the stable boundary is the Win32, COM, and .NET APIs, which are in user space. The kernel boundary is not stable[1] however, which also matters. Even Linus Torvalds has trouble distributing his hobby dive-computer software to Linux![2] He has no issues with MacOS and Windows, because they have stable APIs and distribution mechanisms. With Linux, the kernel is stable, but essentially nothing else is. If you want to distribute software to everyone, then this is a complex problem with many moving parts that you have to manage yourself. Containers sidestep this by packaging up the distribution along with the software being deployed. This works because the kernel ABI is stable. [1] In the Windows world you don't really need to package up the "Windows distribution" because there is only one: Microsoft Windows. Conversely, Windows containers aren't truly portable because the Windows kernel ABI isn't stable. However Microsoft changed this with Windows 11 and Server 2022, the kernel ABI is now "stable", or at least sufficiently stable for Server Core containers to be portable for 5+ years as-is. [2] He's complaining about "desktop applications", but the exact same rant would apply to server software also. This one talk made me understand containers and why they're so important in the Linux world: https://www.youtube.com/watch?v=Pzl1B7nB9Kc https://www.youtube.com/watch?v=Pzl1B7nB9Kc
    - moondev 4y ago
      Even if Windows containers are not as portable, the ability to manage them from the same Kubernetes controlplane that manages Linux containers seems pretty powerful. It also enables you to do it outside of Azure/Service Fabric on other clouds or metal. https://kubernetes.io/docs/concepts/windows/ https://kubernetes.io/docs/concepts/windows/ https://docs.microsoft.com/en-us/azure/aks/windows-faq?tabs=azure-cli https://docs.microsoft.com/en-us/azure/aks/windows-faq?tabs=...
      - jiggawatts 4y ago
        You're trading one convenience for a world of hurt if you use Kubernetes with Windows. Just take a casual glance at the "Windows containers" issues list on GitHub: https://github.com/microsoft/Windows-Containers/issues https://github.com/microsoft/Windows-Containers/issues Some of the very recently closed issues were jaw-dropping, such as totally broken networking in common scenarios. DNS resolution is very different in Windows compared to Linux, making much of the "neatness" of Kubernetes pod-to-pod communication not work. There is no maximum memory limit in Windows kernel "job objects" (equivalent to cgroups), so one memory leak in one pod can kill an entire node or cluster. This is very hard to solve, and I've seen it take out Service Fabric clusters also. Etc, etc...
        moondev 4y ago
        Thanks for sharing, it all seems terrifying but still very interesting to explore. I was curious how the Windows nodes would even join - turns out they support cloud-init just like Linux machines. https://cloudbase.it/cloudbase-init/ https://cloudbase.it/cloudbase-init/ In theory this means they could be managed then by something like KubeVirt instead of treating them like a worker node. https://kubevirt.io/user-guide/virtual_machines/startup_scripts/#cloud-init https://kubevirt.io/user-guide/virtual_machines/startup_scri... Excited to see this space continue to evolve for sure.
        jiggawatts 4y ago
        Cloudbase-init is interesting, but Microsoft doesn't support or (apparently) use it. There's no mention of it in their docs: https://www.google.com/search?q=site%3Adocs.microsoft.com+%22cloudbase-init%22 https://www.google.com/search?q=site%3Adocs.microsoft.com+%2... New Windows VMs are generally built using PowerShell DSC or a similar native tool.
- voidfunc 4y ago
  Kubernetes is heavily used internally at MS as well, especially for newer stuff.
  - jiggawatts 4y ago
    Are they running Windows on it? If so, they're very brave people.
    - voidfunc 4y ago
      Mixed workloads. Not everything inside MS is Windows, in fact, more than people probably realize is running on Linux. We have our own internal Linux distribution (Mariner) and own Kinvolk (which maintained CoreOS/Container Linux/Flatcar Linux).
x86x87 4y ago
Lol. Famous last words. IMHO K8s is a trojan horse from Google. They are irrelevant in the cloud space and are trying to stay in the game. K8s is not Borg. Not even close. It's a toy that has limited use. The revolutionary tech is the Linux kernel features that enabled containers, overlay filesystems and docker image format.
- aledalgrande 4y ago
  You do know that you can use k8s on AWS, DO, Azure & your own bare metal right?
  - x86x87 4y ago
    Should you use it though? That's the question!
    - vbezhenar 4y ago
      Why not? Kubernetes brings lot of useful features.
    - aledalgrande 4y ago
      that is up to you, your knowledge and your needs
- michaellee8 4y ago
  That famous tech called cgroup was actually a Google contribution. But I agree that k8s is essentially Google's step to make themselves relavant in Cloud. They have missed the initial opportunity by promoting their PaaS AppEngine instead of something IaaS like ec2 in the beginning of the cloud competition, so Google just play the open-source game and keep releasing stuffs that can be used in all three clouds to lure people to use GCP. But then k8s is a very nice piece of tech that allows one to manage large clusters without vendor lock-in.
  - x86x87 4y ago
    Cgroups, namespaces, apparmor/selinux, overlay filesystems, there is much more to containers than just cgroups. The no vendor lock is looks great on paper but you are locked in day one. (Eg on aws you probably use IAM, LB, ASG for K8 Nodes - you can maybe move it to another cloud but the effort is going to be significant). Cloud agnosticism is a lie.

by jstx1 4y ago

jiggawatts 4y ago

A standard in the Linux world. Windows doesn’t have “distributions”, so there’s less need for containers in that space. Microsoft uses Service Fabric internally for the Azure back-end, which covers most of the other use-cases of Kubernetes.

Jenk 4y ago
Service Fabric needs to die. It is an abomination on software. Why, why, *WHY* does each SF service need to have some terrible wrapper code that forces it to run in, and only in, SF? Oh ya.. vendor lock-in. No thanks. Yes, I'm aware that MS added support for docker containers in SF. That integration is dreadful. They did about as good a job of it as the WCF team did of shoe-horning "RESTFul Services" in. > Windows doesn’t have “distributions”, so there’s less need for containers in that space. I don't follow. What does distributions have to do with need for containers?
- jiggawatts 4y ago
  > What does distributions have to do with need for containers? A lot (but not everything). It's important to recognise where the "stable API/ABI boundaries" are. In Linux, the stable boundary is the kernel. Userspace is unstable. There are many distributions, with many small and big differences. Different C runtimes, different folder structures, different configs, modules, etc... In Windows, the stable boundary is the Win32, COM, and .NET APIs, which are in user space. The kernel boundary is not stable[1] however, which also matters. Even Linus Torvalds has trouble distributing his hobby dive-computer software to Linux![2] He has no issues with MacOS and Windows, because they have stable APIs and distribution mechanisms. With Linux, the kernel is stable, but essentially nothing else is. If you want to distribute software to everyone, then this is a complex problem with many moving parts that you have to manage yourself. Containers sidestep this by packaging up the distribution along with the software being deployed. This works because the kernel ABI is stable. [1] In the Windows world you don't really need to package up the "Windows distribution" because there is only one: Microsoft Windows. Conversely, Windows containers aren't truly portable because the Windows kernel ABI isn't stable. However Microsoft changed this with Windows 11 and Server 2022, the kernel ABI is now "stable", or at least sufficiently stable for Server Core containers to be portable for 5+ years as-is. [2] He's complaining about "desktop applications", but the exact same rant would apply to server software also. This one talk made me understand containers and why they're so important in the Linux world: https://www.youtube.com/watch?v=Pzl1B7nB9Kc https://www.youtube.com/watch?v=Pzl1B7nB9Kc
  - moondev 4y ago
    Even if Windows containers are not as portable, the ability to manage them from the same Kubernetes controlplane that manages Linux containers seems pretty powerful. It also enables you to do it outside of Azure/Service Fabric on other clouds or metal. https://kubernetes.io/docs/concepts/windows/ https://kubernetes.io/docs/concepts/windows/ https://docs.microsoft.com/en-us/azure/aks/windows-faq?tabs=azure-cli https://docs.microsoft.com/en-us/azure/aks/windows-faq?tabs=...
    - jiggawatts 4y ago
      You're trading one convenience for a world of hurt if you use Kubernetes with Windows. Just take a casual glance at the "Windows containers" issues list on GitHub: https://github.com/microsoft/Windows-Containers/issues https://github.com/microsoft/Windows-Containers/issues Some of the very recently closed issues were jaw-dropping, such as totally broken networking in common scenarios. DNS resolution is very different in Windows compared to Linux, making much of the "neatness" of Kubernetes pod-to-pod communication not work. There is no maximum memory limit in Windows kernel "job objects" (equivalent to cgroups), so one memory leak in one pod can kill an entire node or cluster. This is very hard to solve, and I've seen it take out Service Fabric clusters also. Etc, etc...
      - moondev 4y ago
        Thanks for sharing, it all seems terrifying but still very interesting to explore. I was curious how the Windows nodes would even join - turns out they support cloud-init just like Linux machines. https://cloudbase.it/cloudbase-init/ https://cloudbase.it/cloudbase-init/ In theory this means they could be managed then by something like KubeVirt instead of treating them like a worker node. https://kubevirt.io/user-guide/virtual_machines/startup_scripts/#cloud-init https://kubevirt.io/user-guide/virtual_machines/startup_scri... Excited to see this space continue to evolve for sure.
        jiggawatts 4y ago
        Cloudbase-init is interesting, but Microsoft doesn't support or (apparently) use it. There's no mention of it in their docs: https://www.google.com/search?q=site%3Adocs.microsoft.com+%22cloudbase-init%22 https://www.google.com/search?q=site%3Adocs.microsoft.com+%2... New Windows VMs are generally built using PowerShell DSC or a similar native tool.
voidfunc 4y ago
Kubernetes is heavily used internally at MS as well, especially for newer stuff.
- jiggawatts 4y ago
  Are they running Windows on it? If so, they're very brave people.
  - voidfunc 4y ago
    Mixed workloads. Not everything inside MS is Windows, in fact, more than people probably realize is running on Linux. We have our own internal Linux distribution (Mariner) and own Kinvolk (which maintained CoreOS/Container Linux/Flatcar Linux).

x86x87 4y ago

Lol. Famous last words. IMHO K8s is a trojan horse from Google. They are irrelevant in the cloud space and are trying to stay in the game. K8s is not Borg. Not even close. It's a toy that has limited use. The revolutionary tech is the Linux kernel features that enabled containers, overlay filesystems and docker image format.

aledalgrande 4y ago
You do know that you can use k8s on AWS, DO, Azure & your own bare metal right?
- x86x87 4y ago
  Should you use it though? That's the question!
  - vbezhenar 4y ago
    Why not? Kubernetes brings lot of useful features.
  - aledalgrande 4y ago
    that is up to you, your knowledge and your needs
michaellee8 4y ago
That famous tech called cgroup was actually a Google contribution. But I agree that k8s is essentially Google's step to make themselves relavant in Cloud. They have missed the initial opportunity by promoting their PaaS AppEngine instead of something IaaS like ec2 in the beginning of the cloud competition, so Google just play the open-source game and keep releasing stuffs that can be used in all three clouds to lure people to use GCP. But then k8s is a very nice piece of tech that allows one to manage large clusters without vendor lock-in.
- x86x87 4y ago
  Cgroups, namespaces, apparmor/selinux, overlay filesystems, there is much more to containers than just cgroups. The no vendor lock is looks great on paper but you are locked in day one. (Eg on aws you probably use IAM, LB, ASG for K8 Nodes - you can maybe move it to another cloud but the effort is going to be significant). Cloud agnosticism is a lie.