3 ms·

Even if Windows containers are not as portable, the ability to manage them from the same Kubernetes controlplane that manages Linux containers seems pretty powe

Even if Windows containers are not as portable, the ability to manage them from the same Kubernetes controlplane that manages Linux containers seems pretty powerful. It also enables you to do it outside of Azure/Service Fabric on other clouds or metal. https://kubernetes.io/docs/concepts/windows/ https://kubernetes.io/docs/concepts/windows/ https://docs.microsoft.com/en-us/azure/aks/windows-faq?tabs=azure-cli https://docs.microsoft.com/en-us/azure/aks/windows-faq?tabs=...

jiggawatts 4y ago
You're trading one convenience for a world of hurt if you use Kubernetes with Windows. Just take a casual glance at the "Windows containers" issues list on GitHub: https://github.com/microsoft/Windows-Containers/issues https://github.com/microsoft/Windows-Containers/issues Some of the very recently closed issues were jaw-dropping, such as totally broken networking in common scenarios. DNS resolution is very different in Windows compared to Linux, making much of the "neatness" of Kubernetes pod-to-pod communication not work. There is no maximum memory limit in Windows kernel "job objects" (equivalent to cgroups), so one memory leak in one pod can kill an entire node or cluster. This is very hard to solve, and I've seen it take out Service Fabric clusters also. Etc, etc...
- moondev 4y ago
  Thanks for sharing, it all seems terrifying but still very interesting to explore. I was curious how the Windows nodes would even join - turns out they support cloud-init just like Linux machines. https://cloudbase.it/cloudbase-init/ https://cloudbase.it/cloudbase-init/ In theory this means they could be managed then by something like KubeVirt instead of treating them like a worker node. https://kubevirt.io/user-guide/virtual_machines/startup_scripts/#cloud-init https://kubevirt.io/user-guide/virtual_machines/startup_scri... Excited to see this space continue to evolve for sure.
  - jiggawatts 4y ago
    Cloudbase-init is interesting, but Microsoft doesn't support or (apparently) use it. There's no mention of it in their docs: https://www.google.com/search?q=site%3Adocs.microsoft.com+%22cloudbase-init%22 https://www.google.com/search?q=site%3Adocs.microsoft.com+%2... New Windows VMs are generally built using PowerShell DSC or a similar native tool.

jiggawatts 4y ago

You're trading one convenience for a world of hurt if you use Kubernetes with Windows. Just take a casual glance at the "Windows containers" issues list on GitHub: https://github.com/microsoft/Windows-Containers/issues https://github.com/microsoft/Windows-Containers/issues Some of the very recently closed issues were jaw-dropping, such as totally broken networking in common scenarios. DNS resolution is very different in Windows compared to Linux, making much of the "neatness" of Kubernetes pod-to-pod communication not work. There is no maximum memory limit in Windows kernel "job objects" (equivalent to cgroups), so one memory leak in one pod can kill an entire node or cluster. This is very hard to solve, and I've seen it take out Service Fabric clusters also. Etc, etc...

moondev 4y ago
Thanks for sharing, it all seems terrifying but still very interesting to explore. I was curious how the Windows nodes would even join - turns out they support cloud-init just like Linux machines. https://cloudbase.it/cloudbase-init/ https://cloudbase.it/cloudbase-init/ In theory this means they could be managed then by something like KubeVirt instead of treating them like a worker node. https://kubevirt.io/user-guide/virtual_machines/startup_scripts/#cloud-init https://kubevirt.io/user-guide/virtual_machines/startup_scri... Excited to see this space continue to evolve for sure.
- jiggawatts 4y ago
  Cloudbase-init is interesting, but Microsoft doesn't support or (apparently) use it. There's no mention of it in their docs: https://www.google.com/search?q=site%3Adocs.microsoft.com+%22cloudbase-init%22 https://www.google.com/search?q=site%3Adocs.microsoft.com+%2... New Windows VMs are generally built using PowerShell DSC or a similar native tool.