3 ms·

While I see what you're getting at, I find the comment funny after the first thing the article leads with is that certain things are a pain in Kubernetes.

beeboobaa3 2y ago
It's a pain because kubernetes is designed to run multiple workloads on multiple servers. So if you want to access the VPN from some kubernetes containers you're going to have to figure something out. But nothing is stopping you from just joining all your hosts into the VPN, just like a traditional deployment. Or set it up on your network gateway. This would make it available to all of your containers. Great. You're done. But if that's not what you want, you'll need to figure something out.
- Etheryte 2y ago
  I'm not sure if I follow. You said Kubernetes is great for building any kind of a platform, but then when someone wants controlled access to a VPN it suddenly turns into a no, not like that? Giving only certain parts of your architecture access to certain capabilities is far from a niche use case.
  - themgt 2y ago
    Giving only certain parts of your architecture access to certain capabilities is far from a niche use case What is the "normal" best practice here then? I would just spin up multiple single-node k3s VM clusters and hook the AI k3s VM to the VPN and the others not.
  - beeboobaa3 2y ago
    > You said Kubernetes is great for building any kind of a platform I didn't, that was someone else. I wouldn't say any kind of platform, but it is a great foundation for many platforms. > it suddenly turns into a no, not like that Not at all. Read my post and the OP again! Several valid solutions have been offered, some that work out of the box, some that require a little tinkering. If you want to do a traditional VM deployment you'd segment your workloads per hosts and put some hosts in the VPN. Sure. Cool. You can do exactly the same with kubernetes with node pools. Only when you want to have some workloads that have access and some that don't running on the same host machine you might need to do some tinkering. Just like you'd have to do otherwise. Kubernetes really changes nothing, other than giving you ways to deal with it. Of course there's plenty of ready built solutions for this you can just plug in, too. Search for Service Mesh.
nonameiguess 2y ago
Sort of. What the author is trying to do is actually quite easy if the entire setup had been Kubernetes. k3s allows you to connect nodes to your cluster over a Tailscale VPN. If you do that, you don't need to expose your remote AI server to any network at all except the Kubernetes internal network. I'm guessing fly.io doesn't just give you bare servers you can run k3s on, though. The only real difficulty here is hooking up totally different abstraction engines that aren't designed to work with each other. Putting the blame specifically on one of them and not the other doesn't make sense.
jauntywundrkind 2y ago
The pluggable storage (CSI) and networking (CNI) aspects of Kube tend to be a little limiting, in that a cluster generally only expects to have one at a time running. There is Multus, which is extremely well regarded CNI that enables mixing and matching different CNI Plugins. It works great, but it does involve diving into an additional level of complexity, and it can be exciting/unstable building the setup. Maybe the author didn't have rights over the cluster to do this or interest in mucking about deeply in this admittedly complex/highly capable system was limited. Or maybe just using an adding like this was appealing as non disruptive! This seems like a pretty creative & direct way to extend what they already had. There are good well tread options for higher power, for mixed modes of networking, but this strikes me as a nice way to add more without having to revamp the base cluster, which I found super cool & direct. https://github.com/k8snetworkplumbingwg/multus-cni https://github.com/k8snetworkplumbingwg/multus-cni

beeboobaa3 2y ago

It's a pain because kubernetes is designed to run multiple workloads on multiple servers. So if you want to access the VPN from some kubernetes containers you're going to have to figure something out. But nothing is stopping you from just joining all your hosts into the VPN, just like a traditional deployment. Or set it up on your network gateway. This would make it available to all of your containers. Great. You're done. But if that's not what you want, you'll need to figure something out.

Etheryte 2y ago
I'm not sure if I follow. You said Kubernetes is great for building any kind of a platform, but then when someone wants controlled access to a VPN it suddenly turns into a no, not like that? Giving only certain parts of your architecture access to certain capabilities is far from a niche use case.
- themgt 2y ago
  Giving only certain parts of your architecture access to certain capabilities is far from a niche use case What is the "normal" best practice here then? I would just spin up multiple single-node k3s VM clusters and hook the AI k3s VM to the VPN and the others not.
- beeboobaa3 2y ago
  > You said Kubernetes is great for building any kind of a platform I didn't, that was someone else. I wouldn't say any kind of platform, but it is a great foundation for many platforms. > it suddenly turns into a no, not like that Not at all. Read my post and the OP again! Several valid solutions have been offered, some that work out of the box, some that require a little tinkering. If you want to do a traditional VM deployment you'd segment your workloads per hosts and put some hosts in the VPN. Sure. Cool. You can do exactly the same with kubernetes with node pools. Only when you want to have some workloads that have access and some that don't running on the same host machine you might need to do some tinkering. Just like you'd have to do otherwise. Kubernetes really changes nothing, other than giving you ways to deal with it. Of course there's plenty of ready built solutions for this you can just plug in, too. Search for Service Mesh.

nonameiguess 2y ago

Sort of. What the author is trying to do is actually quite easy if the entire setup had been Kubernetes. k3s allows you to connect nodes to your cluster over a Tailscale VPN. If you do that, you don't need to expose your remote AI server to any network at all except the Kubernetes internal network. I'm guessing fly.io doesn't just give you bare servers you can run k3s on, though. The only real difficulty here is hooking up totally different abstraction engines that aren't designed to work with each other. Putting the blame specifically on one of them and not the other doesn't make sense.

jauntywundrkind 2y ago

The pluggable storage (CSI) and networking (CNI) aspects of Kube tend to be a little limiting, in that a cluster generally only expects to have one at a time running. There is Multus, which is extremely well regarded CNI that enables mixing and matching different CNI Plugins. It works great, but it does involve diving into an additional level of complexity, and it can be exciting/unstable building the setup. Maybe the author didn't have rights over the cluster to do this or interest in mucking about deeply in this admittedly complex/highly capable system was limited. Or maybe just using an adding like this was appealing as non disruptive! This seems like a pretty creative & direct way to extend what they already had. There are good well tread options for higher power, for mixed modes of networking, but this strikes me as a nice way to add more without having to revamp the base cluster, which I found super cool & direct. https://github.com/k8snetworkplumbingwg/multus-cni https://github.com/k8snetworkplumbingwg/multus-cni