4 ms·

> * Globally distributed Non-requirement? > * Tailnet as the default network stack That would probably be the first thing I look to rip out if I ever was to

by znpy 1y ago

> * Globally distributed Non-requirement? > * Tailnet as the default network stack That would probably be the first thing I look to rip out if I ever was to use that. Kubernetes assuming the underlying host only has a single NIC has been a plague for the industry, setting it back ~20 years and penalizing everyone that's not running on the cloud. Thank god there are multiple CNI implementation. Only recently with Multus (https://www.redhat.com/en/blog/demystifying-multus https://www.redhat.com/en/blog/demystifying-multus) some sense seem to be coming back into that part of the infrastructure. > * Multi-tenant from the ground up How would this be any different from kubernetes? > * Bittorrent as the default storage stack Might be interesting, unless you also mean seeding public container images. Egress traffic is crazy expensive.

nine_k 1y ago
> Non-requirement > the first thing I look to rip out This only shows how varied the requirements are across the industry. One size does not fit all, hence multiple materially different solutions spring up. This is only good.
- znpy 1y ago
  > One size does not fit all, hence multiple materially different solutions spring up. Sooo… like what kubernetes does today?
mountainriver 1y ago
>> * Globally distributed >Non-requirement? It is a requirement because you can't find GPUs in a single region reliably and Kubernetes doesn't run on multiple regions. >> * Tailnet as the default network stack > That would probably be the first thing I look to rip out if I ever was to use that. This is fair, we find it very useful because it easily scales cross clouds and even bridges them locally. It was the simplest solution we could implement to get those properties, but in no way would we need to be married to it. >> * Multi-tenant from the ground up > How would this be any different from kubernetes? Kuberentes is deeply not multi-tenant, anyone who has tried to make a multi-tenant solution over kube has dealt with this. I've done it at multiple companies now, its a mess. >> * Bittorrent as the default storage stack > Might be interesting, unless you also mean seeding public container images. Egress traffic is crazy expensive. Yeah egress cost is a concern here, but its lazy so you don't pay for it unless you need it. This seemed like the lightest solution to sync data when you do live migrations cross cloud. For instance, I need to move my dataset and ML model to another cloud, or just replicate it there.
- senorrib 1y ago
  > Kubernetes doesn't run on multiple regions How so? You can definitely use annotations in nodes and provision them in different regions.
  - mountainriver 1y ago
    It's not really built to be globally scalable, Mesos was built to be globally scalable. I don't know of any major hosted kubernetes provider that does this.
  - znpy 1y ago
    The user you're responding to probably doesn't really understand how kubernetes works, and the trade-offs that the etcd has taken to allow in order to be "distributed". But you're right. You launch node pretty much anywhere, as long as you have network connectivity (and you don't even need full network connectivity, a couple of tcp ports open are enough). It's not really recommended (due to latency), but you can also run the control-plane across different regions.
    - mountainriver 1y ago
      I've contributed to Kubernetes core and worked for its creators for some time. I really love how crappy the k8s die hards treat people.
stackskipton 1y ago
What is use case for multiple NICs outside bonding for hardware failure? Every time I’ve had multiple NICs on a server with different IPs, I’ve regretted it.
- mdaniel 1y ago
  I'd guess management access, or the old school way of doing vLANs. Kubernetes offers Network Policies to solve the risk of untrusted workloads in the cluster accessing both pods and ports on pods that they shouldn't https://kubernetes.io/docs/concepts/services-networking/network-policies/#networkpolicy-resource https://kubernetes.io/docs/concepts/services-networking/netw... Network Policies are also defense in depth, since another Pod would need to know its sibling Pod's name or IP to reach it directly, the correct boundary for such things is not to expose management toys in the workload's Service, rather create a separate Service that just exposes those management ports Akin to: interface Awesome { String getFavoriteColor(); } interface Management { void setFavoriteColor(String value); } class MyPod implements Awesome, Management {} but then only make either Awesome, or Management, available to the consumers of each behavior
- znpy 1y ago
  A nic dedicated to SAN traffic, for example. People being serious about networked storage don’t run their storage network i/o on the same nic where they serve traffic.

by znpy 1y ago

nine_k 1y ago

> Non-requirement > the first thing I look to rip out This only shows how varied the requirements are across the industry. One size does not fit all, hence multiple materially different solutions spring up. This is only good.

znpy 1y ago
> One size does not fit all, hence multiple materially different solutions spring up. Sooo… like what kubernetes does today?

mountainriver 1y ago

>> * Globally distributed >Non-requirement? It is a requirement because you can't find GPUs in a single region reliably and Kubernetes doesn't run on multiple regions. >> * Tailnet as the default network stack > That would probably be the first thing I look to rip out if I ever was to use that. This is fair, we find it very useful because it easily scales cross clouds and even bridges them locally. It was the simplest solution we could implement to get those properties, but in no way would we need to be married to it. >> * Multi-tenant from the ground up > How would this be any different from kubernetes? Kuberentes is deeply not multi-tenant, anyone who has tried to make a multi-tenant solution over kube has dealt with this. I've done it at multiple companies now, its a mess. >> * Bittorrent as the default storage stack > Might be interesting, unless you also mean seeding public container images. Egress traffic is crazy expensive. Yeah egress cost is a concern here, but its lazy so you don't pay for it unless you need it. This seemed like the lightest solution to sync data when you do live migrations cross cloud. For instance, I need to move my dataset and ML model to another cloud, or just replicate it there.

senorrib 1y ago
> Kubernetes doesn't run on multiple regions How so? You can definitely use annotations in nodes and provision them in different regions.
- mountainriver 1y ago
  It's not really built to be globally scalable, Mesos was built to be globally scalable. I don't know of any major hosted kubernetes provider that does this.
- znpy 1y ago
  The user you're responding to probably doesn't really understand how kubernetes works, and the trade-offs that the etcd has taken to allow in order to be "distributed". But you're right. You launch node pretty much anywhere, as long as you have network connectivity (and you don't even need full network connectivity, a couple of tcp ports open are enough). It's not really recommended (due to latency), but you can also run the control-plane across different regions.
  - mountainriver 1y ago
    I've contributed to Kubernetes core and worked for its creators for some time. I really love how crappy the k8s die hards treat people.

stackskipton 1y ago

What is use case for multiple NICs outside bonding for hardware failure? Every time I’ve had multiple NICs on a server with different IPs, I’ve regretted it.

mdaniel 1y ago
I'd guess management access, or the old school way of doing vLANs. Kubernetes offers Network Policies to solve the risk of untrusted workloads in the cluster accessing both pods and ports on pods that they shouldn't https://kubernetes.io/docs/concepts/services-networking/network-policies/#networkpolicy-resource https://kubernetes.io/docs/concepts/services-networking/netw... Network Policies are also defense in depth, since another Pod would need to know its sibling Pod's name or IP to reach it directly, the correct boundary for such things is not to expose management toys in the workload's Service, rather create a separate Service that just exposes those management ports Akin to: interface Awesome { String getFavoriteColor(); } interface Management { void setFavoriteColor(String value); } class MyPod implements Awesome, Management {} but then only make either Awesome, or Management, available to the consumers of each behavior
znpy 1y ago
A nic dedicated to SAN traffic, for example. People being serious about networked storage don’t run their storage network i/o on the same nic where they serve traffic.