5 ms·

We have started working on a sort of Kubernetes 2.0 with https://github.com/agentsea/nebulous https://github.com/agentsea/nebulous -- still pre-alpha Things we

We have started working on a sort of Kubernetes 2.0 with https://github.com/agentsea/nebulous https://github.com/agentsea/nebulous -- still pre-alpha Things we are aiming to improve: * Globally distributed * Lightweight, can easily run as a single binary on your laptop while still scaling to thousands of nodes in the cloud. * Tailnet as the default network stack * Bittorrent as the default storage stack * Multi-tenant from the ground up * Live migration as a first class citizen Most of these needs were born out of building modern machine learning products, and the subsequent GPU scarcity. With ML taking over the world though this may be the norm soon.

hhh 1y ago
Wow… Cool stuff, the live migration is very interesting. We do autoscaling across clusters across clouds right now based on pricing, but actual live migration is a different beast
znpy 1y ago
> * Globally distributed Non-requirement? > * Tailnet as the default network stack That would probably be the first thing I look to rip out if I ever was to use that. Kubernetes assuming the underlying host only has a single NIC has been a plague for the industry, setting it back ~20 years and penalizing everyone that's not running on the cloud. Thank god there are multiple CNI implementation. Only recently with Multus (https://www.redhat.com/en/blog/demystifying-multus https://www.redhat.com/en/blog/demystifying-multus) some sense seem to be coming back into that part of the infrastructure. > * Multi-tenant from the ground up How would this be any different from kubernetes? > * Bittorrent as the default storage stack Might be interesting, unless you also mean seeding public container images. Egress traffic is crazy expensive.
- nine_k 1y ago
  > Non-requirement > the first thing I look to rip out This only shows how varied the requirements are across the industry. One size does not fit all, hence multiple materially different solutions spring up. This is only good.
  - znpy 1y ago
    > One size does not fit all, hence multiple materially different solutions spring up. Sooo… like what kubernetes does today?
- mountainriver 1y ago
  >> * Globally distributed >Non-requirement? It is a requirement because you can't find GPUs in a single region reliably and Kubernetes doesn't run on multiple regions. >> * Tailnet as the default network stack > That would probably be the first thing I look to rip out if I ever was to use that. This is fair, we find it very useful because it easily scales cross clouds and even bridges them locally. It was the simplest solution we could implement to get those properties, but in no way would we need to be married to it. >> * Multi-tenant from the ground up > How would this be any different from kubernetes? Kuberentes is deeply not multi-tenant, anyone who has tried to make a multi-tenant solution over kube has dealt with this. I've done it at multiple companies now, its a mess. >> * Bittorrent as the default storage stack > Might be interesting, unless you also mean seeding public container images. Egress traffic is crazy expensive. Yeah egress cost is a concern here, but its lazy so you don't pay for it unless you need it. This seemed like the lightest solution to sync data when you do live migrations cross cloud. For instance, I need to move my dataset and ML model to another cloud, or just replicate it there.
  - senorrib 1y ago
    > Kubernetes doesn't run on multiple regions How so? You can definitely use annotations in nodes and provision them in different regions.
    - mountainriver 1y ago
      It's not really built to be globally scalable, Mesos was built to be globally scalable. I don't know of any major hosted kubernetes provider that does this.
    - znpy 1y ago
      The user you're responding to probably doesn't really understand how kubernetes works, and the trade-offs that the etcd has taken to allow in order to be "distributed". But you're right. You launch node pretty much anywhere, as long as you have network connectivity (and you don't even need full network connectivity, a couple of tcp ports open are enough). It's not really recommended (due to latency), but you can also run the control-plane across different regions.
- stackskipton 1y ago
  What is use case for multiple NICs outside bonding for hardware failure? Every time I’ve had multiple NICs on a server with different IPs, I’ve regretted it.
  - mdaniel 1y ago
    I'd guess management access, or the old school way of doing vLANs. Kubernetes offers Network Policies to solve the risk of untrusted workloads in the cluster accessing both pods and ports on pods that they shouldn't https://kubernetes.io/docs/concepts/services-networking/network-policies/#networkpolicy-resource https://kubernetes.io/docs/concepts/services-networking/netw... Network Policies are also defense in depth, since another Pod would need to know its sibling Pod's name or IP to reach it directly, the correct boundary for such things is not to expose management toys in the workload's Service, rather create a separate Service that just exposes those management ports Akin to: interface Awesome { String getFavoriteColor(); } interface Management { void setFavoriteColor(String value); } class MyPod implements Awesome, Management {} but then only make either Awesome, or Management, available to the consumers of each behavior
  - znpy 1y ago
    A nic dedicated to SAN traffic, for example. People being serious about networked storage don’t run their storage network i/o on the same nic where they serve traffic.
Thaxll 1y ago
This is not Kubernetes, this a custom made solution to run GPU.
- nine_k 1y ago
  Since it still can consume Kubernetes manifests, it's of interest for k8s practitioners. Since k8s manifests are a language, there can be multiple implementations of it, and multiple dialects will necessarily spring up.
- mountainriver 1y ago
  Which is the future of everything and Kuberentes does a very bad job at
  - mdaniel 1y ago
    You stopped typing; what does Kubernetes do a bad job at with relation to scheduling workloads that declare they need at least 1 GPU resource but should be limited to no more than 4 GPU resources on a given Node? https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/#:~:text=you%20can%20consume%20these%20gpus%20from%20your%20containers%20by%20requesting%20the%20custom%20gpu%20resource https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus...
    - mountainriver 1y ago
      Because they aren't consistently available in one region, and Kubernetes can't scale globally. If you don't believe me, go try to train LLMs on Kubernetes with on demand resources, good luck!
      - mdaniel 1y ago
        Oh, ok, sorry, I didn't realize that we were conflating Kubernetes, a container orchestration control plane, with your favorite cloud provider not putting in enough GPU hardware orders per region. My fault
mdaniel 1y ago
heh, I think you didn't read the room given this directory https://github.com/agentsea/nebulous/blob/v0.1.88/deploy/charts/nebulous/Chart.yaml https://github.com/agentsea/nebulous/blob/v0.1.88/deploy/cha... Also, ohgawd please never ever do this ever ohgawd https://github.com/agentsea/nebulous/blob/v0.1.88/deploy/charts/nebulous/templates/provider_aws.yaml#L12-L13 https://github.com/agentsea/nebulous/blob/v0.1.88/deploy/cha...
- mountainriver 1y ago
  Why not? We can run on Kube and extend it to multi-region when needed, or we can run on any VM as a single binary, or just your laptop. If you mean Helm, yeah I hate it but it is the most common standard. Also not sure what you mean by the secret, that is secure.
  - mdaniel 1y ago
    Secure from what, friend? It's a credential leak waiting to happen, to say nothing of the need to now manage IAM Users in AWS. That is the 2000s way of authenticating with AWS, and reminds me of people who still use passwords for ssh. Sure, it works fine, until some employee leaves and takes the root password with them
    - mountainriver 1y ago
      Secure credentials are a leak waiting to happen? Scope them, encrypt them and they are secure. We also support workload identity in a lot of places but often just using API keys is simpler

hhh 1y ago

Wow… Cool stuff, the live migration is very interesting. We do autoscaling across clusters across clouds right now based on pricing, but actual live migration is a different beast

znpy 1y ago

> * Globally distributed Non-requirement? > * Tailnet as the default network stack That would probably be the first thing I look to rip out if I ever was to use that. Kubernetes assuming the underlying host only has a single NIC has been a plague for the industry, setting it back ~20 years and penalizing everyone that's not running on the cloud. Thank god there are multiple CNI implementation. Only recently with Multus (https://www.redhat.com/en/blog/demystifying-multus https://www.redhat.com/en/blog/demystifying-multus) some sense seem to be coming back into that part of the infrastructure. > * Multi-tenant from the ground up How would this be any different from kubernetes? > * Bittorrent as the default storage stack Might be interesting, unless you also mean seeding public container images. Egress traffic is crazy expensive.

nine_k 1y ago
> Non-requirement > the first thing I look to rip out This only shows how varied the requirements are across the industry. One size does not fit all, hence multiple materially different solutions spring up. This is only good.
- znpy 1y ago
  > One size does not fit all, hence multiple materially different solutions spring up. Sooo… like what kubernetes does today?
mountainriver 1y ago
>> * Globally distributed >Non-requirement? It is a requirement because you can't find GPUs in a single region reliably and Kubernetes doesn't run on multiple regions. >> * Tailnet as the default network stack > That would probably be the first thing I look to rip out if I ever was to use that. This is fair, we find it very useful because it easily scales cross clouds and even bridges them locally. It was the simplest solution we could implement to get those properties, but in no way would we need to be married to it. >> * Multi-tenant from the ground up > How would this be any different from kubernetes? Kuberentes is deeply not multi-tenant, anyone who has tried to make a multi-tenant solution over kube has dealt with this. I've done it at multiple companies now, its a mess. >> * Bittorrent as the default storage stack > Might be interesting, unless you also mean seeding public container images. Egress traffic is crazy expensive. Yeah egress cost is a concern here, but its lazy so you don't pay for it unless you need it. This seemed like the lightest solution to sync data when you do live migrations cross cloud. For instance, I need to move my dataset and ML model to another cloud, or just replicate it there.
- senorrib 1y ago
  > Kubernetes doesn't run on multiple regions How so? You can definitely use annotations in nodes and provision them in different regions.
  - mountainriver 1y ago
    It's not really built to be globally scalable, Mesos was built to be globally scalable. I don't know of any major hosted kubernetes provider that does this.
  - znpy 1y ago
    The user you're responding to probably doesn't really understand how kubernetes works, and the trade-offs that the etcd has taken to allow in order to be "distributed". But you're right. You launch node pretty much anywhere, as long as you have network connectivity (and you don't even need full network connectivity, a couple of tcp ports open are enough). It's not really recommended (due to latency), but you can also run the control-plane across different regions.
stackskipton 1y ago
What is use case for multiple NICs outside bonding for hardware failure? Every time I’ve had multiple NICs on a server with different IPs, I’ve regretted it.
- mdaniel 1y ago
  I'd guess management access, or the old school way of doing vLANs. Kubernetes offers Network Policies to solve the risk of untrusted workloads in the cluster accessing both pods and ports on pods that they shouldn't https://kubernetes.io/docs/concepts/services-networking/network-policies/#networkpolicy-resource https://kubernetes.io/docs/concepts/services-networking/netw... Network Policies are also defense in depth, since another Pod would need to know its sibling Pod's name or IP to reach it directly, the correct boundary for such things is not to expose management toys in the workload's Service, rather create a separate Service that just exposes those management ports Akin to: interface Awesome { String getFavoriteColor(); } interface Management { void setFavoriteColor(String value); } class MyPod implements Awesome, Management {} but then only make either Awesome, or Management, available to the consumers of each behavior
- znpy 1y ago
  A nic dedicated to SAN traffic, for example. People being serious about networked storage don’t run their storage network i/o on the same nic where they serve traffic.

Thaxll 1y ago

This is not Kubernetes, this a custom made solution to run GPU.

nine_k 1y ago
Since it still can consume Kubernetes manifests, it's of interest for k8s practitioners. Since k8s manifests are a language, there can be multiple implementations of it, and multiple dialects will necessarily spring up.
mountainriver 1y ago
Which is the future of everything and Kuberentes does a very bad job at
- mdaniel 1y ago
  You stopped typing; what does Kubernetes do a bad job at with relation to scheduling workloads that declare they need at least 1 GPU resource but should be limited to no more than 4 GPU resources on a given Node? https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/#:~:text=you%20can%20consume%20these%20gpus%20from%20your%20containers%20by%20requesting%20the%20custom%20gpu%20resource https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus...
  - mountainriver 1y ago
    Because they aren't consistently available in one region, and Kubernetes can't scale globally. If you don't believe me, go try to train LLMs on Kubernetes with on demand resources, good luck!
    - mdaniel 1y ago
      Oh, ok, sorry, I didn't realize that we were conflating Kubernetes, a container orchestration control plane, with your favorite cloud provider not putting in enough GPU hardware orders per region. My fault

mdaniel 1y ago

heh, I think you didn't read the room given this directory https://github.com/agentsea/nebulous/blob/v0.1.88/deploy/charts/nebulous/Chart.yaml https://github.com/agentsea/nebulous/blob/v0.1.88/deploy/cha... Also, ohgawd please never ever do this ever ohgawd https://github.com/agentsea/nebulous/blob/v0.1.88/deploy/charts/nebulous/templates/provider_aws.yaml#L12-L13 https://github.com/agentsea/nebulous/blob/v0.1.88/deploy/cha...

mountainriver 1y ago
Why not? We can run on Kube and extend it to multi-region when needed, or we can run on any VM as a single binary, or just your laptop. If you mean Helm, yeah I hate it but it is the most common standard. Also not sure what you mean by the secret, that is secure.
- mdaniel 1y ago
  Secure from what, friend? It's a credential leak waiting to happen, to say nothing of the need to now manage IAM Users in AWS. That is the 2000s way of authenticating with AWS, and reminds me of people who still use passwords for ssh. Sure, it works fine, until some employee leaves and takes the root password with them
  - mountainriver 1y ago
    Secure credentials are a leak waiting to happen? Scope them, encrypt them and they are secure. We also support workload identity in a lot of places but often just using API keys is simpler