7 ms·

Virtual Kubernetes

150 points by alexellisuk 7y ago

lpasselin 7y ago
It cwould B useful to run multiple clusters for home/work but using a single cluster. I would like to be able to let my clients start jobs and use their private cluster and append their machines to it, running in my cluster. Is there a way to do this with kubernetes currently?
colemickens 7y ago
See also: SAP's Gardener. Their website reminds me of a spam landing page more than a valuable, robust project, so I'll link an official Kubernetes blog post about it: https://kubernetes.io/blog/2018/05/17/gardener/ https://kubernetes.io/blog/2018/05/17/gardener/ EDIT: per the replies, it does look like a fairly different focus, my apologies, just got excited when I saw the virtualized control plane. (To save you reading the blog, Gardener launches a control plane in an existing cluster to manage workloads on a different, dedicated set of worker nodes.) It will be fun to think about k3v and multi-tenancy.
- p932 7y ago
  Gardener project focus is orchestration of multiple Kubernetes clusters on IaaS cloud providers, whereas k3v focus on running a dedicated control plane over an existing Kubernetes cluster. Some overlaps but different project's goals.
  - deleted 7y ago
    [deleted]
  - deleted 7y ago
    [deleted]
  - cwyers 7y ago
    Yeah, Gardener looks like it's meant to address one tenant on multiple k8s clusters, and k3v looks like it's meant to address multitenancy in a single k8s cluster.
gtirloni 7y ago
Is it fair to call it an k8s API server proxy?
steve76 7y ago
I never been a fan. Too much ingress/egress garbage. What port is where, and such and such. Hey! I got an idea! Let's turn all this against them! A simple npm which "abstracts" the freely available storage space of Facebook, instagram, ... ... .... discord. You could encrypt plain text, and then just have netlify build read it and write it in your app! :-0 !!! Then just interface with a netlify bot server side, with write permissions !!! :-0 !!! OMG ITS LIKE BITCOIN OR SOMETHING!!! Happy Sunday! -J-
PopeDotNinja 7y ago
It'd be an adventure debugging that...
- MuffinFlavored 7y ago
  what are some common k8s debugging rabbit holes that people have to go down on the day to day?
  - PopeDotNinja 7y ago
    I'm not a K8s wizard, but have a little experience with it. I found there to be a steep learning curve with simultaneously learning RBAC, side cards, Helm, YAML templating, storage, logging, operators, service discovery, etc. There's so much hype around Kubernetes that it can be hard to sift through the noise to figure out what you really need to know. I found many people wanted to sell you on how "how all you have to do is...", but before you even get a basic service deployed, you can easily end up with a complex problem that is unique to your situation. And unique can mean you ain't gonna find an answer to your problem on the Google. That becomes even more of an adventure when you're trying to do something any a little non-standard because K8s doesn't expose an API endpoint to solve a problem you have. Now throw slicing up one cluster into N virtual clusters and... debugging that sounds like an adventure.
    - pojzon 7y ago
      Sorry for hijack, May I ask about the sources you would propose to go through for someone starting out with Kubernetes ? I was able to (with examples) create a simple K8s cluster with dashboard and Im learning now about external dns. What else would be good to get to after ?
      - PopeDotNinja 7y ago
        I mostly learned from the K8s docs and brute force trial & error. My first recommendation would be to learn Kubernetes on the platform to which you'll be deploying, if that is possible & practical. For example, you can create a lot of work for yourself trying to debug the differences between Docker, Minikube, and Google Kubernetes Engine. Every provider supports Kubernetes in a slightly different way (in my experience). My 2nd recommendation would be to put in as much time as you can learning to read logs/errors/messages. You'll learn a lot about your cluster(s) and how it is laid out by tracking down the errors the system is generating. They aren't always going to be where you expect them to be. My last recommendation would be to not allow yourself to become intimidated by people who sound like they know more than yourself. The K8s hype machine is in full growth mode, and while K8s is very cool in many ways, there's a pretty darn good chance that someone who insists "all you gotta do is..." doesn't actually know what they are talking about and/or how what they understand maps to the nuances of your particular challenge. Be prepared to grind on hard problems. Clusters of containers are still very much in like sailing to The New World. Few know it all that well, and those who do probably don't work with you :)
dahfizz 7y ago
Why? Kubernetes is already a virtualization layer on top of a virtualization layer. Do we need more virtualization layers? All it does is add complexity and degrade performance.
polskibus 7y ago
What's the story of multitenancy in Kubernetes? Is it true that it does not provide sufficient isolation between services by default? Is this something that could be handled by configuration only or is it a bigger issue with Kubernetes design? What's the best way to make Kubernetes the layer that isolates tenants in a multitenant platform?
- gtirloni 7y ago
  What's sufficient isolation? It has pods, pod security policies, network policies, maybe even hypervisor instead of Docker, etc. It's pretty much built for multitenancy.
- ptrincr 7y ago
  From a service point of view, you should be able to provide isolation between services using namespaces, istio and Calico. You can read about "Soft multi-tenancy" here: https://istio.io/blog/2018/soft-multitenancy/ https://istio.io/blog/2018/soft-multitenancy/ That works well to isolate services/apps/teams from each other, but also allows services to be shared where required. I think if you wanted true multitenancy it would be simpler to spin up a second cluster, especially if you are in gcp and controlling everything using a infrastructure as code solution such as terraform.
- nielsole 7y ago
  File descriptors and inotify watches can still be exhausted. In somewhat older versions also pids. If you allocate memory fast enough(if you're a best effort pod), oomkiller instead of kubelet evictions happen, which can be kind of unpredictable. Similarly you can fill up storage. You don't want to be colocated with bad pods.
- meddlepal 7y ago
  It provides soft multi tenancy via namespaces. You can then use NetworkPolicy if the CNI supports it to manage what other stuff can communicate with the Pods in that namespace. You can also schedule Pods on Nodes that are for a specific purpose (eg only used by a specific team or product) by using various scheduler stuff like taints and affinity/anti-affinity. Or you can write a custom scheduler. In general I feel like all the tools are there in Kubernetes to build a good multitenant experience but it's also just easier to run a number of clusters, especially if you're cloud hosting where the control plane is usually free and you pay only for the workers. I would though ask, what is sufficient isolation?
unfunco 7y ago
> This allows one to take one physical Kubernetes cluster and chop it up into smaller virtual clusters. What is the benefit, or difference of this over namespaces in Kubernetes? Namespaces provide resource quotas, and they work with RBAC too, the benefits listed in the README and their descriptions, they sound exactly what namespaces already provide.
- jayvanguard 7y ago
  That was my question as well.
- sz4kerto 7y ago
  Possibly: testing and learning environments.
- rtempaccount1 7y ago
  (IMO) this isn't intended to be a product, it's an experiment in different ways to run Kubernetes workloads
  - alexellisuk 7y ago
    k3s is on the way to becoming "production-ready", this appears to use k3s in the code. Perhaps it could be once it's proven out as something the community can use?
- p932 7y ago
  Currently if you have only access to a namepsace on a shared Kubernetes cluster without cluster wide admin control you won't be able to: - Create cluster wide RBAC ClusterRole or ClusterRoleBinding - Create or get access to cluster-scoped resources (nodes, CRD) - Use custom webhooks for example using sidecar injection Many of the things that complex Kubernetes deployments are doing nowadays.
  - alexellisuk 7y ago
    The point is that it's unsafe to allow tenants ClusterRole / admin on a shared cluster, but this is needed for many CRDs and Operators. The Operator pattern is getting more and more popular, and most of then need ClusterRole. As the service provider (internal team, or SaaS provider), this is a liability. The aim, from reading the README.md, is to provide the ability for each tenant to be ClusterRole / admin within their own cluster, hosted in a larger real cluster. Jessie Frazelle has talked about this before, I'm not sure if this is the exact blog link, but it's related: Kubernetes in Kubernetes - https://blog.jessfraz.com/post/hard-multi-tenancy-in-kubernetes/ https://blog.jessfraz.com/post/hard-multi-tenancy-in-kuberne...
fen4o 7y ago
For anyone looking for K8S multitenancy - here is an excellent talk by Alibaba / Alicloud about their approach: - https://www.youtube.com/watch?v=ptOLV2wjQUw https://www.youtube.com/watch?v=ptOLV2wjQUw - https://static.sched.com/hosted_files/kccnceu19/3f/kubecon-speak.pdf https://static.sched.com/hosted_files/kccnceu19/3f/kubecon-s...
- erulabs 7y ago
  I've just quit my job at Stripe and founded a startup to tackle K8S hard multi-tenancy, particularly for companies who want to "build a cloud" for their customers but aren't sure where to start. If anyone is interested please feel free to drop me an email! https://kubesail.com https://kubesail.com
  - sieabahlpark 7y ago
    Wouldn't this just be a more expensive aws since you're more than likely be hosted on a major cloud provider?
    - erulabs 7y ago
      Yes in theory, no in practice. Our public shared tier (Heroku-like offering) is on AWS for now - but we have a few advantages in that we can run reserved spot instances and use far less overhead than we would with virtual machines. We will more than likely move this to dedicated hardware eventually since we use almost nothing AWS has to offer other than compute, network and NVMe. Our free tier is better than AWS's ever will be, because although we might offer fewer resources (CPU/memory), our offer is automatically highly-available out of the box, and you share a server with _much_ better network and diskIO than AWS can offer, which means different but typically similar performance. We also don't up charge for services like Postgres and Redis (they use the same resources as any other "app" or container). Also we have an open API (kubernetes!) you can use on your laptop or dedicated gear! But the more important "no in practice" is for companies who offer some value add - say for example Intragram wanted to allow you to run some code that manipulated your images server-side (or more realistically, think PagerDuty allowing custom web-hooks from their servers based on code you write on their site). Hopefully one day large companies that need a public cloud but cannot use Amazon for Reasons(tm) also see us as an option as well. One final "no in practice" is due to our tool's ultimate goal to be to nicely encourage developers to build cloud native apps - and we believe high-available applications are actually "cheaper" than commodity-priced single-point-of-failure applications. See any website you use your credit card on that crashes all the time. I'd like to think of it as "isn't this just a better Heroku?" instead of a more expensive AWS :P
alexellisuk 7y ago
It's also worth looking at the work of Pau and Angel from k8spin - https://k8spin.cloud https://k8spin.cloud They are trying to build something like OpenShift, but self-service and for vanilla Kubernetes.
- drondin 7y ago
  Thanks Alex! Pau here, please do not hesitate to contact us if you are interested in our platform. Our mission is to allow users to benefit of kubernetes whithin seconds. We take care of ingress, SSL certificates, backups... so you can focus on what really matters. We are going to bill by resources/minutes wich is really interesting for CI/CD where you need a real cluster intead of KinD. We are going live in a few days, join our slack or email me for feedback :)
okop 7y ago
Why is Rancher allowed to shill their crap?

150 points by alexellisuk 7y ago

It cwould B useful to run multiple clusters for home/work but using a single cluster. I would like to be able to let my clients start jobs and use their private cluster and append their machines to it, running in my cluster. Is there a way to do this with kubernetes currently?

colemickens 7y ago

See also: SAP's Gardener. Their website reminds me of a spam landing page more than a valuable, robust project, so I'll link an official Kubernetes blog post about it: https://kubernetes.io/blog/2018/05/17/gardener/ https://kubernetes.io/blog/2018/05/17/gardener/ EDIT: per the replies, it does look like a fairly different focus, my apologies, just got excited when I saw the virtualized control plane. (To save you reading the blog, Gardener launches a control plane in an existing cluster to manage workloads on a different, dedicated set of worker nodes.) It will be fun to think about k3v and multi-tenancy.

p932 7y ago
Gardener project focus is orchestration of multiple Kubernetes clusters on IaaS cloud providers, whereas k3v focus on running a dedicated control plane over an existing Kubernetes cluster. Some overlaps but different project's goals.
- deleted 7y ago
  [deleted]
- deleted 7y ago
  [deleted]
- cwyers 7y ago
  Yeah, Gardener looks like it's meant to address one tenant on multiple k8s clusters, and k3v looks like it's meant to address multitenancy in a single k8s cluster.

gtirloni 7y ago

Is it fair to call it an k8s API server proxy?

steve76 7y ago

I never been a fan. Too much ingress/egress garbage. What port is where, and such and such. Hey! I got an idea! Let's turn all this against them! A simple npm which "abstracts" the freely available storage space of Facebook, instagram, ... ... .... discord. You could encrypt plain text, and then just have netlify build read it and write it in your app! :-0 !!! Then just interface with a netlify bot server side, with write permissions !!! :-0 !!! OMG ITS LIKE BITCOIN OR SOMETHING!!! Happy Sunday! -J-

PopeDotNinja 7y ago

It'd be an adventure debugging that...

MuffinFlavored 7y ago
what are some common k8s debugging rabbit holes that people have to go down on the day to day?
- PopeDotNinja 7y ago
  I'm not a K8s wizard, but have a little experience with it. I found there to be a steep learning curve with simultaneously learning RBAC, side cards, Helm, YAML templating, storage, logging, operators, service discovery, etc. There's so much hype around Kubernetes that it can be hard to sift through the noise to figure out what you really need to know. I found many people wanted to sell you on how "how all you have to do is...", but before you even get a basic service deployed, you can easily end up with a complex problem that is unique to your situation. And unique can mean you ain't gonna find an answer to your problem on the Google. That becomes even more of an adventure when you're trying to do something any a little non-standard because K8s doesn't expose an API endpoint to solve a problem you have. Now throw slicing up one cluster into N virtual clusters and... debugging that sounds like an adventure.
  - pojzon 7y ago
    Sorry for hijack, May I ask about the sources you would propose to go through for someone starting out with Kubernetes ? I was able to (with examples) create a simple K8s cluster with dashboard and Im learning now about external dns. What else would be good to get to after ?
    - PopeDotNinja 7y ago
      I mostly learned from the K8s docs and brute force trial & error. My first recommendation would be to learn Kubernetes on the platform to which you'll be deploying, if that is possible & practical. For example, you can create a lot of work for yourself trying to debug the differences between Docker, Minikube, and Google Kubernetes Engine. Every provider supports Kubernetes in a slightly different way (in my experience). My 2nd recommendation would be to put in as much time as you can learning to read logs/errors/messages. You'll learn a lot about your cluster(s) and how it is laid out by tracking down the errors the system is generating. They aren't always going to be where you expect them to be. My last recommendation would be to not allow yourself to become intimidated by people who sound like they know more than yourself. The K8s hype machine is in full growth mode, and while K8s is very cool in many ways, there's a pretty darn good chance that someone who insists "all you gotta do is..." doesn't actually know what they are talking about and/or how what they understand maps to the nuances of your particular challenge. Be prepared to grind on hard problems. Clusters of containers are still very much in like sailing to The New World. Few know it all that well, and those who do probably don't work with you :)

dahfizz 7y ago

Why? Kubernetes is already a virtualization layer on top of a virtualization layer. Do we need more virtualization layers? All it does is add complexity and degrade performance.

polskibus 7y ago

What's the story of multitenancy in Kubernetes? Is it true that it does not provide sufficient isolation between services by default? Is this something that could be handled by configuration only or is it a bigger issue with Kubernetes design? What's the best way to make Kubernetes the layer that isolates tenants in a multitenant platform?

gtirloni 7y ago
What's sufficient isolation? It has pods, pod security policies, network policies, maybe even hypervisor instead of Docker, etc. It's pretty much built for multitenancy.
ptrincr 7y ago
From a service point of view, you should be able to provide isolation between services using namespaces, istio and Calico. You can read about "Soft multi-tenancy" here: https://istio.io/blog/2018/soft-multitenancy/ https://istio.io/blog/2018/soft-multitenancy/ That works well to isolate services/apps/teams from each other, but also allows services to be shared where required. I think if you wanted true multitenancy it would be simpler to spin up a second cluster, especially if you are in gcp and controlling everything using a infrastructure as code solution such as terraform.
nielsole 7y ago
File descriptors and inotify watches can still be exhausted. In somewhat older versions also pids. If you allocate memory fast enough(if you're a best effort pod), oomkiller instead of kubelet evictions happen, which can be kind of unpredictable. Similarly you can fill up storage. You don't want to be colocated with bad pods.
meddlepal 7y ago
It provides soft multi tenancy via namespaces. You can then use NetworkPolicy if the CNI supports it to manage what other stuff can communicate with the Pods in that namespace. You can also schedule Pods on Nodes that are for a specific purpose (eg only used by a specific team or product) by using various scheduler stuff like taints and affinity/anti-affinity. Or you can write a custom scheduler. In general I feel like all the tools are there in Kubernetes to build a good multitenant experience but it's also just easier to run a number of clusters, especially if you're cloud hosting where the control plane is usually free and you pay only for the workers. I would though ask, what is sufficient isolation?

unfunco 7y ago

> This allows one to take one physical Kubernetes cluster and chop it up into smaller virtual clusters. What is the benefit, or difference of this over namespaces in Kubernetes? Namespaces provide resource quotas, and they work with RBAC too, the benefits listed in the README and their descriptions, they sound exactly what namespaces already provide.

jayvanguard 7y ago
That was my question as well.
sz4kerto 7y ago
Possibly: testing and learning environments.
rtempaccount1 7y ago
(IMO) this isn't intended to be a product, it's an experiment in different ways to run Kubernetes workloads
- alexellisuk 7y ago
  k3s is on the way to becoming "production-ready", this appears to use k3s in the code. Perhaps it could be once it's proven out as something the community can use?
p932 7y ago
Currently if you have only access to a namepsace on a shared Kubernetes cluster without cluster wide admin control you won't be able to: - Create cluster wide RBAC ClusterRole or ClusterRoleBinding - Create or get access to cluster-scoped resources (nodes, CRD) - Use custom webhooks for example using sidecar injection Many of the things that complex Kubernetes deployments are doing nowadays.
- alexellisuk 7y ago
  The point is that it's unsafe to allow tenants ClusterRole / admin on a shared cluster, but this is needed for many CRDs and Operators. The Operator pattern is getting more and more popular, and most of then need ClusterRole. As the service provider (internal team, or SaaS provider), this is a liability. The aim, from reading the README.md, is to provide the ability for each tenant to be ClusterRole / admin within their own cluster, hosted in a larger real cluster. Jessie Frazelle has talked about this before, I'm not sure if this is the exact blog link, but it's related: Kubernetes in Kubernetes - https://blog.jessfraz.com/post/hard-multi-tenancy-in-kubernetes/ https://blog.jessfraz.com/post/hard-multi-tenancy-in-kuberne...

fen4o 7y ago

For anyone looking for K8S multitenancy - here is an excellent talk by Alibaba / Alicloud about their approach: - https://www.youtube.com/watch?v=ptOLV2wjQUw https://www.youtube.com/watch?v=ptOLV2wjQUw - https://static.sched.com/hosted_files/kccnceu19/3f/kubecon-speak.pdf https://static.sched.com/hosted_files/kccnceu19/3f/kubecon-s...

erulabs 7y ago
I've just quit my job at Stripe and founded a startup to tackle K8S hard multi-tenancy, particularly for companies who want to "build a cloud" for their customers but aren't sure where to start. If anyone is interested please feel free to drop me an email! https://kubesail.com https://kubesail.com
- sieabahlpark 7y ago
  Wouldn't this just be a more expensive aws since you're more than likely be hosted on a major cloud provider?
  - erulabs 7y ago
    Yes in theory, no in practice. Our public shared tier (Heroku-like offering) is on AWS for now - but we have a few advantages in that we can run reserved spot instances and use far less overhead than we would with virtual machines. We will more than likely move this to dedicated hardware eventually since we use almost nothing AWS has to offer other than compute, network and NVMe. Our free tier is better than AWS's ever will be, because although we might offer fewer resources (CPU/memory), our offer is automatically highly-available out of the box, and you share a server with _much_ better network and diskIO than AWS can offer, which means different but typically similar performance. We also don't up charge for services like Postgres and Redis (they use the same resources as any other "app" or container). Also we have an open API (kubernetes!) you can use on your laptop or dedicated gear! But the more important "no in practice" is for companies who offer some value add - say for example Intragram wanted to allow you to run some code that manipulated your images server-side (or more realistically, think PagerDuty allowing custom web-hooks from their servers based on code you write on their site). Hopefully one day large companies that need a public cloud but cannot use Amazon for Reasons(tm) also see us as an option as well. One final "no in practice" is due to our tool's ultimate goal to be to nicely encourage developers to build cloud native apps - and we believe high-available applications are actually "cheaper" than commodity-priced single-point-of-failure applications. See any website you use your credit card on that crashes all the time. I'd like to think of it as "isn't this just a better Heroku?" instead of a more expensive AWS :P

alexellisuk 7y ago

It's also worth looking at the work of Pau and Angel from k8spin - https://k8spin.cloud https://k8spin.cloud They are trying to build something like OpenShift, but self-service and for vanilla Kubernetes.

drondin 7y ago
Thanks Alex! Pau here, please do not hesitate to contact us if you are interested in our platform. Our mission is to allow users to benefit of kubernetes whithin seconds. We take care of ingress, SSL certificates, backups... so you can focus on what really matters. We are going to bill by resources/minutes wich is really interesting for CI/CD where you need a real cluster intead of KinD. We are going live in a few days, join our slack or email me for feedback :)

okop 7y ago

Why is Rancher allowed to shill their crap?