5 ms·

This highlights a sorta-weird consequence of this pricing change: suddenly pricing incentivizes you to use namespacing instead of clusters for separating enviro

by lvh 7y ago

This highlights a sorta-weird consequence of this pricing change: suddenly pricing incentivizes you to use namespacing instead of clusters for separating environments. (As a security person: ugh.)

rolleiflex 7y ago
That’s interesting - I think you’re right. We might move our staging cluster into our main production deployment. More likely though, AWS or OpenShift running on bare metal on a beefy ATX tower in the office. We want to have production and staging as close to each other as possible, so this is an additional reason and a p0 flag on reducing the dependency on Google-specific bits of Kubernetes as much as possible, hopefully also useful for our exit strategy as well.
- lazyier 7y ago
  Kubespray works well for me for setting up a bare bones kubernetes cluster for the lab. I'll use helm to install metallb for the load balancer, which you can then tie into whatever egress controller you like to use. For persistent storage a simple NFS server is the bees knees. Works very well and a NFS provisioned is a helm install. Very nice, especially, over 10GbE. Do NOT dismiss NFSv4. It's actually very nice for this sort of thing. I just use a small separate Linux box with software raid on it for that. If you want to have the cluster self-host storage or need high availability then GlusterFS works great, but it's more overhead to manage. Then you just use normal helm install routines to install and setup logging, dashboards, and all that. Openshift is going to be a lot better for people who want to do multi-tenant stuff in a corporate enterprise environment. Like you have different teams of people, each with their own realm of responsibility. Openshift's UI and general approach is pretty good about allowing groups to self-manage without impacting one another. The additional security is a double edged. Fantastic if you need it, but annoying barrier to entry for users if you don't. As far as AWS goes... EKS recently lowered their cost from 20 cents per hour to 10 cents. So costs for the cluster is on par with what Google is charging. Azure doesn't charge for cluster management (yet), IIRC.
  - freedomben 7y ago
    Have you used NFS for persistent storage in prod much? I know people do it, but numerous solutions architects have cautioned against it.
    - yetanotherme 7y ago
      My experience with NFS over the years has taught me to avoid it. Yes, it mostly works. And then every once a while you have a client that either panics or hangs. Despite the versions of Linux, BSD, Solaris, Windows changing over the decades. The server end is usually a lot more stable. But that's of little to no comfort to know that yes, other clients are fine. However, if you can tolerate client side failure then go for it.
  - geerlingguy 7y ago
    (replying to freedomben): NFS has worked fairly well for persistent file storage that doesn't require high performance for reads/writes (e.g. good for media storage for a website with a CDN fronting a lot of traffic, good for some kinds of other data storage). It would be a terrible solution for things like database storage or other high-performance needs (clustering and separate PVs with high IOPS storage would be better here).
    - lazyier 7y ago
      It's good to have multiple options if you want to host databases in the cluster. For example you could use NFS for 90% of the storage needs for logging and sharing files between pods. Then use local storage, FCOE, or iSCSI-backed PVs for databases. If you are doing bare hardware and your requirements for latency are not too stringent then not hosting databases in the cluster is also a good approach. Just used dedicated systems. If you can get state out of the cluster then that makes things easier. All of this depends on a huge number of other factors, of course.
  - lazyier 7y ago
    > Have you used NFS for persistent storage in prod much? I think NFS is heavily underrated. It's a good match for things like hosting VM images on a cluster and for Kubernetes. In the past I really wanted to use things iSCSI for hosting VM images and such things, but I've found that NFS is actually a lot faster for a lot of things. There are complications to NFS, of course, but they haven't caused me problems. I would be happy to use it in production, and have recommended it, but it's not unconditional. It depends on a number of different factors. The only problem with NFS is how do you manage the actual NFS infrastructure? How much experience does your org have with NFS? Do you already have a existing file storage solution in production you can expand and use that with Kubernetes? Like if your organization already has a lot of servers running ZFS, then that is a nice thing to leverage for NFS persistent storage. Since you already have expertise in-house it would be a mistake not to take advantage of it. I wouldn't recommend this approach for people not already doing it, though. If you can afford some sort of enterprise-grade storage appliance that takes care of dedupe, checksums, failovers, and all that happy stuff, then that's great. Use that and it'll solve your problems. Especially if there is some sort of NFS provisoner that Kubernetes supports. The only place were I would say it's a 'Hard No' is if you have some sort of high scalability requirements. Like if you wanted to start some web hosting company or needed to have hundreds of nodes in a cluster. In that case then distributed file systems is what you need... Self-hosted storage aka "Hyper Converged Infrastructure". The cost and overhead of managing these things is then relative small to the size of the cluster and what you are trying to do. It's scary to me to have a cluster self-host storage because storage can use a huge amount of ram and cpu at the worst times. You can go from a happy low-resource cluster, then a node fails or other component takes a shit, and then while everything is recovering and checksum'ng (and lord knows what) the resource usage goes through the roof right during a critical time. The 'perfect storm' scenarios.
- bluhbi 7y ago
  What? Shouldn't you try to make the creation and deletion of your staging cluster cheap instead of moving it to somewhere else? And if that is your central infrastructure, shouldn't it be worth the money? I do get the issue with having cheap and beefy hardware somewhere else, i do that as well, but only for private. My hourly salary spending or wasting time on stuff like that costs the company more than just paying for an additional cluster with the same settings but perhaps with much less Nodes. If more than one person is using it, the multiplication effects for suddenly unproductive people, is much higher. Also that decreases the per head cost.
lokar 7y ago
You can dedicated nodes by namespace, at which point the isolation is pretty strong.
- dharmab 7y ago
  * Assuming you also configure strong RBAC, network isolation and don't let persistent volumes cross-talk
- yoshiat 7y ago
  Nit: we don't recommend dedicated nodes for isolation. PTAL at https://www.youtube.com/watch?v=6rMGRvcjvKc https://www.youtube.com/watch?v=6rMGRvcjvKc And the guidance from GKE is at https://cloud.google.com/kubernetes-engine/docs/best-practices/enterprise-multitenancy https://cloud.google.com/kubernetes-engine/docs/best-practic...
sethvargo 7y ago
As also a security person (:wave:), you can use dedicated node pools and workload identity to isolate workloads in the same cluster.
- lvh 7y ago
  Workload identity is a GCP-specific beta feature for mapping to GCP IAM, right?
  - bluhbi 7y ago
    yes
  - yoshiat 7y ago
    Yes and it's going GA. https://cloud.google.com/blog/products/containers-kubernetes/kubernetes-engine-features-and-guidance-to-help-lock-down-your-containers https://cloud.google.com/blog/products/containers-kubernetes... https://cloud.google.com/kubernetes-engine/docs/best-practices/enterprise-multitenancy https://cloud.google.com/kubernetes-engine/docs/best-practic...
mrbrowning 7y ago
I suspect I'm in the minority on this, but I would love for k8s to have hierarchical namespaces. As much as they add complexity, there are a lot of cases where they're just reifying complexity that's already there, like when deployments are namespaced by environment (e.g. "dev-{service}", "prod-{service}", etc.) and so the hierarchy is already present but flattened into an inaccessible string representation. There are other solutions to this, but they all seem to extract their cost in terms of more manual fleet management.
- aludwin 7y ago
  Hey - I'm a member of the multitenancy working group (wg-multitenancy). We're working on a project called the Hierarchical Namespace Controller (aka HNC - read about it at http://bit.ly/38YYhE0 http://bit.ly/38YYhE0). This tries to add some hierarchical behaviour to K8s without actually modifying k/k, which means we're still forced to have unique names for all namespaces in a cluster - e.g., you still need dev-service and prod-service. But it does add a consistent way to talk about hierarchy, some nice integrations and builtin behaviours. Do you want to mention anything more about what you're hoping to get out of hierarchy? Is it just a management tool, is it for access control, metering/observability, etc...? Thanks, A
  - mrbrowning 7y ago
    Hey, thanks for asking! My interests in it are primarily for quota management -- in my experience, this is inevitably a hierarchical concern, in that you frequently run into the case of wanting to allot a certain cluster-wide quota to a large organizational unit, and similarly subdivide that quota between smaller organizational subunits. Being able to model that hierarchy with namespaces localizes changes more effectively: if you want to increase the larger unit's quota in a flat namespace world, for example, there's no way to talk about that unit's quota except as the sum of all of its constituent namespace quotas.
    - aludwin 7y ago
      Thanks! We're not currently planning on implementing a hierarchical resource quota in HNC, but HNC is trying to define a definition of hierarchy that could certainly be used to create a HRQ. Give me a shout if you're interested in contributing.
outworlder 7y ago
Assuming you can do that, and your system is not using namespacing for its own purposes.
- aludwin 7y ago
  I know kubeflow can use namespaces for its own purposes, but otherwise I thought that was quite rare. Namespaces are intended to be used for exactly this usecase (isolating teams and/or workloads). What kind of system have you seen where this isn't true?
  - lvh 7y ago
    We've seen plenty of examples where people do this. Sometimes it's different teams (e.g. the ML team is namespaced away from the primary customer flow) and sometimes it's for different customers. Really, it sounds like we're in agreement about why this happens, the confusion is just whether or not that normally happens within a company in the course of doing business?
    - aludwin 7y ago
      Gotcha. If you're looking for subnamespacing, HNC offers self-service subnamespace creation, have you looked into that? https://github.com/kubernetes-sigs/multi-tenancy/tree/master/incubator/hnc https://github.com/kubernetes-sigs/multi-tenancy/tree/master... (My apologies if we've chatted about this before in another venue, I'm losing track of whom I've already talked to)
  - bboreham 7y ago
    I use namespaces to isolate names. So I can have a service called memcached in namespace A and also one in namespace B.
yoshiat 7y ago
Yes, namespace alone isn't sufficient for isolation. Would you be able to look at our latest Multi-Tenancy best practices? https://cloud.google.com/kubernetes-engine/docs/best-practices/enterprise-multitenancy https://cloud.google.com/kubernetes-engine/docs/best-practic... It's a living product which comes with Terraform modules. We introduced various features to enable doing Multi-Tenancy as well (and more on their way!)
- aaronblohowiak 7y ago
  I’m sorry if i am reading it wrong, but this guide to multi-tenancy seems to suggest not being multi-tennant and instead running a separate cluster per project. This seems more like scaling single-tenancy than multi-tenant (no bin packing oppty for instance.) or did i read it wrong?
  - yoshiat 7y ago
    Sorry if it was confusing. You need to read into more about how to set up in-cluster multi-tenancy. We do recommend robust configurations for production setup (e.g. dev, staging and production) however you can certainly squash and skip it if not necessary. Thanks for the feedback though. We'll consider adding such notes explicitly.
    - aaronblohowiak 7y ago
      >You need to read into more about how to set up in-cluster multi-tenancy. I am trying to do that. Where would you suggest? Throughout the comments on this post, when people suggest namespace, pod or node level separation you ask them to PTAL and read the link which suggests the single-tenant cluster-per-project approach (that is under the Multi-tenant cluster, confusingly.) The link you sent talks about cluster-per-project, which is not multi-tenancy as I understand it. Perhaps a different name would be less confusing (robust federated cluster administration?)
      - yoshiat 7y ago
        "This guide provides best practices to safely and efficiently set up multiple multi-tenant clusters for an enterprise organization." This "multiple" multi-tenant clusters part isn't coming through. Please do jump into "Securing the cluster" section to cut corners and learn what to do in a single cluster. We're fixing the sections to avoid the confusions. Thanks for the feedback! https://cloud.google.com/kubernetes-engine/docs/best-practices/enterprise-multitenancy#securing_the_cluster https://cloud.google.com/kubernetes-engine/docs/best-practic...
blazespin 7y ago
Or move to minikube and friends. If it's a dev environment you can usually get away with such things.
- atombender 7y ago
  Kubernetes consumes a lot of CPU even when idle, due to the polling design, which makes Minikube is a really poor fit for developer machines. It's well known [1] to sit there eating 20-30% CPU, draining your battery and frustrating your life while doing absolutely nothing. This applies to all the Kubernetes distributions, including Kind and Docker Desktop. Not sure if the same applies to K3s, though. [1] https://github.com/kubernetes/minikube/issues/3207 https://github.com/kubernetes/minikube/issues/3207, https://github.com/docker/for-mac/issues/3065 https://github.com/docker/for-mac/issues/3065, https://github.com/docker/for-mac/issues/3539 https://github.com/docker/for-mac/issues/3539, etc; there must be dozens and dozens of these
  - rcarmo 7y ago
    It does. My k3s setups (like https://github.com/rcarmo/azure-k3s-cluster https://github.com/rcarmo/azure-k3s-cluster) take up nearly 100% of the puny master node I allocate to them, and kill my Raspberry Pi SD cards as well. Swarm, in comparison, is much friendlier (and you can use it for dev/test across multiple machines just fine)

by lvh 7y ago

This highlights a sorta-weird consequence of this pricing change: suddenly pricing incentivizes you to use namespacing instead of clusters for separating environments. (As a security person: ugh.)

rolleiflex 7y ago

That’s interesting - I think you’re right. We might move our staging cluster into our main production deployment. More likely though, AWS or OpenShift running on bare metal on a beefy ATX tower in the office. We want to have production and staging as close to each other as possible, so this is an additional reason and a p0 flag on reducing the dependency on Google-specific bits of Kubernetes as much as possible, hopefully also useful for our exit strategy as well.

lazyier 7y ago
Kubespray works well for me for setting up a bare bones kubernetes cluster for the lab. I'll use helm to install metallb for the load balancer, which you can then tie into whatever egress controller you like to use. For persistent storage a simple NFS server is the bees knees. Works very well and a NFS provisioned is a helm install. Very nice, especially, over 10GbE. Do NOT dismiss NFSv4. It's actually very nice for this sort of thing. I just use a small separate Linux box with software raid on it for that. If you want to have the cluster self-host storage or need high availability then GlusterFS works great, but it's more overhead to manage. Then you just use normal helm install routines to install and setup logging, dashboards, and all that. Openshift is going to be a lot better for people who want to do multi-tenant stuff in a corporate enterprise environment. Like you have different teams of people, each with their own realm of responsibility. Openshift's UI and general approach is pretty good about allowing groups to self-manage without impacting one another. The additional security is a double edged. Fantastic if you need it, but annoying barrier to entry for users if you don't. As far as AWS goes... EKS recently lowered their cost from 20 cents per hour to 10 cents. So costs for the cluster is on par with what Google is charging. Azure doesn't charge for cluster management (yet), IIRC.
- freedomben 7y ago
  Have you used NFS for persistent storage in prod much? I know people do it, but numerous solutions architects have cautioned against it.
  - yetanotherme 7y ago
    My experience with NFS over the years has taught me to avoid it. Yes, it mostly works. And then every once a while you have a client that either panics or hangs. Despite the versions of Linux, BSD, Solaris, Windows changing over the decades. The server end is usually a lot more stable. But that's of little to no comfort to know that yes, other clients are fine. However, if you can tolerate client side failure then go for it.
- geerlingguy 7y ago
  (replying to freedomben): NFS has worked fairly well for persistent file storage that doesn't require high performance for reads/writes (e.g. good for media storage for a website with a CDN fronting a lot of traffic, good for some kinds of other data storage). It would be a terrible solution for things like database storage or other high-performance needs (clustering and separate PVs with high IOPS storage would be better here).
  - lazyier 7y ago
    It's good to have multiple options if you want to host databases in the cluster. For example you could use NFS for 90% of the storage needs for logging and sharing files between pods. Then use local storage, FCOE, or iSCSI-backed PVs for databases. If you are doing bare hardware and your requirements for latency are not too stringent then not hosting databases in the cluster is also a good approach. Just used dedicated systems. If you can get state out of the cluster then that makes things easier. All of this depends on a huge number of other factors, of course.
- lazyier 7y ago
  > Have you used NFS for persistent storage in prod much? I think NFS is heavily underrated. It's a good match for things like hosting VM images on a cluster and for Kubernetes. In the past I really wanted to use things iSCSI for hosting VM images and such things, but I've found that NFS is actually a lot faster for a lot of things. There are complications to NFS, of course, but they haven't caused me problems. I would be happy to use it in production, and have recommended it, but it's not unconditional. It depends on a number of different factors. The only problem with NFS is how do you manage the actual NFS infrastructure? How much experience does your org have with NFS? Do you already have a existing file storage solution in production you can expand and use that with Kubernetes? Like if your organization already has a lot of servers running ZFS, then that is a nice thing to leverage for NFS persistent storage. Since you already have expertise in-house it would be a mistake not to take advantage of it. I wouldn't recommend this approach for people not already doing it, though. If you can afford some sort of enterprise-grade storage appliance that takes care of dedupe, checksums, failovers, and all that happy stuff, then that's great. Use that and it'll solve your problems. Especially if there is some sort of NFS provisoner that Kubernetes supports. The only place were I would say it's a 'Hard No' is if you have some sort of high scalability requirements. Like if you wanted to start some web hosting company or needed to have hundreds of nodes in a cluster. In that case then distributed file systems is what you need... Self-hosted storage aka "Hyper Converged Infrastructure". The cost and overhead of managing these things is then relative small to the size of the cluster and what you are trying to do. It's scary to me to have a cluster self-host storage because storage can use a huge amount of ram and cpu at the worst times. You can go from a happy low-resource cluster, then a node fails or other component takes a shit, and then while everything is recovering and checksum'ng (and lord knows what) the resource usage goes through the roof right during a critical time. The 'perfect storm' scenarios.
bluhbi 7y ago
What? Shouldn't you try to make the creation and deletion of your staging cluster cheap instead of moving it to somewhere else? And if that is your central infrastructure, shouldn't it be worth the money? I do get the issue with having cheap and beefy hardware somewhere else, i do that as well, but only for private. My hourly salary spending or wasting time on stuff like that costs the company more than just paying for an additional cluster with the same settings but perhaps with much less Nodes. If more than one person is using it, the multiplication effects for suddenly unproductive people, is much higher. Also that decreases the per head cost.

lokar 7y ago

You can dedicated nodes by namespace, at which point the isolation is pretty strong.

dharmab 7y ago
* Assuming you also configure strong RBAC, network isolation and don't let persistent volumes cross-talk
yoshiat 7y ago
Nit: we don't recommend dedicated nodes for isolation. PTAL at https://www.youtube.com/watch?v=6rMGRvcjvKc https://www.youtube.com/watch?v=6rMGRvcjvKc And the guidance from GKE is at https://cloud.google.com/kubernetes-engine/docs/best-practices/enterprise-multitenancy https://cloud.google.com/kubernetes-engine/docs/best-practic...

sethvargo 7y ago

As also a security person (:wave:), you can use dedicated node pools and workload identity to isolate workloads in the same cluster.

lvh 7y ago
Workload identity is a GCP-specific beta feature for mapping to GCP IAM, right?
- bluhbi 7y ago
  yes
- yoshiat 7y ago
  Yes and it's going GA. https://cloud.google.com/blog/products/containers-kubernetes/kubernetes-engine-features-and-guidance-to-help-lock-down-your-containers https://cloud.google.com/blog/products/containers-kubernetes... https://cloud.google.com/kubernetes-engine/docs/best-practices/enterprise-multitenancy https://cloud.google.com/kubernetes-engine/docs/best-practic...

mrbrowning 7y ago

I suspect I'm in the minority on this, but I would love for k8s to have hierarchical namespaces. As much as they add complexity, there are a lot of cases where they're just reifying complexity that's already there, like when deployments are namespaced by environment (e.g. "dev-{service}", "prod-{service}", etc.) and so the hierarchy is already present but flattened into an inaccessible string representation. There are other solutions to this, but they all seem to extract their cost in terms of more manual fleet management.

aludwin 7y ago
Hey - I'm a member of the multitenancy working group (wg-multitenancy). We're working on a project called the Hierarchical Namespace Controller (aka HNC - read about it at http://bit.ly/38YYhE0 http://bit.ly/38YYhE0). This tries to add some hierarchical behaviour to K8s without actually modifying k/k, which means we're still forced to have unique names for all namespaces in a cluster - e.g., you still need dev-service and prod-service. But it does add a consistent way to talk about hierarchy, some nice integrations and builtin behaviours. Do you want to mention anything more about what you're hoping to get out of hierarchy? Is it just a management tool, is it for access control, metering/observability, etc...? Thanks, A
- mrbrowning 7y ago
  Hey, thanks for asking! My interests in it are primarily for quota management -- in my experience, this is inevitably a hierarchical concern, in that you frequently run into the case of wanting to allot a certain cluster-wide quota to a large organizational unit, and similarly subdivide that quota between smaller organizational subunits. Being able to model that hierarchy with namespaces localizes changes more effectively: if you want to increase the larger unit's quota in a flat namespace world, for example, there's no way to talk about that unit's quota except as the sum of all of its constituent namespace quotas.
  - aludwin 7y ago
    Thanks! We're not currently planning on implementing a hierarchical resource quota in HNC, but HNC is trying to define a definition of hierarchy that could certainly be used to create a HRQ. Give me a shout if you're interested in contributing.

outworlder 7y ago

Assuming you can do that, and your system is not using namespacing for its own purposes.

aludwin 7y ago
I know kubeflow can use namespaces for its own purposes, but otherwise I thought that was quite rare. Namespaces are intended to be used for exactly this usecase (isolating teams and/or workloads). What kind of system have you seen where this isn't true?
- lvh 7y ago
  We've seen plenty of examples where people do this. Sometimes it's different teams (e.g. the ML team is namespaced away from the primary customer flow) and sometimes it's for different customers. Really, it sounds like we're in agreement about why this happens, the confusion is just whether or not that normally happens within a company in the course of doing business?
  - aludwin 7y ago
    Gotcha. If you're looking for subnamespacing, HNC offers self-service subnamespace creation, have you looked into that? https://github.com/kubernetes-sigs/multi-tenancy/tree/master/incubator/hnc https://github.com/kubernetes-sigs/multi-tenancy/tree/master... (My apologies if we've chatted about this before in another venue, I'm losing track of whom I've already talked to)
- bboreham 7y ago
  I use namespaces to isolate names. So I can have a service called memcached in namespace A and also one in namespace B.

yoshiat 7y ago

Yes, namespace alone isn't sufficient for isolation. Would you be able to look at our latest Multi-Tenancy best practices? https://cloud.google.com/kubernetes-engine/docs/best-practices/enterprise-multitenancy https://cloud.google.com/kubernetes-engine/docs/best-practic... It's a living product which comes with Terraform modules. We introduced various features to enable doing Multi-Tenancy as well (and more on their way!)

aaronblohowiak 7y ago
I’m sorry if i am reading it wrong, but this guide to multi-tenancy seems to suggest not being multi-tennant and instead running a separate cluster per project. This seems more like scaling single-tenancy than multi-tenant (no bin packing oppty for instance.) or did i read it wrong?
- yoshiat 7y ago
  Sorry if it was confusing. You need to read into more about how to set up in-cluster multi-tenancy. We do recommend robust configurations for production setup (e.g. dev, staging and production) however you can certainly squash and skip it if not necessary. Thanks for the feedback though. We'll consider adding such notes explicitly.
  - aaronblohowiak 7y ago
    >You need to read into more about how to set up in-cluster multi-tenancy. I am trying to do that. Where would you suggest? Throughout the comments on this post, when people suggest namespace, pod or node level separation you ask them to PTAL and read the link which suggests the single-tenant cluster-per-project approach (that is under the Multi-tenant cluster, confusingly.) The link you sent talks about cluster-per-project, which is not multi-tenancy as I understand it. Perhaps a different name would be less confusing (robust federated cluster administration?)
    - yoshiat 7y ago
      "This guide provides best practices to safely and efficiently set up multiple multi-tenant clusters for an enterprise organization." This "multiple" multi-tenant clusters part isn't coming through. Please do jump into "Securing the cluster" section to cut corners and learn what to do in a single cluster. We're fixing the sections to avoid the confusions. Thanks for the feedback! https://cloud.google.com/kubernetes-engine/docs/best-practices/enterprise-multitenancy#securing_the_cluster https://cloud.google.com/kubernetes-engine/docs/best-practic...

blazespin 7y ago

Or move to minikube and friends. If it's a dev environment you can usually get away with such things.

atombender 7y ago
Kubernetes consumes a lot of CPU even when idle, due to the polling design, which makes Minikube is a really poor fit for developer machines. It's well known [1] to sit there eating 20-30% CPU, draining your battery and frustrating your life while doing absolutely nothing. This applies to all the Kubernetes distributions, including Kind and Docker Desktop. Not sure if the same applies to K3s, though. [1] https://github.com/kubernetes/minikube/issues/3207 https://github.com/kubernetes/minikube/issues/3207, https://github.com/docker/for-mac/issues/3065 https://github.com/docker/for-mac/issues/3065, https://github.com/docker/for-mac/issues/3539 https://github.com/docker/for-mac/issues/3539, etc; there must be dozens and dozens of these
- rcarmo 7y ago
  It does. My k3s setups (like https://github.com/rcarmo/azure-k3s-cluster https://github.com/rcarmo/azure-k3s-cluster) take up nearly 100% of the puny master node I allocate to them, and kill my Raspberry Pi SD cards as well. Swarm, in comparison, is much friendlier (and you can use it for dev/test across multiple machines just fine)