6 ms·

I think this is underrated as a design flaw for how Linux tends to be used in 2024. At its most benign it's an anachronism and potential source of complexity, a

I think this is underrated as a design flaw for how Linux tends to be used in 2024. At its most benign it's an anachronism and potential source of complexity, as its worst it's a major source of security vulnerabilities and unintended behavior (eg linux multitenancy was designed for two people in the same lab sharing a server, not for running completely untrusted workloads at huge scale, so it doesn't really implement resource fairness or protect against DoS well). I haven't had a chance to try it out but this is why I think Talos linux (https://www.talos.dev/ https://www.talos.dev/) is a step in the right direction for Linux as it is used for cloud/servers. Though personally I think multitenancy esp. regarding containerized applications/cgroups is a bigger problem and I don't know if they're addressing that.

mikepurvis 2y ago
My impression was that all the hosted k8s providers are doing multitenancy with if not full per-customer VMs, at least additional abstractions like gVisor. Are there some that aren't? Or are you referring here more to untrusted/shared in the sense of platforms like Github Actions just running everyone's different loads on the same pool of kernels?
- weitendorf 2y ago
  Right, I'm talking about shared-kernel multitenancy. Shared-kernel multitenancy isn't just about reducing the OS overhead from host + one or more VMs (or sandboxes) to just a single host, it's also about not having to continually start and stop virtual machines/sandboxes, which introduces its own resource overhead as well as a latency hit (which essentially always coincides with resource pressure from increased usage, since that's why you're scaling up) every time it's done. Also, even VMs and sandboxes don't really protect against DoS/resource fairness/noisy neighbor problems that well in many cases. Why does this matter? Incurring kernel/sandbox boot overhead on cold start/scaling makes it so that services have to over-provision resources to account for potential future resource needs. This wastes a lot of compute. I also think it's incredibly wasteful for companies to manage their own K8s cluster (if K8s supported multitenancy you'd probably want only one instance per-datacenter, and move whatever per-cluster settings people depend on to per-service. This is also much closer to "how Google does things" and why I think Kubernetes sucks to use compared to Borg), again because of all the stranded compute, and also because of the complexity of setting it up and managing it - but without shared-kernel multitenancy, multi-tenant K8s has to employ a lot of complicated workarounds in userspace. Or you can use a serverless product, ie pay for someone else's implementation of those workarounds, and still suffer some of the resource/latency overhead from lack of shared kernel multitenancy. This is one of the problems I want to address with my company BTW, but it would take years if not decades to get there, which is why I'm starting with something else.
3eb7988a1663 2y ago
Actually, I have been wondering if using a Linux system as multi-user could be a boon in security. As single user, each and every process has full and complete control of $HOME. Instead, I would prefer all applications were sandboxed to their own little respective areas with minimal access to data unless explicitly authorized. Without going full QubeOS, get some amount of application separation so my photo utility does not have permissions to read ~.ssh. Create a user account for each application (Firefox, Email, PDFReader, etc). Run each of those applications as the foreign user account. Each application now has its own $HOME with minimal user data. Barring a root-escalation or user-separation bug, the data in your true HOME should be isolated. Even process/environment variable space should be segregated. This also has a win in that it becomes possible to better segregate the threat model of less trusted applications. Doing granular network permissions per application is a bit hairy in Linux, but it is trivial to fully deny network access to a specific user account. Not true isolation, but for the semi-trusted development environment, gets you a little something.
- weitendorf 2y ago
  For shared local usage or "pet" (as opposed to cattle) servers I'd agree, and in fact this is close to what Linux was designed for, since I'd consider the multiplexing lab server to also be a semi-trusted environment. I'm referring more to how Linux is used in vast pools of "cattle" servers in Cloud, locally by eg one main user (who doesn't need multi-user but probably still needs some notion of "admin" and per-program permissions), or in a corporate setting (where the actual identity system is managed remotely). This is probably >99% of Linux environments.
- bombcar 2y ago
  Qubes takes that to an extreme: https://www.qubes-os.org/intro/ https://www.qubes-os.org/intro/ and runs every application in a virtual machine.
  - rendaw 2y ago
    I was experimenting with taking that to a median using containers in nixos. IMO the distinguishing feature of qubes is the fact that there's chrome indicating the security level of a window based on its vm - I put together https://github.com/andrewbaxter/filterway https://github.com/andrewbaxter/filterway to use with window manager rules to hopefully get the same result.
cedws 2y ago
I agree that multi-user should go away for modern server workloads, however, users are used as a blast door. Mainly because Linux's security model is lacking. systemd for example commonly runs services under separate users to make it more difficult for a compromised application to elevate privileges. Android does something similar AFAIK. Users should have never became a security boundary to isolate applications, but they unfortunately have, and there's not really an alternative.
- weitendorf 2y ago
  This is why I think multitenancy is the more important problem (though both are related), because it's the key to solving shared-kernel application permissions without "users". Containers were a step in the right direction but aren't a sufficient security boundary in themselves - what is currently handled by the "container runtime"/sandbox needs to be built into the kernel IMO.
- goodpoint 2y ago
  > Linux's security model is lacking It's not lacking at all. The root + users model is common not only across OSes but also all sort of physical devices.
  - moring 2y ago
    Linux's security model doesn't become better just because everybody is doing it that way, and besides that, everybody is doing it because they are copying Linux.
    - goodpoint 2y ago
      > Linux's security model doesn't become better just because everybody is doing it that way I didn't claim it does. > everybody is doing it because they are copying Linux. This is not true. The model existed decades before Linux.
  - stubish 2y ago
    Nah, its been lacking since inception, with people trying things like chroot jails and suid bits decades before Linux was a twinkle in an eye, and we still regularly fail at running untrusted code.

mikepurvis 2y ago

My impression was that all the hosted k8s providers are doing multitenancy with if not full per-customer VMs, at least additional abstractions like gVisor. Are there some that aren't? Or are you referring here more to untrusted/shared in the sense of platforms like Github Actions just running everyone's different loads on the same pool of kernels?

weitendorf 2y ago
Right, I'm talking about shared-kernel multitenancy. Shared-kernel multitenancy isn't just about reducing the OS overhead from host + one or more VMs (or sandboxes) to just a single host, it's also about not having to continually start and stop virtual machines/sandboxes, which introduces its own resource overhead as well as a latency hit (which essentially always coincides with resource pressure from increased usage, since that's why you're scaling up) every time it's done. Also, even VMs and sandboxes don't really protect against DoS/resource fairness/noisy neighbor problems that well in many cases. Why does this matter? Incurring kernel/sandbox boot overhead on cold start/scaling makes it so that services have to over-provision resources to account for potential future resource needs. This wastes a lot of compute. I also think it's incredibly wasteful for companies to manage their own K8s cluster (if K8s supported multitenancy you'd probably want only one instance per-datacenter, and move whatever per-cluster settings people depend on to per-service. This is also much closer to "how Google does things" and why I think Kubernetes sucks to use compared to Borg), again because of all the stranded compute, and also because of the complexity of setting it up and managing it - but without shared-kernel multitenancy, multi-tenant K8s has to employ a lot of complicated workarounds in userspace. Or you can use a serverless product, ie pay for someone else's implementation of those workarounds, and still suffer some of the resource/latency overhead from lack of shared kernel multitenancy. This is one of the problems I want to address with my company BTW, but it would take years if not decades to get there, which is why I'm starting with something else.

3eb7988a1663 2y ago

Actually, I have been wondering if using a Linux system as multi-user could be a boon in security. As single user, each and every process has full and complete control of $HOME. Instead, I would prefer all applications were sandboxed to their own little respective areas with minimal access to data unless explicitly authorized. Without going full QubeOS, get some amount of application separation so my photo utility does not have permissions to read ~.ssh. Create a user account for each application (Firefox, Email, PDFReader, etc). Run each of those applications as the foreign user account. Each application now has its own $HOME with minimal user data. Barring a root-escalation or user-separation bug, the data in your true HOME should be isolated. Even process/environment variable space should be segregated. This also has a win in that it becomes possible to better segregate the threat model of less trusted applications. Doing granular network permissions per application is a bit hairy in Linux, but it is trivial to fully deny network access to a specific user account. Not true isolation, but for the semi-trusted development environment, gets you a little something.

weitendorf 2y ago
For shared local usage or "pet" (as opposed to cattle) servers I'd agree, and in fact this is close to what Linux was designed for, since I'd consider the multiplexing lab server to also be a semi-trusted environment. I'm referring more to how Linux is used in vast pools of "cattle" servers in Cloud, locally by eg one main user (who doesn't need multi-user but probably still needs some notion of "admin" and per-program permissions), or in a corporate setting (where the actual identity system is managed remotely). This is probably >99% of Linux environments.
bombcar 2y ago
Qubes takes that to an extreme: https://www.qubes-os.org/intro/ https://www.qubes-os.org/intro/ and runs every application in a virtual machine.
- rendaw 2y ago
  I was experimenting with taking that to a median using containers in nixos. IMO the distinguishing feature of qubes is the fact that there's chrome indicating the security level of a window based on its vm - I put together https://github.com/andrewbaxter/filterway https://github.com/andrewbaxter/filterway to use with window manager rules to hopefully get the same result.

cedws 2y ago

I agree that multi-user should go away for modern server workloads, however, users are used as a blast door. Mainly because Linux's security model is lacking. systemd for example commonly runs services under separate users to make it more difficult for a compromised application to elevate privileges. Android does something similar AFAIK. Users should have never became a security boundary to isolate applications, but they unfortunately have, and there's not really an alternative.

weitendorf 2y ago
This is why I think multitenancy is the more important problem (though both are related), because it's the key to solving shared-kernel application permissions without "users". Containers were a step in the right direction but aren't a sufficient security boundary in themselves - what is currently handled by the "container runtime"/sandbox needs to be built into the kernel IMO.
goodpoint 2y ago
> Linux's security model is lacking It's not lacking at all. The root + users model is common not only across OSes but also all sort of physical devices.
- moring 2y ago
  Linux's security model doesn't become better just because everybody is doing it that way, and besides that, everybody is doing it because they are copying Linux.
  - goodpoint 2y ago
    > Linux's security model doesn't become better just because everybody is doing it that way I didn't claim it does. > everybody is doing it because they are copying Linux. This is not true. The model existed decades before Linux.
- stubish 2y ago
  Nah, its been lacking since inception, with people trying things like chroot jails and suid bits decades before Linux was a twinkle in an eye, and we still regularly fail at running untrusted code.