4 ms·

Linux Kernel 5.14

126 points by nitinreddy88 5y ago

tim-- 5y ago
The memfd_secret() system call is pretty exciting (https://lwn.net/Articles/836724/ https://lwn.net/Articles/836724/). Once the secretmem feature is enabled at boot time, you are hardened against kernel flaws. A userspace process (running with elevated privileges) will only be able to perform secrets exfiltration using ptrace. A better source for these changes are the two merge window roundups from LWN: - https://lwn.net/Articles/861248/ https://lwn.net/Articles/861248/ - https://lwn.net/Articles/861695/ https://lwn.net/Articles/861695/
- Retr0id 5y ago
  It's worth noting that it isn't a magic bullet - it only hardens against certain classes of kernel flaw. If the kernel is compromised, it is trivial to extract the secrets (this may seem obvious, but the strength of the feature has been over-hyped by some publications) https://github.com/JonathonReinhart/nosecmem https://github.com/JonathonReinhart/nosecmem
  - tedunangst 5y ago
    Yeah, I'm kinda unclear on exactly what it prevents. Shouldn't the kernel, if not compromised, be preventing these mappings anyway?
    - geofft 5y ago
      Linux has a "direct map" of all physical memory in kernelspace, which I guess makes things easier. See https://www.kernel.org/doc/Documentation/x86/x86_64/mm.txt https://www.kernel.org/doc/Documentation/x86/x86_64/mm.txt "direct mapping of all physical memory (page_offset_base)". (I think this is not uncommon: JOS, which my undergrad operating systems class used, does this.) Memory allocated with memfd_secret gets removed from the direct map, which means that from any other process's address space, there is no virtual address (regardless of memory protection) that corresponds to those physical addresses. So any exploit that gets only gets you memory read/write but not kernelspace code execution (various types of wild pointers, etc., as well as whatever exciting speculative execution vulnerability will undoubtedly show up next year) cannot compromise the data in a memfd_secret mapping.
      - rfoo 5y ago
        > any exploit that gets only gets you memory read/write but not kernelspace code execution That's an odd assumption. There are still a lot of universal tricks for turning a truly arbitrary R/W primitive into code execution, e.g. just write to the page table. OTOH, this does guard against arbitrary read primitives, which, as you mentioned, is especially useful in post-Spectre days.
      - tedunangst 5y ago
        Ah, okay. I get it. There has been some loose talk to reduce the amount of memory mapped in the direct map for openbsd.
    - Retr0id 5y ago
      I'm speculating a bit here, but I think it could guard against side-channel attacks against the kernel. e.g. if you had a sidechannel that leaks data from the kernel, the kernel can't disclose data that it can't actually read.
  - ggm 5y ago
    Attacks on s/w and h/w systems are asymmetric warfare. Until the ptrace or other attacks are scripted, the barrier to entry is usefully high. Once they are scripted, it can be close to zero, depending. OTOH if the ptrace and other approaches leave a signature, then you remain at risk but able to know. And, it has wheel-of-life qualities as your attacker codes to hide the signature, you code to keep the visibility open, and to try and defeat the underlying attack. And so it goes.
viraptor 5y ago
Is there something bad happening with kernelnewbies? They're behind on 5.13 and 5.14 is just a placeholder at the moment.
- KingMachiavelli 5y ago
  I think it is often a kernel version or two behind.
- nnx 5y ago
  Hopefully just taking some time off for the holidays
geofft 5y ago
I'm very excited for the better CFS quota enforcement mechanism in 5.14 (https://lwn.net/Articles/844976/ https://lwn.net/Articles/844976/). Kubernetes implements CPU and memory limits for containers (https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ https://kubernetes.io/docs/concepts/configuration/manage-res...). The memory limit works the way you'd expect: there's a memory cgroup, and Linux implements the memory-allocation algorithm based on the memory cgroup's limit and not the total system memory. Wildly-overboard allocations get denied; somewhat-overboard allocations are permitted if you have overcommit turned on, but cause an OOM kill if you try to use all of it. This is at least expected behavior for existing non-Kubernetes applications and (more importantly) for people used to non-Kubernetes application development/deployment: it's just like if you were running on a Linux system with the limited memory. CPU limits are much wackier. Because CPUs are a discrete resource and Kubernetes supports fractional CPU allocations, they can't use CPU pinning. Instead, they use the CFS quota controls, which aren't commonly used outside Kubernetes as far as I can tell. The quota is expressed as basically a fraction: for a process with, say, 0.5 vCPUs allocated, it gets to use 50 ms of CPU time every 100 ms of real time. The way it works at the kernel level, as best as I've been able to understand, is that after a process runs on the CPU for a bit, the kernel looks at the quota and sees if it has exceeded the quota. If it has, the process becomes unschedulable until it has "re-earned" sufficient quota. It does not slow down or proactively de-schedule the process, analogous to how it would have run on a slower or fewer-CPU computer. So, if you have multiple processes or threads running in your container, and there are free cores for them to run, you can easily use up multiple seconds of CPU time in those 100 ms. Then you have to wait to re-earn your quota in 50 ms chunks before your process can run. And all the while the process appears to be runnable, not stuck, it just seems like the kernel is never choosing to schedule it. We have seen, in production, processes stalled for minutes because they evidently ran so far past their quota. We have seen entire system deadlocks when such a stalled process is holding some kernel lock for network I/O, in our case, Ceph (if the kernel releases the process into interruptible sleep, then that process can still hold locks; it will release those locks if signaled and rescheduled, but in this case the process isn't being scheduled at all). And we're far from the only people who have seen awful behavior from CFS quotas. Tim Hockin, Kubernetes co-creator and tech lead, advises people to just never use the CPU limit feature! https://twitter.com/thockin/status/1134193838841401345 https://twitter.com/thockin/status/1134193838841401345 We have done as he advises and just turned it off cluster-wide with --cpu-cfs-quota=false - it helps the Kubernetes scheduler if you set your CPU/memory limits equal to your requests, so this gets us the right Kubernetes behavior without incurring the wacky kernel behavior. If you Google that flag you can find many others who have done the same in their public configurations. There's a Kubernetes feature to use CPU pinning instead, which mentions improved performance as an advantage https://kubernetes.io/blog/2018/07/24/feature-highlight-cpu-manager/ https://kubernetes.io/blog/2018/07/24/feature-highlight-cpu-... , but it only works if you're scheduling whole numbers of CPUs always. A whole lot of applications on our Kubernetes clusters - lightly-accessed web servers, monitoring daemons, etc. - are best described as wanting 0.1 or 0.5 CPUs, in the sense that on (say) an 8-core machine, you could run many dozens of instances of these applications just fine. But that's not how the existing CFS quota mechanism interprets 0.1 or 0.5 CPUs. It seems like the "burst" mechanism here is likely to handle processes that, in a random 100 ms period, actually do run for more than 10 ms, but on an ongoing basis don't average out to using more than one tenth of a CPU. We might be able to turn --cpu-cfs-quota back on as a result.
pb82 5y ago
If you are having trouble with suspend/resume on your Ryzen 4xxx or 5xxx laptop, you might be in luck with this version. See https://gitlab.freedesktop.org/drm/amd/-/issues/1230 https://gitlab.freedesktop.org/drm/amd/-/issues/1230 for details.

126 points by nitinreddy88 5y ago

The memfd_secret() system call is pretty exciting (https://lwn.net/Articles/836724/ https://lwn.net/Articles/836724/). Once the secretmem feature is enabled at boot time, you are hardened against kernel flaws. A userspace process (running with elevated privileges) will only be able to perform secrets exfiltration using ptrace. A better source for these changes are the two merge window roundups from LWN: - https://lwn.net/Articles/861248/ https://lwn.net/Articles/861248/ - https://lwn.net/Articles/861695/ https://lwn.net/Articles/861695/

Retr0id 5y ago
It's worth noting that it isn't a magic bullet - it only hardens against certain classes of kernel flaw. If the kernel is compromised, it is trivial to extract the secrets (this may seem obvious, but the strength of the feature has been over-hyped by some publications) https://github.com/JonathonReinhart/nosecmem https://github.com/JonathonReinhart/nosecmem
- tedunangst 5y ago
  Yeah, I'm kinda unclear on exactly what it prevents. Shouldn't the kernel, if not compromised, be preventing these mappings anyway?
  - geofft 5y ago
    Linux has a "direct map" of all physical memory in kernelspace, which I guess makes things easier. See https://www.kernel.org/doc/Documentation/x86/x86_64/mm.txt https://www.kernel.org/doc/Documentation/x86/x86_64/mm.txt "direct mapping of all physical memory (page_offset_base)". (I think this is not uncommon: JOS, which my undergrad operating systems class used, does this.) Memory allocated with memfd_secret gets removed from the direct map, which means that from any other process's address space, there is no virtual address (regardless of memory protection) that corresponds to those physical addresses. So any exploit that gets only gets you memory read/write but not kernelspace code execution (various types of wild pointers, etc., as well as whatever exciting speculative execution vulnerability will undoubtedly show up next year) cannot compromise the data in a memfd_secret mapping.
    - rfoo 5y ago
      > any exploit that gets only gets you memory read/write but not kernelspace code execution That's an odd assumption. There are still a lot of universal tricks for turning a truly arbitrary R/W primitive into code execution, e.g. just write to the page table. OTOH, this does guard against arbitrary read primitives, which, as you mentioned, is especially useful in post-Spectre days.
    - tedunangst 5y ago
      Ah, okay. I get it. There has been some loose talk to reduce the amount of memory mapped in the direct map for openbsd.
  - Retr0id 5y ago
    I'm speculating a bit here, but I think it could guard against side-channel attacks against the kernel. e.g. if you had a sidechannel that leaks data from the kernel, the kernel can't disclose data that it can't actually read.
- ggm 5y ago
  Attacks on s/w and h/w systems are asymmetric warfare. Until the ptrace or other attacks are scripted, the barrier to entry is usefully high. Once they are scripted, it can be close to zero, depending. OTOH if the ptrace and other approaches leave a signature, then you remain at risk but able to know. And, it has wheel-of-life qualities as your attacker codes to hide the signature, you code to keep the visibility open, and to try and defeat the underlying attack. And so it goes.

viraptor 5y ago

Is there something bad happening with kernelnewbies? They're behind on 5.13 and 5.14 is just a placeholder at the moment.

KingMachiavelli 5y ago
I think it is often a kernel version or two behind.
nnx 5y ago
Hopefully just taking some time off for the holidays

geofft 5y ago

I'm very excited for the better CFS quota enforcement mechanism in 5.14 (https://lwn.net/Articles/844976/ https://lwn.net/Articles/844976/). Kubernetes implements CPU and memory limits for containers (https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ https://kubernetes.io/docs/concepts/configuration/manage-res...). The memory limit works the way you'd expect: there's a memory cgroup, and Linux implements the memory-allocation algorithm based on the memory cgroup's limit and not the total system memory. Wildly-overboard allocations get denied; somewhat-overboard allocations are permitted if you have overcommit turned on, but cause an OOM kill if you try to use all of it. This is at least expected behavior for existing non-Kubernetes applications and (more importantly) for people used to non-Kubernetes application development/deployment: it's just like if you were running on a Linux system with the limited memory. CPU limits are much wackier. Because CPUs are a discrete resource and Kubernetes supports fractional CPU allocations, they can't use CPU pinning. Instead, they use the CFS quota controls, which aren't commonly used outside Kubernetes as far as I can tell. The quota is expressed as basically a fraction: for a process with, say, 0.5 vCPUs allocated, it gets to use 50 ms of CPU time every 100 ms of real time. The way it works at the kernel level, as best as I've been able to understand, is that after a process runs on the CPU for a bit, the kernel looks at the quota and sees if it has exceeded the quota. If it has, the process becomes unschedulable until it has "re-earned" sufficient quota. It does not slow down or proactively de-schedule the process, analogous to how it would have run on a slower or fewer-CPU computer. So, if you have multiple processes or threads running in your container, and there are free cores for them to run, you can easily use up multiple seconds of CPU time in those 100 ms. Then you have to wait to re-earn your quota in 50 ms chunks before your process can run. And all the while the process appears to be runnable, not stuck, it just seems like the kernel is never choosing to schedule it. We have seen, in production, processes stalled for minutes because they evidently ran so far past their quota. We have seen entire system deadlocks when such a stalled process is holding some kernel lock for network I/O, in our case, Ceph (if the kernel releases the process into interruptible sleep, then that process can still hold locks; it will release those locks if signaled and rescheduled, but in this case the process isn't being scheduled at all). And we're far from the only people who have seen awful behavior from CFS quotas. Tim Hockin, Kubernetes co-creator and tech lead, advises people to just never use the CPU limit feature! https://twitter.com/thockin/status/1134193838841401345 https://twitter.com/thockin/status/1134193838841401345 We have done as he advises and just turned it off cluster-wide with --cpu-cfs-quota=false - it helps the Kubernetes scheduler if you set your CPU/memory limits equal to your requests, so this gets us the right Kubernetes behavior without incurring the wacky kernel behavior. If you Google that flag you can find many others who have done the same in their public configurations. There's a Kubernetes feature to use CPU pinning instead, which mentions improved performance as an advantage https://kubernetes.io/blog/2018/07/24/feature-highlight-cpu-manager/ https://kubernetes.io/blog/2018/07/24/feature-highlight-cpu-... , but it only works if you're scheduling whole numbers of CPUs always. A whole lot of applications on our Kubernetes clusters - lightly-accessed web servers, monitoring daemons, etc. - are best described as wanting 0.1 or 0.5 CPUs, in the sense that on (say) an 8-core machine, you could run many dozens of instances of these applications just fine. But that's not how the existing CFS quota mechanism interprets 0.1 or 0.5 CPUs. It seems like the "burst" mechanism here is likely to handle processes that, in a random 100 ms period, actually do run for more than 10 ms, but on an ongoing basis don't average out to using more than one tenth of a CPU. We might be able to turn --cpu-cfs-quota back on as a result.

pb82 5y ago

If you are having trouble with suspend/resume on your Ryzen 4xxx or 5xxx laptop, you might be in luck with this version. See https://gitlab.freedesktop.org/drm/amd/-/issues/1230 https://gitlab.freedesktop.org/drm/amd/-/issues/1230 for details.