5 ms·

I don't understand the point of this... if they are open source Linux Kernel contributions, shouldn't they be upstreamed into the Linux kernel? Is Facebook jus

I don't understand the point of this... if they are open source Linux Kernel contributions, shouldn't they be upstreamed into the Linux kernel? Is Facebook just throwing out stuff from their private branch of the Linux kernel in hopes that somebody has free time to upstream the changes? Not being snarky, I genuinely don't understand the point of this...

ISL 8y ago
It is a whole lot easier to peruse and use modifications that someone else has written and debugged than to write them oneself. It is generally a cause for celebration when someone releases their work for everyone else to consider for use.
- pweissbrod 8y ago
  The philosophy of the kernel is to work for everyone's need. Its a good question, if these enhancements are of broad benefit and unobtrusive as they seem it's likely better to upstreaming it vs out-of-band patches.
czardoz 8y ago
Some of these are not actually things that should be in the main kernel, which is also why they are being referred to as "components". E.g: Not every laptop user needs to be able to run BPF.
- monocasa 8y ago
  > Not every laptop user needs to be able to run BPF. True for the moment, but I can see that changing. Having a safe VM at the system call layer is a game changer for so many subsystems. It turns the Linux kernel into a hybrid exokernel.
  - DelightOne 8y ago
    How would other subsystems use the safety of the BPF program - which I assume is the safe VM you mention? Do they have to assume that the network stack may not halt and have special code for it or sth ?
    - monocasa 8y ago
      eBPF doesn't _have_ to use the network stack at all. So imagine i2c devices that get pulled out of the kernel. KVM drivers for top half of MMIO so KVM only needs half the number of context switches. Audio DSP kernels that run at interrupt time for super low latency. Really almost any interesting real time work. Etc.
      - DelightOne 8y ago
        > i2c devices that get pulled out of the kernel. I thought eBPF programs run inside the kernel. Or are they _from_ userspace, run inside the kernel and so provide efficient into-kernel calls as well as direct access to certain _allowed_ hardware? I hope I’m not riding around semantics, sorry if I do.
        monocasa 8y ago
        They are programs, provided by user space, that are sandboxed to run safely in kernel space. Anything that isn't a RISC-esque instruction in BPF is a jump to a kernel function, and each flavor of BPF (BPF_PROG_TYPE_*) has it's own table of allowed functions. Most of the program types that are mainlined call functions in the net stack, but that's not intrinsic to the idea of BPF. So in the I2C case, I could see device drivers that get called to handle the weird things that I2C devices do where you're not really sure until master interrupt time exactly what needs to happen next and you need to make a real time decision. So a BPF program that implements a state machine and is run directly from the master interrupt could be more performant and more power efficient than doing it in user space, and safer than doing it in kernel space. Did that answer your question?
        8y ago
  - rurban 8y ago
    It's fast but certainly not safe. it has native arrays, with which you can easily exploit spectre or meltdown issues. to be safe you need to turn it off.
    - monocasa 8y ago
      The MAP_ARRAY access goes through an external function call, which gives you a nice place to hook for mitigations. And this stuff is going to have to be fixed properly, as running sandboxed untrusted code in a higher context is waaayyy to useful in a post Moore's law world. Kernels like XOK relied on similar techniques up and down their system call layer; it had three different virtual machines.
traek 8y ago
Some of the things look like they've already been upstreamed (cgroups2). Tejun Heo, the author of this announcement, is a Linux kernel developer.
- cyphar 8y ago
  Tejun is the maintainer of cgroups (and has been for several years, and has moved companies several times). All of my angst and annoyance with cgroupv2 aside, quite a few of these projects were discussed at the last OSS and he's quite right that some of the OOM handling needs to be out-of-kernel to avoid killing things that are doing actual work.
tssva 8y ago
btrfs has been part of the mainline kernel since 2.6.29, eBPF since 4.1 and cgroups2 since 4.5. That leaves PSI as the only kernel component mentioned that has not yet been upstreamed and the intention is for it to be.
smartmix6 8y ago
Maybe it's for showing off that they contribute...

ISL 8y ago

It is a whole lot easier to peruse and use modifications that someone else has written and debugged than to write them oneself. It is generally a cause for celebration when someone releases their work for everyone else to consider for use.

pweissbrod 8y ago
The philosophy of the kernel is to work for everyone's need. Its a good question, if these enhancements are of broad benefit and unobtrusive as they seem it's likely better to upstreaming it vs out-of-band patches.

czardoz 8y ago

Some of these are not actually things that should be in the main kernel, which is also why they are being referred to as "components". E.g: Not every laptop user needs to be able to run BPF.

monocasa 8y ago
> Not every laptop user needs to be able to run BPF. True for the moment, but I can see that changing. Having a safe VM at the system call layer is a game changer for so many subsystems. It turns the Linux kernel into a hybrid exokernel.
- DelightOne 8y ago
  How would other subsystems use the safety of the BPF program - which I assume is the safe VM you mention? Do they have to assume that the network stack may not halt and have special code for it or sth ?
  - monocasa 8y ago
    eBPF doesn't _have_ to use the network stack at all. So imagine i2c devices that get pulled out of the kernel. KVM drivers for top half of MMIO so KVM only needs half the number of context switches. Audio DSP kernels that run at interrupt time for super low latency. Really almost any interesting real time work. Etc.
    - DelightOne 8y ago
      > i2c devices that get pulled out of the kernel. I thought eBPF programs run inside the kernel. Or are they _from_ userspace, run inside the kernel and so provide efficient into-kernel calls as well as direct access to certain _allowed_ hardware? I hope I’m not riding around semantics, sorry if I do.
      - monocasa 8y ago
        They are programs, provided by user space, that are sandboxed to run safely in kernel space. Anything that isn't a RISC-esque instruction in BPF is a jump to a kernel function, and each flavor of BPF (BPF_PROG_TYPE_*) has it's own table of allowed functions. Most of the program types that are mainlined call functions in the net stack, but that's not intrinsic to the idea of BPF. So in the I2C case, I could see device drivers that get called to handle the weird things that I2C devices do where you're not really sure until master interrupt time exactly what needs to happen next and you need to make a real time decision. So a BPF program that implements a state machine and is run directly from the master interrupt could be more performant and more power efficient than doing it in user space, and safer than doing it in kernel space. Did that answer your question?
        8y ago
- rurban 8y ago
  It's fast but certainly not safe. it has native arrays, with which you can easily exploit spectre or meltdown issues. to be safe you need to turn it off.
  - monocasa 8y ago
    The MAP_ARRAY access goes through an external function call, which gives you a nice place to hook for mitigations. And this stuff is going to have to be fixed properly, as running sandboxed untrusted code in a higher context is waaayyy to useful in a post Moore's law world. Kernels like XOK relied on similar techniques up and down their system call layer; it had three different virtual machines.

traek 8y ago

Some of the things look like they've already been upstreamed (cgroups2). Tejun Heo, the author of this announcement, is a Linux kernel developer.

cyphar 8y ago
Tejun is the maintainer of cgroups (and has been for several years, and has moved companies several times). All of my angst and annoyance with cgroupv2 aside, quite a few of these projects were discussed at the last OSS and he's quite right that some of the OOM handling needs to be out-of-kernel to avoid killing things that are doing actual work.

tssva 8y ago

btrfs has been part of the mainline kernel since 2.6.29, eBPF since 4.1 and cgroups2 since 4.5. That leaves PSI as the only kernel component mentioned that has not yet been upstreamed and the intention is for it to be.

smartmix6 8y ago

Maybe it's for showing off that they contribute...