14 ms·

No More Blue Fridays

481 points by moreati 2y ago

fullspectrumdev 2y ago
This puts an awful lot of stock in the robustness of eBPF. Which is odd, given there’s been a bunch of kernel privesc bugs using eBPF…
joker99 2y ago
I used to work for an EDR vendor and this post glosses over two major and important things. 1. There’s no need for eBPF on windows, it has the ETW framework (event tracing) which is much more powerful and provides applications subscribing to a class of events almost too detailed insights. the issue most AV vendors have with it though is speed. Leading to … 2. eBPF lets you watch. Congrats. It’s something, but it’s not the reason why these tools are deployed. Orgs deploy these tools to prevent or stop potentially bad stuff from executing. The only place this can be done in our operating systems is usually the kernel - for that you need kernel level drivers or various other filter drivers. Crowdstrike screwed the pooch here, yes. But after a couple of days I feel like I haven’t read enough blog posts and articles that crap on Microsoft. It’s their job to build a secure operating system, instead they deliver Windows and because they themselves cannot secure windows, they ship defender… and we use tools like falcon like a bandaid for Microsofts bad security practices
- deleted 2y ago
  [deleted]
- thayne 2y ago
  > eBPF lets you watch. Congrats. It’s something, but it’s not the reason why these tools are deployed. Orgs deploy these tools to prevent or stop potentially bad stuff from executing eBPF let's you prevent things too. seccomp filters can block syscalls. The bigger problem is the performance you mentioned in 1. Crowdstrike's linux agent can work using eBPF instead of a kernel module, and will fall back to that if the current kernel version is more recent than the agent supports. But... then it uses up a lot more CPU.
kjellsbells 2y ago
Lets suppose that eBPF solves this particular problem, eventually, for Windows. Doesn't sidestepping the entire class of Crowdstrike-style fubars require that Microsoft then mandate that no, backward compatibility will not be offered? Back compat seems to be such a shibboleth in the Windows world, but comes at an incredible price. The reasons cited all seem to boil down to keeping some imagined customers' obscure LOB app running for decades. But that seems like an excuse to me. Surely Microsoft would like to shake out the last diehards running some VB5 app on a patched up PC in a factory. Isn't it more beneficial to everyone to start sunsetting acres of ancient NT code and approaches and streamline the entire attack surface?
- pas 2y ago
  it would be enough if MS offered knobs and switches for admins/devs/vendors to disallow non-static-verified stuff in the kernel
- acdha 2y ago
  Backwards compatibility slows things down in the Windows world but it doesn’t halt improvements. In this case, there are two powerful ratchets: 1. Compliance: everyone affected by this bug has auditors. Once safer alternatives are available, the standards like CIS, PCI, etc. will be updated to say you should use the new interface, and every enterprise IT department will have pressure to switch to eBPF tools. We saw this with BootLocker: storage encryption used to be a pain, people resisted it, but over time it became universal because the cost of swimming upstream was too high. 2. Signing. Microsoft can start requiring more proof of need and restrictions for signing drivers. They have to be careful to avoid the appearance of favoritism but after this debacle that’s a LOT easier. I would bet some engineer is working on a draft of mandatory fault handling and testing proof requirements for critical kernel drivers now and I would not be surprised to see it include a timeframe for adopting memory-safe languages.
- another2another 2y ago
  >Surely Microsoft would like to shake out the last diehards running some VB5 app on a patched up PC in a factory. >Isn't it more beneficial to everyone to start sunsetting acres of ancient NT code and approaches and streamline the entire attack surface? If your code somehow still relies on some buggy behaviour to work, then MS shouldn't do anything to preserve that anymore - apparently they used to, but I'm not so sure nowadays. However 'ancient NT' code should probably still function just fine since the Win32 API hasn't changed much for a while, and MS don't actively deprecate function calls (unlike Apple who seem to do it a bit on a whim recently). I would put this down to the API being pretty well designed in the first place.
3np 2y ago
> The worst thing an eBPF program can do is to merely consume more resources than is desirable, such as CPU cycles and memory. This is obviously not true. It might be the worst it can do, by itself, to the currently running kernel. It's not the worst it can do to the machine or its user(s). There are infinite harmful things an eBPF program can do. As can programs solely in user-space. There is a specific class of vulnerabilities being mitigated by moving code from kernel to BPF. That does not mean that eBPF programs are in general safe.
userbinator 2y ago
In the future, computers will not crash due to bad software updates, even those updates that involve kernel code. 100% BS. Even if they don't "crash" they will "stop functioning as intended" which is just the same. It's absolutely disgusting how this industry is now using this one outage as a talking point to further their totalitarian agenda. It reminds me of how Google went after adblockers with their new extension model that also promised more "security". It's time we realised what they're really trying to do. In fact, I wonder whether this outage was not accidental after all.
yubiox 2y ago
Title reminds me of when microsoft promised no more UAEs back in 92. They just renamed them to GPFs in windows 3.1.
jeffrallen 2y ago
Here's an idea for an interesting hack: a piece of kernel resident code that feeds fake data into eBPF so that an eBPF-based antimalware will see nothing bad as the malware goes about it's merry way. Sandboxes are safe, but are ultimately virtual machines, and virtual machines can be made to live in a world that's not real.
amluto 2y ago
> In the future, computers will not crash due to bad software updates, even those updates that involve kernel code. In the future, these updates will push eBPF code. eBPF is fantastic, and it can be used for many purposes and improve a lot of things, but this is IMO overselling it. Assuming that BPF itself it free of bugs, it’s still a rather large sprawl of kernel hooks, and those hooks invoke eBPF code, which can call right back into the kernel. Here’s a list: https://www.man7.org/linux/man-pages/man7/bpf-helpers.7.html https://www.man7.org/linux/man-pages/man7/bpf-helpers.7.html bpf_probe_read_kernel() is particularly heavily used, and it is not safe. It tries fairly hard not to OOPS or crash, but it is definitely not perfect. The rest of that list contains plenty of this that will easily take down a system, even if it doesn’t actually oops or panic in the process. And, of course, any tool that detects userspace “malicious behavior” and stops it can start calling everything malicious, and the computer becomes unusable. Meanwhile, eBPF has no real security model on the userspace side. Actual attachment of an eBPF program goes through the bpf() syscall, not through sensibly permissioned operations on the underlying kernel objects being attached to, and there is nothing whatsoever that confines eBPF to, say, a container that uses it. (See bpf_probe_read_kernel() -- it's fundamentally able to read all kernel memory.) So, IMO, most of the benefit of eBPF over ordinary kernel C code is that eBPF is kind of like writing code in a safe language with a limited unsafe API surface. It's a huge improvement for this sort of work, but it is not perfect by any means. > The verifier is rigorous -- the Linux implementation has over 20,000 lines of code The verifier is absurdly complex. I'd rather see something based on formal methods than 20kLOC of hand-written logic.
- School-Cotton 2y ago
  How is it possible to panic using bpf_probe_read_kernel ? Can you give an example that works on the current kernel version?
  - amluto 2y ago
    I'm not sure that "panic" is the right word here. bpf_probe_read_kernel boils down to copy_from_kernel_nofault, which checks for an "allowed" address and then does the access. Any page faults turn into error returns instead of OOPSes. x86 disallows user addresses, the vsyscall page, and non canonical addresses. Doing this from bpf assumes that all "allowed" addresses are side-effect-free and will either succeed or cleanly fault. Off the top of my head, MMIO space (including, oddities like the APIC page on CPUs that still have that) and TDX memory are not in this category.
ninju 2y ago
So a couple of questions 1) Is CrowdStrike Falcon using eBPF for their Linux offering? 2) Would the faulty patch update get caught by the eBPF verifier?
xg15 2y ago
> In the future, computers will not crash due to bad software updates, even those updates that involve kernel code. In the future, these updates will push eBPF code. Assuming every security critical system will be on a recent enough kernel to support this...
- efee22 2y ago
  I think with a LTS distribution you should get very far these days when it comes to implementing such sensors.
  - chasil 2y ago
    On rhel8 variants, you can use the Oracle UEK to get eBPF. https://blogs.oracle.com/linux/post/oracle-linux-and-bpf https://blogs.oracle.com/linux/post/oracle-linux-and-bpf $ cat /etc/redhat-release /etc/oracle-release /proc/version Red Hat Enterprise Linux release 8.10 (Ootpa) Oracle Linux Server release 8.10 Linux version 5.15.0-203.146.5.1.el8uek.x86_64 (mockbuild@host-100-100-224-48) (gcc (GCC) 11.2.1 20220127 (Red Hat 11.2.1-9.2.0.1), GNU ld version 2.36.1-4.0.1.el8_6) #2 SMP Thu Feb 8 17:14:39 PST 2024
- dijit 2y ago
  And assuming there's no bugs in the BPF code... Oh wait: https://news.ycombinator.com/item?id=41031699 https://news.ycombinator.com/item?id=41031699
  - efee22 2y ago
    RHEL kernel.. right. Imho, I'd trust an upstream stable kernel far more than a RHEL one for production which has dozen of feature backports and an internal kABI to maintain.. granted RH has a QA team, but it is still impossible to test everything beforehand.
    - worthless-trash 2y ago
      On the upside, non root users can't insert ebpf code, so its a priv'ed operation, not like other distros.
usrme 2y ago
Does anyone know how far along the eBPF implementation for Windows actually is? In the sense that it could start feasibly replacing existing kernel drivers.
CoastalCoder 2y ago
> If your company is paying for commercial software that includes kernel drivers or kernel modules, you can make eBPF a requirement. Are they saying that device drivers should be written in eBPF? Or maybe their drivers should expose an eBPF API? I assume some driver code still needs to reside in the actual kernel.
- prmoustache 2y ago
  These tool wouldn't need kernel drivers, only to target the eBPF userspace API: https://www.kernel.org/doc/html/latest/userspace-api/ebpf/index.html https://www.kernel.org/doc/html/latest/userspace-api/ebpf/in...
asynchronous 2y ago
Is there a reason for the lack of naming+shaming Crowdstrike in this blogpost? Was it to not give them any more publicity, good or bad?
- StevenWaterman 2y ago
  If you consider kernel programming to be inherently unsafe, then you would consider this to be inevitable, meaning it's not really the specific company's fault. They were just the unlucky ones.
  - efee22 2y ago
    Agree, Crowdstrike was an unlucky one, but it is more about the issue in general. If I remember correctly, also others like sysdig user their own kernel modules for collection.
  - asynchronous 2y ago
    I still hold true that testing even improperly would have caught this before it hit worldwide. But I suppose you are right, that doesn’t help the argument being made here.
    - ForOldHack 2y ago
      Wasnt that the job of AI/co-pilot/clippy /D.E.P? "Would you like me to try and execute a random blank file?" And of course QA. I was unaffected, but was fielding calls from customers. My update Tuesday is the week after, so in-between MS and my updates, I am very suspicious of everything. I was also unaffected by 22H2, and spent time fielding calls.
  - lordnacho 2y ago
    They could have helped their luck by doing some of the common sense things suggested in the article. For instance, why not find a subset of your customers that are low risk, push it out to them, and see what happens? Or perhaps have your own fleet of example installations to run things on first. None of which depends on any specific technology.
kayo_20211030 2y ago
This isn't right. If I need a system to run with a piece of code, then it shouldn't run at all if that piece of code is broken. Ignoring the failure is perverse. Let's say that the driver code ensures that some medical machine has safety locks (safeguards) in place to make sure that piece of equipment won't fry you to a crisp; I'd prefer that the whole thing not run at all rather than blithely operate with the safeguards disabled. It's turtles all the way down.
- emn13 2y ago
  The system clearly already behaves that way (i.e. ignores failure) - after all, the fix was to simply delete the offending file. If that's an option, then loader can do that too. It can and perhaps even is smarter, such as "fallback onto previous version". Furthermore, the reaction to a malformed state need not be "ignore". It could disable restricted user login; or turn off the screen. If the worry is that this is viable to abuse by malware, well, if the malware can already rewrite the on-disk files for the AV, I wonder whether it's really a good idea to trust the system itself to be able to deal with that. It'd probably be safer to just report that up the security foodchain, and potentially let some external system take measures such as disable or restrict network access. Better yet, such measures don't even require the same capabilities to intervene in the system, merely to observe - which makes the AV system less likely to serve as a malware vector itself or to cause bugs like this.
- Smaug123 2y ago
  I think the premise is false? It's up to the eBPF implementor what to do in the case of invalid input; the kernel could choose to perform a controlled shutdown in that case. (I have no idea what e.g. Linux actually does here, but one could imagine worlds where the action it takes on invalid input is configurable.) Also your statement is sometimes not true, although I certainly sympathise in the mainline case. In some contexts you really do need to keep on trucking. The first example to spring to mind is "the guidance computers on an automated Mars lander"; the round-trip to Earth is simply too long to defer responsibility in that case. If you shut down then you will crash, but if you do your best from a corrupted state then you merely probably crash, which is presumably better.
aurelien 2y ago
[flagged]
shrx 2y ago
From the article: > If the verifier finds any unsafe code, the program is rejected and not executed. The verifier is rigorous -- the Linux implementation has over 20,000 lines of code [0] -- with contributions from industry (e.g., Meta, Isovalent, Google) and academia (e.g., Rutgers University, University of Washington). [0] links to https://github.com/torvalds/linux/blob/master/kernel/bpf/verifier.c https://github.com/torvalds/linux/blob/master/kernel/bpf/ver... which has this interesting comment at the top: /* bpf_check() is a static code analyzer that walks eBPF program * instruction by instruction and updates register/stack state. * All paths of conditional branches are analyzed until 'bpf_exit' insn. * * The first pass is depth-first-search to check that the program is a DAG. * It rejects the following programs: * - larger than BPF_MAXINSNS insns * - if loop is present (detected via back-edge) ... I haven't inspected the code, but I thought that checking for infinite loops would imply solving the halting problem. Where's the catch?
- ahepp 2y ago
  If you’re wrong about the loop, you’ll still hit BPF_MAXINSNS, so it’s fine to use heuristics that could produce a false negative right?
- deleted 2y ago
  [deleted]
- dtx1 2y ago
  I have no insight into this particular project but you could work around the halting problem by only allowing loops you can proof will not go infinite. That would of course imply rejecting loops that won't go infinite but can't be proven not to.
- hiddencost 2y ago
  Unterminated loops might be a better phrasing.
- efee22 2y ago
  Infinite loops are not possible and would get rejected by the verifier since it cannot solve the halting problem. Here is a good overview on the options available: https://ebpf-docs.dylanreimerink.nl/linux/concepts/loops/ https://ebpf-docs.dylanreimerink.nl/linux/concepts/loops/
skywhopper 2y ago
The implicit assumption of the article is that eBPF code can't crash a kernel, but the article itself eventually admits that it can and has done, including last month. eBPF is a safer way of providing kernel-extension functionality, for sure, but presenting it as the perfect solution is just asking to have your argument dismissed. eBPF is not perfect. And there's plenty of things it can't do. The very sandbox rules that limit how long its programs may run and what they can do also make it entirely inappropriate for certain tasks. Let's please stop pretending there's a silver bullet.
- efee22 2y ago
  It's not a silver bullet, however, it is still better to pushing all the panicable bugs into one community-maintained section (e.g. eBPF verifier). All vendors have an incentive to help get right and this is much better than every vendor shipping their own panicable bugs in their own out of tree kernel modules. Additionally, it's not just the industry looking at eBPF, but also academia in terms of formally verifying these critical sections.
  - lucianbr 2y ago
    "Improves kernel stability" is great. "Prevents kernel crashes" is a plain lie. > In the future, computers will not crash due to bad software updates, even those updates that involve kernel code. Come on. Computers will continue to crash in the future, even when using eBPF. I am quite certain.
- lucianbr 2y ago
  It's casually claiming to have solved the halting problem, at least within some limited but useful context. That should be impossible, and it turns out, it is. I expect it can be solved within some limited contexts, but those contexts are not useful, at least not at the level of "generic kernel code".
  - red_admiral 2y ago
    It solves the halting problem by not being Turing complete. I presume each eBPF runs in a context with bounded memory, requested up front, for one thing; it also disallows jumps unless you can prove the code still halts.
vfclists 2y ago
Yep, another fix to all our problems, a new bandwagon to be jumped on by wall EDR vendors, until ... Here I am using the term "EDR". Until this CrowdStrike debacle I'd never heard it. Only tells how seriously you should take my opinions.
blinkingled 2y ago
Ok. But the good old push code to staging / canary it before mainstream updates was a simpler way of solving the same problem. Crowdstrike knows the computers they're running on, it is trivial to implement a system where only few designated computers download and install the update and report metrics before the update controller decides to push it to next set.
- phartenfeller 2y ago
  Why trust somebody else not messing up? With that in place for windows and crowdstrike billions of dollars would be saved and many lives not negatively impacted ...
- rldjbpin 2y ago
  with the way they handled the debian crashing a little while ago, frankly they are happy to still go ahead with testing this way. still much better way to handle things than pushing to everybody at the same time.
- Archelaos 2y ago
  It would mitigate the problem, but not solve it. You can still imagine a condition that only occurs after the update has been rolled out everywhere. Furthermore, such a bug would still be extremely problematic for the concerned customers, even if not all of them were affected. In addition, it would be necessary to react very quickly in the case of zero-day vulnerabilities.
  - tantalor 2y ago
    (semantic argument warning) "Mitigation" is dealing with an outage/breakage after it occurs, to reduce the impact or get system healthy again. You're talking about "prevention" which keeps it from happening at all. Canarying is generic approach to prevention, and should not be skipped. Avoiding the risk entirely (eBPF) would also help prevent outage, but I think we're deluding ourselves to say it "solves" the problem once and for all; systems will still go down due to bad deploys.
  - blinkingled 2y ago
mrpippy 2y ago
> Once Microsoft's eBPF support for Windows becomes production-ready, Windows security software can be ported to eBPF as well. This doesn’t seem grounded in reality. If you follow the link to the “hooks” that Windows eBPF makes available [1], it’s just for incoming packets and socket operations. IOW, MS is expecting you to use the Berkeley Packet Filter for packet filtering. Not for filtering I/O, or object creation/use, or any of the other million places a driver like Crowdstrike’s hooks into the NT kernel. In addition, they need to be in the kernel in order to monitor all the other 3rd party garbage running in kernel-space. ELAM (early-launch anti-malware) loads anti-malware drivers first so they can monitor everything that other drivers do. I highly doubt this is available to eBPF. If Microsoft intends eBPF to be used to replace kernel-space anti-malware drivers, they have a long, long way to go. [1]: https://microsoft.github.io/ebpf-for-windows/ebpf__structs_8h.html#a0f8242763b15ec665eaa47c6add861a0 https://microsoft.github.io/ebpf-for-windows/ebpf__structs_8...
- shahahqq 2y ago
  I hope though that Microsoft will double down on their eBPF support for Windows after this incident.
  - stackskipton 2y ago
    Doubt it. Microsoft is clearly over Windows. They continue to produce it but every release feels like "Ugh, fine, since you are paying me a ton of money." Internally, Microsoft is running more and more workloads on Linux and externally, I've had .Net team tell me more than once that Linux is preferred environment for .Net. SQL Server team continues to push hard for Linux compatibility with every release. EDIT: Windows Desktop gets more love because they clearly see that as important market. I'm talking more Windows Server.
    - WatchDog 2y ago
      SQL Server has supported Linux since 2017
    - kevincox 2y ago
Scene_Cast2 2y ago
How much extra security does this provide on top of HLK?
xyzzy123 2y ago
So many problems though! including commercial monocultures, lack of update consent, blast radius issues, etc etc. There's a commons in our pockets but that is very difficult to regulate for. The will keep putting the gun to your head until you keep choosing the monoculture.
- shahahqq 2y ago
  worrisome indeed that now the world knows how many users are affected by crowdstrike so the bad guys just need to poke deeper there
kevin_nisbet 2y ago
I hate to dispute with someone like Brendan Gregg, but I'm hoping vendors in this space take a more holistic approach to investigating the complete failure chain. I personally tend to get cautious when there is a proposal that x will solve the problem that occurred on y date, especially 3 days after the failure. It may be true, but if we don't do the analysis we could leave ourselves open to blindspots. There may also be plenty of alternative approaches that should be considered and appropriately discarded. I think the part I specifically dispute is the only negative outcome is wasted CPU cycles. That's likely the case for the class of bug, but there are plenty of failure modes where a bad ruleset could badly brick a system and make it hard to recover. That's not to say eBPF based security modules isn't the right choice for many vendors, just that let's understand what risks they do and do not avoid, and what part of the failure chain they particularly address.
- mirashii 2y ago
  Just because you have not been aware of the discussions on this topic that have been happening for years, doesn't mean that they haven't been happening. This isn't some new analysis formed 3 days after an incident, this is the generally accepted consensus among many experts who have been working in the space, introducing these new APIs specifically to improve stability, security, etc. of systems.
- ohmyiv 2y ago
  > I personally tend to get cautious when there is a proposal that x will solve the problem that occurred on y date, especially 3 days after the failure. Microsoft has been working on eBPF for a few years at least. https://opensource.microsoft.com/blog/2021/05/10/making-ebpf-work-on-windows/ https://opensource.microsoft.com/blog/2021/05/10/making-ebpf... https://lwn.net/Articles/857215/ https://lwn.net/Articles/857215/ If you're really concerned, they have discussions and communication channels where you're invited to air your concerns. They're listed on their github: https://github.com/microsoft/ebpf-for-windows https://github.com/microsoft/ebpf-for-windows Who knows, maybe they already have answers to your concerns. If not, they can address them there.
the8472 2y ago
If the filters are loaded at boot and hook into everything then a bug can still lock down the system to a point where it can't be operated or patched anymore (e.g. because you loaded an empty whitelist). So it could end up replacing a boot loop with another form of DoS. If microsoft includes a hardcoded whitelist that covers some essentials needed for recovery that could make a bug in such a tool easier to fix, but could still cause effective downtimes (system running but unusuable) until such a fix is delivered.
twen_ty 2y ago
Can someone tell me what's the advantage of eBPF over a user mode driver? The article makes it look it eBPF is have your cake and eat it too solution which is too good to be true? Can you run graphics drivers in eBPF for example?
- tptacek 2y ago
  No, you can't run arbitrary general-purpose programs in eBPF, and you cannot run graphics drivers in it. You generally can't run programs with unprovably bounded loops in eBPF, and your program can interact with the kernel only through a small series of explicitly enumerated "helpers" (for any given type of eBPF program, you probably have about 20 of these in total).
  - chrisjj 2y ago
    > You generally can't run programs with unprovably bounded loops in eBPF Surely that bars CrowdStrike's check for unprovably bounded vulnerabilities.
- chasil 2y ago
  This is the wiki. I haven't kept up, but this isn't a kernel module. "eBPF is a technology that can run programs in a privileged context such as the operating system kernel. It is the successor to the Berkeley Packet Filter (BPF, with the "e" originally meaning "extended") filtering mechanism in Linux and is also used in non-networking parts of the Linux kernel as well." https://en.wikipedia.org/wiki/EBPF https://en.wikipedia.org/wiki/EBPF
- bewo001 2y ago
  AFAIK, an ebpf function can only access memory it got handed as an argument or as result from a very limited number of kernel functions. Your function will not load if you don't have boundary checks. Fighting the ebpf validator is a bit like fighting Rust's borrow checker; annoying, at times it's too conservative and rejects perfectly correct code, but it will protect you from panics. Loops will only be accepted if the validator can prove they'll end in time; this means it can be a pain to make the validator to accept a loop. Also, ebpf is a processor-independent byte code, so vectorizing code is not possible (unless the byte code interpreter itself does it). Given all its restrictions, I doubt something complex like a graphics driver would be possible. But then, I know nothing about graphics driver programming.
WaitWaitWha 2y ago
eBPF == extended Berkeley Packet Filter https://en.wikipedia.org/wiki/Berkeley_Packet_Filter https://en.wikipedia.org/wiki/Berkeley_Packet_Filter
- kayge 2y ago
  Thanks! This was not a familiar acronym to me... and after some digging[0] apparently it's no longer an acronym: "BPF originally stood for Berkeley Packet Filter, but now that eBPF (extended BPF) can do so much more than packet filtering, the acronym no longer makes sense. eBPF is now considered a standalone term that doesn’t stand for anything." [0] https://ebpf.io/what-is-ebpf/ https://ebpf.io/what-is-ebpf/
CodeWriter23 2y ago
> an unprecedented example of the inherent dangers of kernel programming I take issue with that. Kernel programming was not to blame; looking up addresses from a file and accessing those memory locations without any validation is. The same technique would yield the same result at any Ring.
- lucianbr 2y ago
  Obviously in userspace it would only crash the running program and not the entire operating system? It's a significant difference. All of the service interruptions would have been just "computer temporarily not protected by crowdstrike agent". Not the same thing at all.
  - chrisjj 2y ago
    > Obviously in userspace it would only crash the running program and not the entire operating system? It's a significant difference. Significant and often far worse. It would leave the machine running unprotected.
  - CodeWriter23 2y ago
    > It's a significant difference. When various apps running the world are crashing, unable to execute because malware protection is failing, there is no difference.
    - macobrien 2y ago
      _No_ difference oversells it, IMO -- the fact that the entire OS crashed is what made fixing the bug so arduous, since it required in-person intervention. To be sure, running the code in userspace would still cause unacceptable service interruptions, but the fix could be applied remotely.
- nine_k 2y ago
  At Ring 3 it would crash an app, not the entire OS. Yes, the kernel is fine and is not to blame. But running basically a rootkit controlled by a third party indeed is to blame.
nkozyra 2y ago
I don't do any kernel stuff so I'm out of my element, but doesn't the fact that Crowdstrike & Linux kernel eBPF already caused kernel crashes[1] sort of downplay the rosiness of the state of things? [1]: https://access.redhat.com/solutions/7068083 https://access.redhat.com/solutions/7068083
- guipsp 2y ago
  This is specifically addressed in the post you are replying to
  - nkozyra 2y ago
    Can you elaborate? What I see about Linux is that Crowdstrike was in the process of adopting eBPF which is ostensibly immune to kernel panics, but that issue shows their eBPF implementation specifically causing a kernel panic.
    - olddustytrail 2y ago
      Yes, the elaboration is that the same link you posted is included in the article you're supposed to have just read.
      - nkozyra 2y ago
        I've read it three times now. The only thing they say about it is this: "This doesn't mean that eBPF has solved nothing, substituting a vendor's bug for its own. Fixing these bugs in eBPF means fixing these bugs for all eBPF vendors, and more quickly improving the security of everyone." Which is exactly what I'm asking about. If eBPF has some inherent advantage, why did it fail in precisely the same way alreay?
mschuster91 2y ago
> If your company is paying for commercial software that includes kernel drivers or kernel modules, you can make eBPF a requirement. It's possible for Linux today, and Windows soon. While some vendors have already proactively adopted eBPF (thank you), others might need a little encouragement from their paying customers. How about Microsoft's large government and commercial customers make it a requirement that MS does not develop a single new feature for the next two fucking years or however long it takes to go through the entirety of the Windows+Office+Exchange code base and to make sure there are no security issues in there? We don't need ads in the start menu, we don't need telemetry, we don't need desktop Outlook becoming a rotten slow and useless web app, we don't need AI, we certainly don't need Recall. We need an OS environment that doesn't need a Patch Tuesday where we have to check if the update doesn't break half the canary machines. And while MS is at that they can also take the goddamn time and rework the entire configuration stack. I swear to god, it drives me nuts. There's stuff that's only accessible via the registry (and there is no comprehensive documentation showing exactly what any key in the registry can do - large parts of that are MS-internal!), there's stuff only accessible via GPO, there's stuff hidden in CPLs dating back to Windows 3.11, and there's stuff in Windows' newest UI/settings framework.
throwaway2037 2y ago
The blog post says: > eBPF, which is immune to such crashes. I tried to Google about this, but I cannot find anything definitive. It looks like you can still break things. Can an expert on eBPF please comment on this claim? This is the best that I could find: https://stackoverflow.com/questions/70403212/why-is-ebpf-said-to-be-safer-than-lkm https://stackoverflow.com/questions/70403212/why-is-ebpf-sai...
- School-Cotton 2y ago
  eBPF programs cannot crash the kernel, assuming there are no bugs in the eBPF verifier. There have been such bugs in the past but they seem to be getting more and more rare.
  - throwaway984393 2y ago
    [dead]
  - javierhonduco 2y ago
    Or in other parts of the kernel. It's been the case in multiple occasions that buggy locking (or more generalised, missing 'resource' release) has caused problems for perfectly safe BPF programs. For example, see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1033398 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1033398 and the fix https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/mm/maccess.c?id=d319f344561de23e810515d109c7278919bff7b0 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
    - School-Cotton 2y ago
      This is actually exactly the bug I was thinking of, so fair point! (I work at PS now and am aware you worked on debugging it a while back).
  - rwmj 2y ago
    This isn't really true. eBPF programs in Linux have access to a large set of helper functions written in plain C. https://lwn.net/Articles/856005/ https://lwn.net/Articles/856005/
__MatrixMan__ 2y ago
Maybe we should start taking Fridays off to commemorate the event, which probably would have been less bad if more people spent less time with their nose to the grindstone and had more time to stop and think about how it all was shaping up and how they could influence that shape.
ReleaseCandidat 2y ago
Sorry, but neither eBPF nor Rust nor formal verification nor ... is going to solve that problem. Repeat after me: there are no technical solutions to social problems. As long as the result of such an outage is basically a "oh, a software problem! shrug", _nothing_ will change.
Yawrehto 2y ago
1. How does eBPF solve this? It makes it more difficult, sure, but it'll almost always be possible to cause a crash, if you try hard enough. 2. More importantly, the problem is rarely fixable by changing technology, because typically, problems are caused by people and their connections: social/corporate pressures, profit-seeking, mental health being treated as unimportant, et cetera. eBPF can't fix those, and as long as corporations have social structures that penalize thoroughness and caution, and incentivize getting 'the most stuff' done, this will persist as a problem.
- School-Cotton 2y ago
  > it'll almost always be possible to cause a crash, if you try hard enough. If you think you know a way to crash the Linux kernel by loading and running an eBPF program, you should report a bug.
uticus 2y ago
> eBPF programs cannot crash the entire system because they are safety-checked by a software verifier and are effectively run in a sandbox. Isn’t one of the purposes of an OS to police software? I get that this has to do with the OS itself, but what does watching the watchers accomplish other than adding a layer which must then be watched? Why not reduce complexity instead of naively trusting that the new complexity will be better long term?
- MetaWhirledPeas 2y ago
  Right? I might spend a few minutes seeing if an AI chatbot can explain all the justifications that lead to using something like CrowdStrike in the first place.
- riskable 2y ago
  eBPF isn't "watching the watchers" it's just a tool that lets other tools access low-level things in the kernel via a very picky sandbox. Think of it like this: Old way: Load kernel driver, hook into bazillions of system calls (doing whatever it is you want to do), pray you don't screw anything up (otherwise you can get a panic though not necessarily--Linux is quite robust). eBPF way: Just ask eBPF to tell you what you want by giving it some eBPF-specific instructions. There's a rundown on how it works here: https://ebpf.io/what-is-ebpf/ https://ebpf.io/what-is-ebpf/
  - uticus 2y ago
    > eBPF isn't "watching the watchers"… > …via a very picky sandbox… When the eBPF is a CrowdStrike mechanism, and eBPF is “picky,” it is clearly “watching the watchers.”
risenshinetech 2y ago
Thank God some superheros have finally come along to make sure code never crashes any computers ever again! /s
- chrisjj 2y ago
  Sometimes a point is such that only sarcasm can get it across sufficiently. This is such a point.
aurelien 2y ago
[flagged]
- deleted 2y ago
  [deleted]
klooney 2y ago
First io_uring, now eBPF. Kind of wild.
throwaway984393 2y ago
[dead]
tracker1 2y ago
I don't buy it... didn't a bug from RedHat + Crowdstrike have a similar panic issue? I understand in that case it was because of RedHat, but still. I don't think this, by itself will change much.
kaliszad 2y ago
"These security agents will then be safe and unable to cause a Windows kernel crash." Unless of course there is a bug in eBPF (https://access.redhat.com/solutions/7068083 https://access.redhat.com/solutions/7068083) @brendangregg and the kernel panics/ BSoDs anyway which you mention later in the article of course.
- acdha 2y ago
  This is true but the kernel gets more scrutiny and has better priorities. Only CrowdStrike audits and hardens the CS kernel driver, so things like proactive improvements are competing in a single Jira board against marketing’s request for new features (want to bet that was all AI until Friday?) whereas the kernel eBPF implementation might be improved by people at other security vendors, distributions like Red Hat or Ubuntu or a major cloud provider (all of whom fund serious security audits and have engineers who care a lot about robustness), or academic researchers. “Many eyes” is a bit dubious in general but the Linux kernel is pretty much the best case for it being true.
- ec109685 2y ago
  Benefit of fixing that bug is that all ebpf programs benefit versus every security vendor needing to ensure they write perfect c code.
throw0101d 2y ago
Meta: > eBPF (no longer an acronym) […] Any reason why the official acronym was done away with?
- Jedd 2y ago
  Technically it was never an acronym - rather an initialism or abbreviation.
- riskable 2y ago
  Because it used to stand for extended Berkeley Packet Filter and it has since moved far, far beyond just packets. It now hooks into the entire network stack, security, and does observability/tracing for nearly anything and everything in the kernel ("nearly" because some stuff runs when the kernel boots up--before eBPF is loaded--and never again after that).
- sandywaffles 2y ago
  Because eBPF is no longer just packet filtering? It's now used in loads of hook pionts unrelated to packets or filtering at all.
bfrog 2y ago
I wonder if microkernels ever had this kind of bullshit. Had it been a microkernel, would we all be sitting twiddling our thumbs on friday? Hot take: No.
dveeden2 2y ago
So eBPF is giving us eBFP (enhanced Blue Friday Protection)?
muth02446 2y ago
```The verifier is rigorous -- the Linux implementation has over 20,000 lines of code -- with contributions from industry (e.g., Meta, Isovalent, Google) and academia (e.g., Rutgers University, University of Washington). The safety this provides is a key benefit of eBPF, along with heightened security and lower resource usage. ``` Wow, 20k is not exactly encouraging. Besides the extra attack surface, who can vouch for such a large code base?
- haberman 2y ago
  I had exactly the same thought. I don’t know if that 20k number was supposed to inspire confidence, but for me it did the opposite. It would have inspired confidence if it was 300 lines of code. My impression is that the WebAssembly verifier is much simpler.
  - deleted 2y ago
    [deleted]
brundolf 2y ago
This sounds like a cool technology, but this was the really egregious problem: > There are other ways to reduce risks during software deployment that can be employed as well: canary testing, staged rollouts, and "resilience engineering" in general You don't need a new technology to implement basic industry-standard quality control
odyssey7 2y ago
"The verifier is rigorous" But the appeal-to-authority evidence that the article presents is not. "-- the Linux implementation has over 20,000 lines of code -- with contributions from industry (e.g., Meta, Isovalent, Google) and academia (e.g., Rutgers University, University of Washington). The safety this provides is a key benefit of eBPF, along with heightened security and lower resource usage."
lazycog512 2y ago
"The major difference between a thing that might go wrong and a thing that cannot possibly go wrong is that when a thing that cannot possibly go wrong goes wrong it usually turns out to be impossible to get at and repair." - Douglas Adams
rezonant 2y ago
> the company behind this outage was already in the process of adopting eBPF, which is immune to such crashes Oh I'm sure they'll find a way.
egorfine 2y ago
One option to prevent this is to not run corporate spyware. But I guess for some industries this isn't an option.
- supriyo-biswas 2y ago
  I don’t understand statements like this. You only need to have some employee install some malware (unintentionally or otherwise); and you have a data breach on your hands.
  - egorfine 2y ago
    I agree it's much more scalable to have a vendor install a spyware on all your workstations and have a centralized data breach.
    - chrisjj 2y ago
      Oh, I wouldn't say that. Viruses are highly scalable.
0xbadcafebee 2y ago
> In the future, computers will not crash due to bad software updates I'm still waiting on my flying car...
tgtweak 2y ago
Even if Microsoft rolls out eBPF and mainstreams it - it will be years before everything is ported over and it still won't address legacy windows versions (which appear to be a good chunk of what was impacted). It's a move in the right direction but it probably won't fully mitigate issues like this for another 5+ years.
- acdha 2y ago
  Sure, but 5 years is not that long ago - for example, if they’d started right before the pandemic it’d be almost done by now. The best time to have done that was 5 years ago but the second best time is now.
ksec 2y ago
The article mentions Windows and Linux. Does anyone know if there will be eBPF for FreeBSD?
titzer 2y ago
WebAssembly is a better choice for sandboxing kernel code. It has a full formal specification with a mechanized proof of type safety, many high-performance implementations, broad toolchain support, is targetable from many languages, and a capability security model.
- rapidlua 2y ago
  Hardly. For starters, wasm doesn’t guarantee that a piece of code terminates in bound time. There are further security guarantees in ebpf such as any lock acquired must be released.
  - titzer 2y ago
    You can apply additional static checks to Wasm, e.g. control flow analysis, and reject programs without obvious loop bounds or unbalanced locking operations. Or you could apply dynamic techniques like tracking acquired locks and automatically releasing them, or charging fuel (gas). The latter is quite common for blockchain runtimes based on Wasm.
  - jules 2y ago
    The eBPF termination checker is buggy anyway; you cannot rely on it.
datadeft 2y ago
It is great that we need a linux kernel feature to be ported to Windows so we don’t have blue Fridays
7e 2y ago
eBPF will be an improvement, I’m sure, but does not mean the end of bugs/DoS in software.
wiresurfer 2y ago
Hey Brendan, > If your company is paying for commercial software that includes kernel drivers or kernel modules, you can make eBPF a requirement. Windows soon, may still be atleast a year ahead. Would that be a fair statement? atleast being the operating keyword here. Specifically in the context of network security software, for eBPF programs to be portable across windows/linux, we would need MSFT to add a lot more hooks and expose internal kernel stucts. Hopefully via a common libbpf definition. Otherwise, I fear, having two versions of the same product, across two OSs would mean more secuirty and quality issues. I guess the point I am trying to make is, we would get there, but we are more than a few years away. I would love to see something like cilium on vanilla windows for a Software defined Company Wide network. We can then start building enterprise network secutiry into it. Baby steps! --- btw, your talks and blog posts about bpftools is godsent!

481 points by moreati 2y ago

This puts an awful lot of stock in the robustness of eBPF. Which is odd, given there’s been a bunch of kernel privesc bugs using eBPF…

joker99 2y ago

I used to work for an EDR vendor and this post glosses over two major and important things. 1. There’s no need for eBPF on windows, it has the ETW framework (event tracing) which is much more powerful and provides applications subscribing to a class of events almost too detailed insights. the issue most AV vendors have with it though is speed. Leading to … 2. eBPF lets you watch. Congrats. It’s something, but it’s not the reason why these tools are deployed. Orgs deploy these tools to prevent or stop potentially bad stuff from executing. The only place this can be done in our operating systems is usually the kernel - for that you need kernel level drivers or various other filter drivers. Crowdstrike screwed the pooch here, yes. But after a couple of days I feel like I haven’t read enough blog posts and articles that crap on Microsoft. It’s their job to build a secure operating system, instead they deliver Windows and because they themselves cannot secure windows, they ship defender… and we use tools like falcon like a bandaid for Microsofts bad security practices

deleted 2y ago
[deleted]
thayne 2y ago
> eBPF lets you watch. Congrats. It’s something, but it’s not the reason why these tools are deployed. Orgs deploy these tools to prevent or stop potentially bad stuff from executing eBPF let's you prevent things too. seccomp filters can block syscalls. The bigger problem is the performance you mentioned in 1. Crowdstrike's linux agent can work using eBPF instead of a kernel module, and will fall back to that if the current kernel version is more recent than the agent supports. But... then it uses up a lot more CPU.

kjellsbells 2y ago

Lets suppose that eBPF solves this particular problem, eventually, for Windows. Doesn't sidestepping the entire class of Crowdstrike-style fubars require that Microsoft then mandate that no, backward compatibility will not be offered? Back compat seems to be such a shibboleth in the Windows world, but comes at an incredible price. The reasons cited all seem to boil down to keeping some imagined customers' obscure LOB app running for decades. But that seems like an excuse to me. Surely Microsoft would like to shake out the last diehards running some VB5 app on a patched up PC in a factory. Isn't it more beneficial to everyone to start sunsetting acres of ancient NT code and approaches and streamline the entire attack surface?

pas 2y ago
it would be enough if MS offered knobs and switches for admins/devs/vendors to disallow non-static-verified stuff in the kernel
acdha 2y ago
Backwards compatibility slows things down in the Windows world but it doesn’t halt improvements. In this case, there are two powerful ratchets: 1. Compliance: everyone affected by this bug has auditors. Once safer alternatives are available, the standards like CIS, PCI, etc. will be updated to say you should use the new interface, and every enterprise IT department will have pressure to switch to eBPF tools. We saw this with BootLocker: storage encryption used to be a pain, people resisted it, but over time it became universal because the cost of swimming upstream was too high. 2. Signing. Microsoft can start requiring more proof of need and restrictions for signing drivers. They have to be careful to avoid the appearance of favoritism but after this debacle that’s a LOT easier. I would bet some engineer is working on a draft of mandatory fault handling and testing proof requirements for critical kernel drivers now and I would not be surprised to see it include a timeframe for adopting memory-safe languages.
another2another 2y ago
>Surely Microsoft would like to shake out the last diehards running some VB5 app on a patched up PC in a factory. >Isn't it more beneficial to everyone to start sunsetting acres of ancient NT code and approaches and streamline the entire attack surface? If your code somehow still relies on some buggy behaviour to work, then MS shouldn't do anything to preserve that anymore - apparently they used to, but I'm not so sure nowadays. However 'ancient NT' code should probably still function just fine since the Win32 API hasn't changed much for a while, and MS don't actively deprecate function calls (unlike Apple who seem to do it a bit on a whim recently). I would put this down to the API being pretty well designed in the first place.

3np 2y ago

> The worst thing an eBPF program can do is to merely consume more resources than is desirable, such as CPU cycles and memory. This is obviously not true. It might be the worst it can do, by itself, to the currently running kernel. It's not the worst it can do to the machine or its user(s). There are infinite harmful things an eBPF program can do. As can programs solely in user-space. There is a specific class of vulnerabilities being mitigated by moving code from kernel to BPF. That does not mean that eBPF programs are in general safe.

userbinator 2y ago

In the future, computers will not crash due to bad software updates, even those updates that involve kernel code. 100% BS. Even if they don't "crash" they will "stop functioning as intended" which is just the same. It's absolutely disgusting how this industry is now using this one outage as a talking point to further their totalitarian agenda. It reminds me of how Google went after adblockers with their new extension model that also promised more "security". It's time we realised what they're really trying to do. In fact, I wonder whether this outage was not accidental after all.

yubiox 2y ago

Title reminds me of when microsoft promised no more UAEs back in 92. They just renamed them to GPFs in windows 3.1.

jeffrallen 2y ago

Here's an idea for an interesting hack: a piece of kernel resident code that feeds fake data into eBPF so that an eBPF-based antimalware will see nothing bad as the malware goes about it's merry way. Sandboxes are safe, but are ultimately virtual machines, and virtual machines can be made to live in a world that's not real.

amluto 2y ago

> In the future, computers will not crash due to bad software updates, even those updates that involve kernel code. In the future, these updates will push eBPF code. eBPF is fantastic, and it can be used for many purposes and improve a lot of things, but this is IMO overselling it. Assuming that BPF itself it free of bugs, it’s still a rather large sprawl of kernel hooks, and those hooks invoke eBPF code, which can call right back into the kernel. Here’s a list: https://www.man7.org/linux/man-pages/man7/bpf-helpers.7.html https://www.man7.org/linux/man-pages/man7/bpf-helpers.7.html bpf_probe_read_kernel() is particularly heavily used, and it is not safe. It tries fairly hard not to OOPS or crash, but it is definitely not perfect. The rest of that list contains plenty of this that will easily take down a system, even if it doesn’t actually oops or panic in the process. And, of course, any tool that detects userspace “malicious behavior” and stops it can start calling everything malicious, and the computer becomes unusable. Meanwhile, eBPF has no real security model on the userspace side. Actual attachment of an eBPF program goes through the bpf() syscall, not through sensibly permissioned operations on the underlying kernel objects being attached to, and there is nothing whatsoever that confines eBPF to, say, a container that uses it. (See bpf_probe_read_kernel() -- it's fundamentally able to read all kernel memory.) So, IMO, most of the benefit of eBPF over ordinary kernel C code is that eBPF is kind of like writing code in a safe language with a limited unsafe API surface. It's a huge improvement for this sort of work, but it is not perfect by any means. > The verifier is rigorous -- the Linux implementation has over 20,000 lines of code The verifier is absurdly complex. I'd rather see something based on formal methods than 20kLOC of hand-written logic.

School-Cotton 2y ago
How is it possible to panic using bpf_probe_read_kernel ? Can you give an example that works on the current kernel version?
- amluto 2y ago
  I'm not sure that "panic" is the right word here. bpf_probe_read_kernel boils down to copy_from_kernel_nofault, which checks for an "allowed" address and then does the access. Any page faults turn into error returns instead of OOPSes. x86 disallows user addresses, the vsyscall page, and non canonical addresses. Doing this from bpf assumes that all "allowed" addresses are side-effect-free and will either succeed or cleanly fault. Off the top of my head, MMIO space (including, oddities like the APIC page on CPUs that still have that) and TDX memory are not in this category.

ninju 2y ago

So a couple of questions 1) Is CrowdStrike Falcon using eBPF for their Linux offering? 2) Would the faulty patch update get caught by the eBPF verifier?

xg15 2y ago

> In the future, computers will not crash due to bad software updates, even those updates that involve kernel code. In the future, these updates will push eBPF code. Assuming every security critical system will be on a recent enough kernel to support this...

efee22 2y ago
I think with a LTS distribution you should get very far these days when it comes to implementing such sensors.
- chasil 2y ago
  On rhel8 variants, you can use the Oracle UEK to get eBPF. https://blogs.oracle.com/linux/post/oracle-linux-and-bpf https://blogs.oracle.com/linux/post/oracle-linux-and-bpf $ cat /etc/redhat-release /etc/oracle-release /proc/version Red Hat Enterprise Linux release 8.10 (Ootpa) Oracle Linux Server release 8.10 Linux version 5.15.0-203.146.5.1.el8uek.x86_64 (mockbuild@host-100-100-224-48) (gcc (GCC) 11.2.1 20220127 (Red Hat 11.2.1-9.2.0.1), GNU ld version 2.36.1-4.0.1.el8_6) #2 SMP Thu Feb 8 17:14:39 PST 2024
dijit 2y ago
And assuming there's no bugs in the BPF code... Oh wait: https://news.ycombinator.com/item?id=41031699 https://news.ycombinator.com/item?id=41031699
- efee22 2y ago
  RHEL kernel.. right. Imho, I'd trust an upstream stable kernel far more than a RHEL one for production which has dozen of feature backports and an internal kABI to maintain.. granted RH has a QA team, but it is still impossible to test everything beforehand.
  - worthless-trash 2y ago
    On the upside, non root users can't insert ebpf code, so its a priv'ed operation, not like other distros.

usrme 2y ago

Does anyone know how far along the eBPF implementation for Windows actually is? In the sense that it could start feasibly replacing existing kernel drivers.

CoastalCoder 2y ago

> If your company is paying for commercial software that includes kernel drivers or kernel modules, you can make eBPF a requirement. Are they saying that device drivers should be written in eBPF? Or maybe their drivers should expose an eBPF API? I assume some driver code still needs to reside in the actual kernel.

prmoustache 2y ago
These tool wouldn't need kernel drivers, only to target the eBPF userspace API: https://www.kernel.org/doc/html/latest/userspace-api/ebpf/index.html https://www.kernel.org/doc/html/latest/userspace-api/ebpf/in...

asynchronous 2y ago

Is there a reason for the lack of naming+shaming Crowdstrike in this blogpost? Was it to not give them any more publicity, good or bad?

StevenWaterman 2y ago
If you consider kernel programming to be inherently unsafe, then you would consider this to be inevitable, meaning it's not really the specific company's fault. They were just the unlucky ones.
- efee22 2y ago
  Agree, Crowdstrike was an unlucky one, but it is more about the issue in general. If I remember correctly, also others like sysdig user their own kernel modules for collection.
- asynchronous 2y ago
  I still hold true that testing even improperly would have caught this before it hit worldwide. But I suppose you are right, that doesn’t help the argument being made here.
  - ForOldHack 2y ago
    Wasnt that the job of AI/co-pilot/clippy /D.E.P? "Would you like me to try and execute a random blank file?" And of course QA. I was unaffected, but was fielding calls from customers. My update Tuesday is the week after, so in-between MS and my updates, I am very suspicious of everything. I was also unaffected by 22H2, and spent time fielding calls.
- lordnacho 2y ago
  They could have helped their luck by doing some of the common sense things suggested in the article. For instance, why not find a subset of your customers that are low risk, push it out to them, and see what happens? Or perhaps have your own fleet of example installations to run things on first. None of which depends on any specific technology.

kayo_20211030 2y ago

This isn't right. If I need a system to run with a piece of code, then it shouldn't run at all if that piece of code is broken. Ignoring the failure is perverse. Let's say that the driver code ensures that some medical machine has safety locks (safeguards) in place to make sure that piece of equipment won't fry you to a crisp; I'd prefer that the whole thing not run at all rather than blithely operate with the safeguards disabled. It's turtles all the way down.

emn13 2y ago
The system clearly already behaves that way (i.e. ignores failure) - after all, the fix was to simply delete the offending file. If that's an option, then loader can do that too. It can and perhaps even is smarter, such as "fallback onto previous version". Furthermore, the reaction to a malformed state need not be "ignore". It could disable restricted user login; or turn off the screen. If the worry is that this is viable to abuse by malware, well, if the malware can already rewrite the on-disk files for the AV, I wonder whether it's really a good idea to trust the system itself to be able to deal with that. It'd probably be safer to just report that up the security foodchain, and potentially let some external system take measures such as disable or restrict network access. Better yet, such measures don't even require the same capabilities to intervene in the system, merely to observe - which makes the AV system less likely to serve as a malware vector itself or to cause bugs like this.
Smaug123 2y ago
I think the premise is false? It's up to the eBPF implementor what to do in the case of invalid input; the kernel could choose to perform a controlled shutdown in that case. (I have no idea what e.g. Linux actually does here, but one could imagine worlds where the action it takes on invalid input is configurable.) Also your statement is sometimes not true, although I certainly sympathise in the mainline case. In some contexts you really do need to keep on trucking. The first example to spring to mind is "the guidance computers on an automated Mars lander"; the round-trip to Earth is simply too long to defer responsibility in that case. If you shut down then you will crash, but if you do your best from a corrupted state then you merely probably crash, which is presumably better.

aurelien 2y ago

[flagged]

shrx 2y ago

From the article: > If the verifier finds any unsafe code, the program is rejected and not executed. The verifier is rigorous -- the Linux implementation has over 20,000 lines of code [0] -- with contributions from industry (e.g., Meta, Isovalent, Google) and academia (e.g., Rutgers University, University of Washington). [0] links to https://github.com/torvalds/linux/blob/master/kernel/bpf/verifier.c https://github.com/torvalds/linux/blob/master/kernel/bpf/ver... which has this interesting comment at the top: /* bpf_check() is a static code analyzer that walks eBPF program * instruction by instruction and updates register/stack state. * All paths of conditional branches are analyzed until 'bpf_exit' insn. * * The first pass is depth-first-search to check that the program is a DAG. * It rejects the following programs: * - larger than BPF_MAXINSNS insns * - if loop is present (detected via back-edge) ... I haven't inspected the code, but I thought that checking for infinite loops would imply solving the halting problem. Where's the catch?

ahepp 2y ago
If you’re wrong about the loop, you’ll still hit BPF_MAXINSNS, so it’s fine to use heuristics that could produce a false negative right?
deleted 2y ago
[deleted]
dtx1 2y ago
I have no insight into this particular project but you could work around the halting problem by only allowing loops you can proof will not go infinite. That would of course imply rejecting loops that won't go infinite but can't be proven not to.
hiddencost 2y ago
Unterminated loops might be a better phrasing.
efee22 2y ago
Infinite loops are not possible and would get rejected by the verifier since it cannot solve the halting problem. Here is a good overview on the options available: https://ebpf-docs.dylanreimerink.nl/linux/concepts/loops/ https://ebpf-docs.dylanreimerink.nl/linux/concepts/loops/

skywhopper 2y ago

The implicit assumption of the article is that eBPF code can't crash a kernel, but the article itself eventually admits that it can and has done, including last month. eBPF is a safer way of providing kernel-extension functionality, for sure, but presenting it as the perfect solution is just asking to have your argument dismissed. eBPF is not perfect. And there's plenty of things it can't do. The very sandbox rules that limit how long its programs may run and what they can do also make it entirely inappropriate for certain tasks. Let's please stop pretending there's a silver bullet.

efee22 2y ago
It's not a silver bullet, however, it is still better to pushing all the panicable bugs into one community-maintained section (e.g. eBPF verifier). All vendors have an incentive to help get right and this is much better than every vendor shipping their own panicable bugs in their own out of tree kernel modules. Additionally, it's not just the industry looking at eBPF, but also academia in terms of formally verifying these critical sections.
- lucianbr 2y ago
  "Improves kernel stability" is great. "Prevents kernel crashes" is a plain lie. > In the future, computers will not crash due to bad software updates, even those updates that involve kernel code. Come on. Computers will continue to crash in the future, even when using eBPF. I am quite certain.
lucianbr 2y ago
It's casually claiming to have solved the halting problem, at least within some limited but useful context. That should be impossible, and it turns out, it is. I expect it can be solved within some limited contexts, but those contexts are not useful, at least not at the level of "generic kernel code".
- red_admiral 2y ago
  It solves the halting problem by not being Turing complete. I presume each eBPF runs in a context with bounded memory, requested up front, for one thing; it also disallows jumps unless you can prove the code still halts.

vfclists 2y ago

Yep, another fix to all our problems, a new bandwagon to be jumped on by wall EDR vendors, until ... Here I am using the term "EDR". Until this CrowdStrike debacle I'd never heard it. Only tells how seriously you should take my opinions.

blinkingled 2y ago

Ok. But the good old push code to staging / canary it before mainstream updates was a simpler way of solving the same problem. Crowdstrike knows the computers they're running on, it is trivial to implement a system where only few designated computers download and install the update and report metrics before the update controller decides to push it to next set.

phartenfeller 2y ago
Why trust somebody else not messing up? With that in place for windows and crowdstrike billions of dollars would be saved and many lives not negatively impacted ...
rldjbpin 2y ago
with the way they handled the debian crashing a little while ago, frankly they are happy to still go ahead with testing this way. still much better way to handle things than pushing to everybody at the same time.
Archelaos 2y ago
It would mitigate the problem, but not solve it. You can still imagine a condition that only occurs after the update has been rolled out everywhere. Furthermore, such a bug would still be extremely problematic for the concerned customers, even if not all of them were affected. In addition, it would be necessary to react very quickly in the case of zero-day vulnerabilities.
- tantalor 2y ago
  (semantic argument warning) "Mitigation" is dealing with an outage/breakage after it occurs, to reduce the impact or get system healthy again. You're talking about "prevention" which keeps it from happening at all. Canarying is generic approach to prevention, and should not be skipped. Avoiding the risk entirely (eBPF) would also help prevent outage, but I think we're deluding ourselves to say it "solves" the problem once and for all; systems will still go down due to bad deploys.
- blinkingled 2y ago

mrpippy 2y ago

> Once Microsoft's eBPF support for Windows becomes production-ready, Windows security software can be ported to eBPF as well. This doesn’t seem grounded in reality. If you follow the link to the “hooks” that Windows eBPF makes available [1], it’s just for incoming packets and socket operations. IOW, MS is expecting you to use the Berkeley Packet Filter for packet filtering. Not for filtering I/O, or object creation/use, or any of the other million places a driver like Crowdstrike’s hooks into the NT kernel. In addition, they need to be in the kernel in order to monitor all the other 3rd party garbage running in kernel-space. ELAM (early-launch anti-malware) loads anti-malware drivers first so they can monitor everything that other drivers do. I highly doubt this is available to eBPF. If Microsoft intends eBPF to be used to replace kernel-space anti-malware drivers, they have a long, long way to go. [1]: https://microsoft.github.io/ebpf-for-windows/ebpf__structs_8h.html#a0f8242763b15ec665eaa47c6add861a0 https://microsoft.github.io/ebpf-for-windows/ebpf__structs_8...

shahahqq 2y ago
I hope though that Microsoft will double down on their eBPF support for Windows after this incident.
- stackskipton 2y ago
  Doubt it. Microsoft is clearly over Windows. They continue to produce it but every release feels like "Ugh, fine, since you are paying me a ton of money." Internally, Microsoft is running more and more workloads on Linux and externally, I've had .Net team tell me more than once that Linux is preferred environment for .Net. SQL Server team continues to push hard for Linux compatibility with every release. EDIT: Windows Desktop gets more love because they clearly see that as important market. I'm talking more Windows Server.
  - WatchDog 2y ago
    SQL Server has supported Linux since 2017
  - kevincox 2y ago

Scene_Cast2 2y ago

How much extra security does this provide on top of HLK?

xyzzy123 2y ago

So many problems though! including commercial monocultures, lack of update consent, blast radius issues, etc etc. There's a commons in our pockets but that is very difficult to regulate for. The will keep putting the gun to your head until you keep choosing the monoculture.

shahahqq 2y ago
worrisome indeed that now the world knows how many users are affected by crowdstrike so the bad guys just need to poke deeper there

kevin_nisbet 2y ago

I hate to dispute with someone like Brendan Gregg, but I'm hoping vendors in this space take a more holistic approach to investigating the complete failure chain. I personally tend to get cautious when there is a proposal that x will solve the problem that occurred on y date, especially 3 days after the failure. It may be true, but if we don't do the analysis we could leave ourselves open to blindspots. There may also be plenty of alternative approaches that should be considered and appropriately discarded. I think the part I specifically dispute is the only negative outcome is wasted CPU cycles. That's likely the case for the class of bug, but there are plenty of failure modes where a bad ruleset could badly brick a system and make it hard to recover. That's not to say eBPF based security modules isn't the right choice for many vendors, just that let's understand what risks they do and do not avoid, and what part of the failure chain they particularly address.

mirashii 2y ago
Just because you have not been aware of the discussions on this topic that have been happening for years, doesn't mean that they haven't been happening. This isn't some new analysis formed 3 days after an incident, this is the generally accepted consensus among many experts who have been working in the space, introducing these new APIs specifically to improve stability, security, etc. of systems.
ohmyiv 2y ago
> I personally tend to get cautious when there is a proposal that x will solve the problem that occurred on y date, especially 3 days after the failure. Microsoft has been working on eBPF for a few years at least. https://opensource.microsoft.com/blog/2021/05/10/making-ebpf-work-on-windows/ https://opensource.microsoft.com/blog/2021/05/10/making-ebpf... https://lwn.net/Articles/857215/ https://lwn.net/Articles/857215/ If you're really concerned, they have discussions and communication channels where you're invited to air your concerns. They're listed on their github: https://github.com/microsoft/ebpf-for-windows https://github.com/microsoft/ebpf-for-windows Who knows, maybe they already have answers to your concerns. If not, they can address them there.

the8472 2y ago

If the filters are loaded at boot and hook into everything then a bug can still lock down the system to a point where it can't be operated or patched anymore (e.g. because you loaded an empty whitelist). So it could end up replacing a boot loop with another form of DoS. If microsoft includes a hardcoded whitelist that covers some essentials needed for recovery that could make a bug in such a tool easier to fix, but could still cause effective downtimes (system running but unusuable) until such a fix is delivered.

twen_ty 2y ago

Can someone tell me what's the advantage of eBPF over a user mode driver? The article makes it look it eBPF is have your cake and eat it too solution which is too good to be true? Can you run graphics drivers in eBPF for example?

tptacek 2y ago
No, you can't run arbitrary general-purpose programs in eBPF, and you cannot run graphics drivers in it. You generally can't run programs with unprovably bounded loops in eBPF, and your program can interact with the kernel only through a small series of explicitly enumerated "helpers" (for any given type of eBPF program, you probably have about 20 of these in total).
- chrisjj 2y ago
  > You generally can't run programs with unprovably bounded loops in eBPF Surely that bars CrowdStrike's check for unprovably bounded vulnerabilities.
chasil 2y ago
This is the wiki. I haven't kept up, but this isn't a kernel module. "eBPF is a technology that can run programs in a privileged context such as the operating system kernel. It is the successor to the Berkeley Packet Filter (BPF, with the "e" originally meaning "extended") filtering mechanism in Linux and is also used in non-networking parts of the Linux kernel as well." https://en.wikipedia.org/wiki/EBPF https://en.wikipedia.org/wiki/EBPF
bewo001 2y ago
AFAIK, an ebpf function can only access memory it got handed as an argument or as result from a very limited number of kernel functions. Your function will not load if you don't have boundary checks. Fighting the ebpf validator is a bit like fighting Rust's borrow checker; annoying, at times it's too conservative and rejects perfectly correct code, but it will protect you from panics. Loops will only be accepted if the validator can prove they'll end in time; this means it can be a pain to make the validator to accept a loop. Also, ebpf is a processor-independent byte code, so vectorizing code is not possible (unless the byte code interpreter itself does it). Given all its restrictions, I doubt something complex like a graphics driver would be possible. But then, I know nothing about graphics driver programming.

WaitWaitWha 2y ago

eBPF == extended Berkeley Packet Filter https://en.wikipedia.org/wiki/Berkeley_Packet_Filter https://en.wikipedia.org/wiki/Berkeley_Packet_Filter

kayge 2y ago
Thanks! This was not a familiar acronym to me... and after some digging[0] apparently it's no longer an acronym: "BPF originally stood for Berkeley Packet Filter, but now that eBPF (extended BPF) can do so much more than packet filtering, the acronym no longer makes sense. eBPF is now considered a standalone term that doesn’t stand for anything." [0] https://ebpf.io/what-is-ebpf/ https://ebpf.io/what-is-ebpf/

CodeWriter23 2y ago

> an unprecedented example of the inherent dangers of kernel programming I take issue with that. Kernel programming was not to blame; looking up addresses from a file and accessing those memory locations without any validation is. The same technique would yield the same result at any Ring.

lucianbr 2y ago
Obviously in userspace it would only crash the running program and not the entire operating system? It's a significant difference. All of the service interruptions would have been just "computer temporarily not protected by crowdstrike agent". Not the same thing at all.
- chrisjj 2y ago
  > Obviously in userspace it would only crash the running program and not the entire operating system? It's a significant difference. Significant and often far worse. It would leave the machine running unprotected.
- CodeWriter23 2y ago
  > It's a significant difference. When various apps running the world are crashing, unable to execute because malware protection is failing, there is no difference.
  - macobrien 2y ago
    _No_ difference oversells it, IMO -- the fact that the entire OS crashed is what made fixing the bug so arduous, since it required in-person intervention. To be sure, running the code in userspace would still cause unacceptable service interruptions, but the fix could be applied remotely.
nine_k 2y ago
At Ring 3 it would crash an app, not the entire OS. Yes, the kernel is fine and is not to blame. But running basically a rootkit controlled by a third party indeed is to blame.

nkozyra 2y ago

I don't do any kernel stuff so I'm out of my element, but doesn't the fact that Crowdstrike & Linux kernel eBPF already caused kernel crashes[1] sort of downplay the rosiness of the state of things? [1]: https://access.redhat.com/solutions/7068083 https://access.redhat.com/solutions/7068083

guipsp 2y ago
This is specifically addressed in the post you are replying to
- nkozyra 2y ago
  Can you elaborate? What I see about Linux is that Crowdstrike was in the process of adopting eBPF which is ostensibly immune to kernel panics, but that issue shows their eBPF implementation specifically causing a kernel panic.
  - olddustytrail 2y ago
    Yes, the elaboration is that the same link you posted is included in the article you're supposed to have just read.
    - nkozyra 2y ago
      I've read it three times now. The only thing they say about it is this: "This doesn't mean that eBPF has solved nothing, substituting a vendor's bug for its own. Fixing these bugs in eBPF means fixing these bugs for all eBPF vendors, and more quickly improving the security of everyone." Which is exactly what I'm asking about. If eBPF has some inherent advantage, why did it fail in precisely the same way alreay?

mschuster91 2y ago

> If your company is paying for commercial software that includes kernel drivers or kernel modules, you can make eBPF a requirement. It's possible for Linux today, and Windows soon. While some vendors have already proactively adopted eBPF (thank you), others might need a little encouragement from their paying customers. How about Microsoft's large government and commercial customers make it a requirement that MS does not develop a single new feature for the next two fucking years or however long it takes to go through the entirety of the Windows+Office+Exchange code base and to make sure there are no security issues in there? We don't need ads in the start menu, we don't need telemetry, we don't need desktop Outlook becoming a rotten slow and useless web app, we don't need AI, we certainly don't need Recall. We need an OS environment that doesn't need a Patch Tuesday where we have to check if the update doesn't break half the canary machines. And while MS is at that they can also take the goddamn time and rework the entire configuration stack. I swear to god, it drives me nuts. There's stuff that's only accessible via the registry (and there is no comprehensive documentation showing exactly what any key in the registry can do - large parts of that are MS-internal!), there's stuff only accessible via GPO, there's stuff hidden in CPLs dating back to Windows 3.11, and there's stuff in Windows' newest UI/settings framework.

throwaway2037 2y ago

The blog post says: > eBPF, which is immune to such crashes. I tried to Google about this, but I cannot find anything definitive. It looks like you can still break things. Can an expert on eBPF please comment on this claim? This is the best that I could find: https://stackoverflow.com/questions/70403212/why-is-ebpf-said-to-be-safer-than-lkm https://stackoverflow.com/questions/70403212/why-is-ebpf-sai...

School-Cotton 2y ago
eBPF programs cannot crash the kernel, assuming there are no bugs in the eBPF verifier. There have been such bugs in the past but they seem to be getting more and more rare.
- throwaway984393 2y ago
  [dead]
- javierhonduco 2y ago
  Or in other parts of the kernel. It's been the case in multiple occasions that buggy locking (or more generalised, missing 'resource' release) has caused problems for perfectly safe BPF programs. For example, see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1033398 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1033398 and the fix https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/mm/maccess.c?id=d319f344561de23e810515d109c7278919bff7b0 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
  - School-Cotton 2y ago
    This is actually exactly the bug I was thinking of, so fair point! (I work at PS now and am aware you worked on debugging it a while back).
- rwmj 2y ago
  This isn't really true. eBPF programs in Linux have access to a large set of helper functions written in plain C. https://lwn.net/Articles/856005/ https://lwn.net/Articles/856005/

__MatrixMan__ 2y ago

Maybe we should start taking Fridays off to commemorate the event, which probably would have been less bad if more people spent less time with their nose to the grindstone and had more time to stop and think about how it all was shaping up and how they could influence that shape.

ReleaseCandidat 2y ago

Sorry, but neither eBPF nor Rust nor formal verification nor ... is going to solve that problem. Repeat after me: there are no technical solutions to social problems. As long as the result of such an outage is basically a "oh, a software problem! shrug", _nothing_ will change.

Yawrehto 2y ago

1. How does eBPF solve this? It makes it more difficult, sure, but it'll almost always be possible to cause a crash, if you try hard enough. 2. More importantly, the problem is rarely fixable by changing technology, because typically, problems are caused by people and their connections: social/corporate pressures, profit-seeking, mental health being treated as unimportant, et cetera. eBPF can't fix those, and as long as corporations have social structures that penalize thoroughness and caution, and incentivize getting 'the most stuff' done, this will persist as a problem.

School-Cotton 2y ago
> it'll almost always be possible to cause a crash, if you try hard enough. If you think you know a way to crash the Linux kernel by loading and running an eBPF program, you should report a bug.

uticus 2y ago

> eBPF programs cannot crash the entire system because they are safety-checked by a software verifier and are effectively run in a sandbox. Isn’t one of the purposes of an OS to police software? I get that this has to do with the OS itself, but what does watching the watchers accomplish other than adding a layer which must then be watched? Why not reduce complexity instead of naively trusting that the new complexity will be better long term?

MetaWhirledPeas 2y ago
Right? I might spend a few minutes seeing if an AI chatbot can explain all the justifications that lead to using something like CrowdStrike in the first place.
riskable 2y ago
eBPF isn't "watching the watchers" it's just a tool that lets other tools access low-level things in the kernel via a very picky sandbox. Think of it like this: Old way: Load kernel driver, hook into bazillions of system calls (doing whatever it is you want to do), pray you don't screw anything up (otherwise you can get a panic though not necessarily--Linux is quite robust). eBPF way: Just ask eBPF to tell you what you want by giving it some eBPF-specific instructions. There's a rundown on how it works here: https://ebpf.io/what-is-ebpf/ https://ebpf.io/what-is-ebpf/
- uticus 2y ago
  > eBPF isn't "watching the watchers"… > …via a very picky sandbox… When the eBPF is a CrowdStrike mechanism, and eBPF is “picky,” it is clearly “watching the watchers.”

risenshinetech 2y ago

Thank God some superheros have finally come along to make sure code never crashes any computers ever again! /s

chrisjj 2y ago
Sometimes a point is such that only sarcasm can get it across sufficiently. This is such a point.

aurelien 2y ago

[flagged]

deleted 2y ago
[deleted]

klooney 2y ago

First io_uring, now eBPF. Kind of wild.

throwaway984393 2y ago

[dead]

tracker1 2y ago

I don't buy it... didn't a bug from RedHat + Crowdstrike have a similar panic issue? I understand in that case it was because of RedHat, but still. I don't think this, by itself will change much.

kaliszad 2y ago

"These security agents will then be safe and unable to cause a Windows kernel crash." Unless of course there is a bug in eBPF (https://access.redhat.com/solutions/7068083 https://access.redhat.com/solutions/7068083) @brendangregg and the kernel panics/ BSoDs anyway which you mention later in the article of course.

acdha 2y ago
This is true but the kernel gets more scrutiny and has better priorities. Only CrowdStrike audits and hardens the CS kernel driver, so things like proactive improvements are competing in a single Jira board against marketing’s request for new features (want to bet that was all AI until Friday?) whereas the kernel eBPF implementation might be improved by people at other security vendors, distributions like Red Hat or Ubuntu or a major cloud provider (all of whom fund serious security audits and have engineers who care a lot about robustness), or academic researchers. “Many eyes” is a bit dubious in general but the Linux kernel is pretty much the best case for it being true.
ec109685 2y ago
Benefit of fixing that bug is that all ebpf programs benefit versus every security vendor needing to ensure they write perfect c code.

throw0101d 2y ago

Meta: > eBPF (no longer an acronym) […] Any reason why the official acronym was done away with?

Jedd 2y ago
Technically it was never an acronym - rather an initialism or abbreviation.
riskable 2y ago
Because it used to stand for extended Berkeley Packet Filter and it has since moved far, far beyond just packets. It now hooks into the entire network stack, security, and does observability/tracing for nearly anything and everything in the kernel ("nearly" because some stuff runs when the kernel boots up--before eBPF is loaded--and never again after that).
sandywaffles 2y ago
Because eBPF is no longer just packet filtering? It's now used in loads of hook pionts unrelated to packets or filtering at all.

bfrog 2y ago

I wonder if microkernels ever had this kind of bullshit. Had it been a microkernel, would we all be sitting twiddling our thumbs on friday? Hot take: No.

dveeden2 2y ago

So eBPF is giving us eBFP (enhanced Blue Friday Protection)?

muth02446 2y ago

```The verifier is rigorous -- the Linux implementation has over 20,000 lines of code -- with contributions from industry (e.g., Meta, Isovalent, Google) and academia (e.g., Rutgers University, University of Washington). The safety this provides is a key benefit of eBPF, along with heightened security and lower resource usage. ``` Wow, 20k is not exactly encouraging. Besides the extra attack surface, who can vouch for such a large code base?

haberman 2y ago
I had exactly the same thought. I don’t know if that 20k number was supposed to inspire confidence, but for me it did the opposite. It would have inspired confidence if it was 300 lines of code. My impression is that the WebAssembly verifier is much simpler.
- deleted 2y ago
  [deleted]

brundolf 2y ago

This sounds like a cool technology, but this was the really egregious problem: > There are other ways to reduce risks during software deployment that can be employed as well: canary testing, staged rollouts, and "resilience engineering" in general You don't need a new technology to implement basic industry-standard quality control

odyssey7 2y ago

"The verifier is rigorous" But the appeal-to-authority evidence that the article presents is not. "-- the Linux implementation has over 20,000 lines of code -- with contributions from industry (e.g., Meta, Isovalent, Google) and academia (e.g., Rutgers University, University of Washington). The safety this provides is a key benefit of eBPF, along with heightened security and lower resource usage."

lazycog512 2y ago

"The major difference between a thing that might go wrong and a thing that cannot possibly go wrong is that when a thing that cannot possibly go wrong goes wrong it usually turns out to be impossible to get at and repair." - Douglas Adams

rezonant 2y ago

> the company behind this outage was already in the process of adopting eBPF, which is immune to such crashes Oh I'm sure they'll find a way.

egorfine 2y ago

One option to prevent this is to not run corporate spyware. But I guess for some industries this isn't an option.

supriyo-biswas 2y ago
I don’t understand statements like this. You only need to have some employee install some malware (unintentionally or otherwise); and you have a data breach on your hands.
- egorfine 2y ago
  I agree it's much more scalable to have a vendor install a spyware on all your workstations and have a centralized data breach.
  - chrisjj 2y ago
    Oh, I wouldn't say that. Viruses are highly scalable.

0xbadcafebee 2y ago

> In the future, computers will not crash due to bad software updates I'm still waiting on my flying car...

tgtweak 2y ago

Even if Microsoft rolls out eBPF and mainstreams it - it will be years before everything is ported over and it still won't address legacy windows versions (which appear to be a good chunk of what was impacted). It's a move in the right direction but it probably won't fully mitigate issues like this for another 5+ years.

acdha 2y ago
Sure, but 5 years is not that long ago - for example, if they’d started right before the pandemic it’d be almost done by now. The best time to have done that was 5 years ago but the second best time is now.

ksec 2y ago

The article mentions Windows and Linux. Does anyone know if there will be eBPF for FreeBSD?

titzer 2y ago

WebAssembly is a better choice for sandboxing kernel code. It has a full formal specification with a mechanized proof of type safety, many high-performance implementations, broad toolchain support, is targetable from many languages, and a capability security model.

rapidlua 2y ago
Hardly. For starters, wasm doesn’t guarantee that a piece of code terminates in bound time. There are further security guarantees in ebpf such as any lock acquired must be released.
- titzer 2y ago
  You can apply additional static checks to Wasm, e.g. control flow analysis, and reject programs without obvious loop bounds or unbalanced locking operations. Or you could apply dynamic techniques like tracking acquired locks and automatically releasing them, or charging fuel (gas). The latter is quite common for blockchain runtimes based on Wasm.
- jules 2y ago
  The eBPF termination checker is buggy anyway; you cannot rely on it.

datadeft 2y ago

It is great that we need a linux kernel feature to be ported to Windows so we don’t have blue Fridays

7e 2y ago

eBPF will be an improvement, I’m sure, but does not mean the end of bugs/DoS in software.

wiresurfer 2y ago

Hey Brendan, > If your company is paying for commercial software that includes kernel drivers or kernel modules, you can make eBPF a requirement. Windows soon, may still be atleast a year ahead. Would that be a fair statement? atleast being the operating keyword here. Specifically in the context of network security software, for eBPF programs to be portable across windows/linux, we would need MSFT to add a lot more hooks and expose internal kernel stucts. Hopefully via a common libbpf definition. Otherwise, I fear, having two versions of the same product, across two OSs would mean more secuirty and quality issues. I guess the point I am trying to make is, we would get there, but we are more than a few years away. I would love to see something like cilium on vanilla windows for a Software defined Company Wide network. We can then start building enterprise network secutiry into it. Baby steps! --- btw, your talks and blog posts about bpftools is godsent!