33 ms·

eBPF on Windows

294 points by praseodym 5y ago

coolspot 5y ago
And that is, kids, how MS Windows piece-by-piece was slowly transformed into a Linux distro.
- etaioinshrdlu 5y ago
  I recently spun up a Windows EC2 server instance in aws, running the latest Windows, and was not impressed. The entire system is not very well set up for operation via SSH. SSH is not even enabled by default on ec2. You have to Remote Desktop in, run a bunch of commands to enable it, and even then I could never figure out how to get authorized_keys to work. Once you have SSH running, you can run commands via command prompt or powershell. Powershell is apparently pretty modern and well designed, but it's quite confusing if you're used to Unix. A few applications I tried to install came as MSI's and they could not be installed via the command line, requiring the graphical interface. It felt like a bad way to run a server. You want fast and lightweight remote access on a server, and you want scriptability. Windows was simply worse in this area. Even if Windows was 100% compatible with Linux apps (including the modern CUDA deep learning stack...), I don't know why one would ever want to run Windows on a server. It lost the mindshare war, and I really see why. It really is that bad of an experience. The main reason I've heard people use Windows server is because they are forced to because of app compatibility (no tech startup would find themselves in this position), or the preference of maintaining a server with clicking in the GUI (again, not a place most of us here find ourselves). Anyways, I'd love to hear from someone, anyone, who likes Windows server more than Linux. The point of this comment is also to state the opinion that Windows is currently a very bad Linux distro.
  - bostonsre 5y ago
    I don't use it in production but yea, I've read that automation for windows servers is mainly geared toward management with powershell. It's been a while since I've used it and it's definitely a lot different than bash, but it seemed petty elegant when I used it last. I like objected oriented programming, so having an object oriented shell was pretty neat.
    - vladvasiliu 5y ago
      But PowerShell feels so clunky and slow to me (disclosure: I'm used to Linux). If you want object-oriented automation tools, Python works much better on Linux. Granted, it's not a shell, but if we're talking automation, then it doesn't really matter. I think the big issue for Windows is that most "admins" aren't in the habit of automating things, or at most they'll write a basic bat file. They still expect to click around their GUIs, so they'll be fairly reticent to install the Windows Server Core version [0]. "To be able to intervene in case something happens". The main issue with this is that they often aren't aware of possibilities offered only via PowerShell[1]. There's also the issue that when they look things over in the GUI, they won't see the configuration that's only visible through PowerShell. --- [0] Windows Server Core comes without a GUI, but it can be managed remotely with the usual tools. However, not all server roles work on it. Remote Desktop Gateway is one such example, even though it doesn't have any "desktop" functionality. [1] For example setting up split-view DNS. This is possible since Windows 2016, but only via PowerShell, and it's impossible to know from the GUI that it's activated. Also, this configuration doesn't replicate through ActiveDirectory.
  - nont 5y ago
    I think mostly they're forced to use Windows server because of their proprietary software like outlook, active directory, iis etc. It was maybe easier to find people to support Windows servers... Just click here here and here to install.
    - cptskippy 5y ago
      All of the modern GUI is just a skin over powershell. Most server functionality/configuration isn't exposed through the GUI.
  - recursive 5y ago
    I don't have much experience with Linux. All our customers (health care, mostly US) have Windows servers. Some of them have some Linux also, but that seems to be a minority. We need to be able to ship software they can run. Most of our development is windows-based.
  - Uehreka 5y ago
    In my experience, even MSIs can be run from the command line, but the amount of effort required to look this up and write a clean script is definitely way more than the equivalent Linux sysadmin task because Windows is just not expecting users to work this way at all. My gripe with installing things on Linux is that every project assumes install means “build from source”, and it’s often annoyingly hard to find the exact yum/apt-get name for something (do I need a PPA? is the name thing, libthing, libthing-dev, libthing2, or something else?) But this is a much smaller issue than anything I run into with Windows Server.
  - EvanAnderson 5y ago
    MSIs, if put together well, easily install silently from the command line. msiexec /qn /i msi-name.msi Edit (now that I have more time): All the tooling to make Windows command-line manageable is there "out of the box". In a former "life" I staged a Windows Server 2008 R2 install that was remotely manageable with SSH (using a third-party SSH server) "out of the box". Similar to a "kickstart" w/ the Anaconda installer, you're talking about chaining scripts to run after the installation to enable desired functionality. It wasn't much different than automating a Linux install. If you enable the serial console you can do some nice command-line management of Windows VMs through your hypervisor's virtual serial port functionality too. I enjoy having the serial console open because I've been able to diagnose "hung" Windows machines (that is, not responding to GUI logon attempts) thru the console. It's very handy.
    - malkia 5y ago
      and /norestart too
  - e12e 5y ago
    I seem to recall authorized_keys works as expected; it goes into %HOME%\.ssh\authorized_keys (I'm not sure/can't recall if there are some similar permission requirements). Anyway, that's eg: c:\users\mcgyver\.ssh\authorized_keys And ssh needs a restart. That said, windows remote management is traditionally via WMI, not powershell. But PS has come a long way, and ssh is a sane transport. And it can function as a tunnel for WMI, RDP and PS (ssh is easier to use for key based auth, disabling password auth).
    - mook 5y ago
      If you're a member of the administrators group, by default it uses something out of C:\ProgramData instead. https://docs.microsoft.com/en-us/windows-server/administration/openssh/openssh_server_configuration#authorizedkeysfile https://docs.microsoft.com/en-us/windows-server/administrati...
      - e12e 5y ago
        But of course, you're not running as Administrator, right? That's why we have runas... :)
  - nullify88 5y ago
    Its about 5 lines of powershell to install and start OpenSSH. Maybe a few more if you want to automatically inject public keys. Well within possibilities of a user_data script that runs on the first boot of an AWS instance. My main pain with OpenSSH on Windows was that its an older 7.7 Windows port that gets installed. However 8.1 is finally bundled as part of the recent KB5001391 and fixes some annoying bugs.
  - jaywalk 5y ago
    I like Windows Server more than Linux. I administer both of them, although I am more familiar with Windows Server which definitely plays a big part in my preference. But I think that's going to be the biggest part of anyone's preference. Everything you complain about Windows Server not supporting is actually supported, but it was your unfamiliarity that was ultimately the issue. "Linux guy who is unfamiliar with Windows Server prefers Linux" isn't particularly shocking news.
    - EvanAnderson 5y ago
      I'm probably pretty similar in experience to you. I "grew up" with Linux (and Xenix) in the early 90s, not being able to afford to buy Windows Server. I transitioned over to Windows Server around NT 3.51 and now handle both. All the tooling for headless / remotely-manageable Windows is "in there" in 2008 and later. Prior versions required more jumping thru hoops to manage via command line, but Resource Kit tools helped.
      - jaywalk 5y ago
        My server administration journey actually began with a pirated copy of Windows 2000 Advanced Server. And I don't feel bad for it, because having started very young with Windows Server has directly led to many tens/hundreds of thousands of dollars in licensing fees paid to Microsoft at my direction :)
        EvanAnderson 5y ago
        You missed a lot of tedious reboots by starting your experience with the Windows NT family w/ Windows 2000. Windows NT 4.0 was a breath of fresh air, as far as the UI went, compared to NT 3.51, but Windows NT didn't really come into its own, in my opinion, until Windows 2000. The tedious "re-apply the service pack and reboot" or "you thought about changing a setting, so therefore reboot" scenarios were nearly removed with Windows 2000. Having plug 'n play and hot-plug hardware work well was great, too.
        5y ago
  - kwdc 5y ago
    I won't comment on the eBPF or WSL or even the Windows as linux debate, but I used to do some Windows admin. Some clarifications are in order. MSIs CAN be installed on servers with no gui required. Or much more commonly via powershell automation. Completely remotely and I'm not referring to remote desktop either. I know its fun to complain about Windows, but really, if you're not using powershell and the right tools such as a Windows Domain, you're kind of like some kind of debian linux admin that refuses to use bash scripts / debian apt / ssh etc plus chef or similar management. It really is that kind of comparison you're making. What would you say about a linux admin that refused to use SSH or any scripting? You'd wonder why they are complaining. Most of Windows Server comes alive when you join the machine to a domain and it gets access to actual infrastructure rather than some lone isolated server. This is to the point where you don't even need to log into the machine via remote desktop or ssh at all. In fact you manage the server from a completely different Windows machine with an admin account within the domain. You can then run the powershell script on your management computer and things happen on the target computer(s). I haven't even mentioned the multiple other tools you get as part of the machines being joined to a domain. Or you can deploy the script to the machine and execute it there. Still no remote desktop required.
    - EvanAnderson 5y ago
      I can't imagine how anybody manages any number of Windows machines w/o a domain. Even just an NT 4.0-style "downlevel" domain (hosted with Samba, if need be) made life so much easier "back in the day". Once Active Directory came out I never looked back. Group Policy makes so many Windows administration tasks reproducible and automated. I keep meaning, year after year, to take at Samba and see how its Active Directory emulation functionality has progressed. I never end up getting around to it. Presumably, since so much of Active Directory is really just leveraging LDAP and a funky proprietary schema, it has progressed significantly. I've had a hard time figuring out how to position Desired State Configuration. It feels like an appeal to Linux sysadmins used to Puppet, Chef, et al. I can already do everything it can do with Group Policy, and I'd want an Active Directory domain for single-sign-on anyway. I have a hard time coming up with a use case for learning all new tooling when I've already got AD and Group Policy.
  - agumonkey 5y ago
    > Powershell is apparently pretty modern and well designed, but it's quite confusing if you're used to Unix. bash is pretty confusing eve if you're used to unix -- Greg's
    - majkinetor 5y ago
      Bash is simply outdated. It grown organically and thats what you got. Now you have zillion scripts and situation is even worst then with cobol. PowerShell was made by Unix people after 2+ decades of bash experience. It solved hundreds of things. Yet people complain it didn't solve few or it takes a bit more to load (no, verbosity = RTFM). You can't ever please I guess.
      - agumonkey 5y ago
        were powershell designers from the unix world ? I didn't know that I don't mind bash warts, history is what it is, I don't sell powershell .. but someone saying powershell is confusing has serious jaws. powershell is really nice and only if you fancy sed/grep everything then you'll consider bash* superior
        majkinetor 5y ago
        sed/grep/awk have nothing to do with bash tho and you can run them equally easy on Windows (cinst sed grep awk) if you like outdated reg ex.
        agumonkey 5y ago
        yes they're not bash but they're an integral part of bash scripts pipe processing
        majkinetor 5y ago
        They are "integral" only because shell is lacking those functions. On some places tools are not there (docker image for example). Some people complain about size of the dotNet runtime, but if you install all those tools bash "needs" for comfy usage, you can equally easy install dotNet.
  - rasputnik6502 5y ago
    I kinda like Windows server, mostly for the stability - both of the runtime and of functionality. Compared to Linux it requires less attention, i can concentrate more on software and less on the system: after two years of not touching i can still find configuration options in more or less same places. With Linux i have to re-learn quite a lot after not paying attention for a while. Missing good ssh/command line but not badly, I don't really need advanced shell scripting.
- encryptluks2 5y ago
  They implement Linux functionality enough to suck at it and then the Windows users that try it thinks it is Linux that sucks and not Windows.
  - cptskippy 5y ago
    The Linux functionality is intended to allow Windows developers to develop for Linux and deploy to Linux.
    - encryptluks2 5y ago
      Compiling on WSL2 would have terrible performance. WSL2 is just there to try to entice companies to say their developers shouldn't need to move to Linux or Mac. No one is seriously developing in WSL.
      - kube-system 5y ago
        Why would they need WSL2 to do that? AD and Office were and still are plenty enough to keep MS on corporate PCs.
      - cptskippy 5y ago
        It's not really targeted at compiling either. Think line of business apps on web technology like .NET Core, PHP, Python, etc... With WSL you can setup some docker containers or do some orchestration with Kubernetes and push it off to the cloud with minimal effort. If you really want a full Linux environment you can spin up a VM in Hyper-V. WSL just makes it easier to do things where a VM is a hassle.
        encryptluks2 5y ago
        Docker containers don't run in WSL.
        jhomedall 5y ago
- tptacek 5y ago
  There's a lot of interesting stuff in this post, including a different architecture for verifying eBPF programs, so it's frustrating to see a bullshitting comment at the top of the thread. I don't like Windows either, but the virality of the eBPF (and dtrace) model is an interesting story that tells us stuff about what operating systems are going to look like 10 years from now, and we need to do better than comments like this.
- pjmlp 5y ago
  On the contrary, that is how Microsoft fixes the big mistake of not having kept POSIX subsytem around and improved it throughout the last 25 years.
- criddell 5y ago
  I thought they agreed to not compete in the Unix world when they sold Xenix? Is it because SCO is dead that they can jump right into Linux?
- dang 5y ago
  "Eschew flamebait. Avoid unrelated controversies and generic tangents." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html Comments like this one take threads in predictable, uninteresting directions, and—what's worse—they often get upvoted, accumulating mass at the top of the thread and choking out the specific interesting stuff. "Windows v Linux" is a classic example of a generic black hole, sucking passing spaceships into a state from which no light can emerge: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Adang%20black%20hole&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que.... To be fair, this is a co-creation between the comment and the upvotes, and the upvotes are more to blame. But the idea of that guideline is to refrain from introducing black holes in the first place, so the spaceship can poke around something interesting in the vicinity without collapsing, screaming, into an irreversible fate. Related explanation here if anyone wants more: https://news.ycombinator.com/item?id=26894739 https://news.ycombinator.com/item?id=26894739. Note the point about diffs—that's key. Diffs are what's interesting!
  - coolspot 5y ago
    Oops, sorry about that! Intention was to highlight how MS, that fought with *nix tooth and nail, now they are adopting more and more ideas from *nix, basically moving towards being best enterprise Linux.
    - pjmlp 5y ago
      You mean the company that used to have the best UNIX for PCs during the early 80's (Xenix), and a POSIX subsystem for NT/2000 for getting governement contracts?
bostonsre 5y ago
I wonder what kind of performance this gets when compared with running eBPF since eBPF was written with performance in mind from the start.
- PoignardAzur 5y ago
  What?
  - bostonsre 5y ago
    BPF or berkely packet filter was written to be a faster replacement of tcpdump. People saw that it was pretty neat and started using it for non-tcp dump like stuff and it became extended BPF/eBPF. I would guess that running eBPF on Windows would be a lot slower, but it would be interesting to see a performance comparison.
    - tptacek 5y ago
      Point of order: BPF wasn't written simply for tcpdump; it's part of a line of research on using PL runtimes to configure and operate networking stacks; so, right after McCanne's BPF paper, you get MPF, which is proposed to do all of demux for Mach.
      - bostonsre 5y ago
        Neat, did not know that.
        tptacek 5y ago
        A long post I wrote about this stuff, taking the history back to the Xerox Alto: https://fly.io/blog/bpf-xdp-packet-filters-and-udp/ https://fly.io/blog/bpf-xdp-packet-filters-and-udp/
- tptacek 5y ago
  They're doing native code generation, like Linux.
morelisp 5y ago
I see we're now firmly on the "extend" step.
- therein 5y ago
  Sounds like this time around they are going with embrace, import, extinguish.
- staticassertion 5y ago
  No, this would just be embrace, they have not extended the ebpf capabilities. It would be "extend" if there were one way comparability, which doesn't seem to be the case.
  - hinkley 5y ago
    If they patent ways to analyze more complex eBPF programs that would be extend in a very literal sense of the term.
    - staticassertion 5y ago
      I don't get what you're trying to convey here. Yes, there are hypothetical situations we could talk about?
mad_vill 5y ago
I wonder how long until I can run cilium on my mixed node kubernetes clusters!
1970-01-01 5y ago
See https://en.wikipedia.org/wiki/Embrace,_extend,_and_extinguish https://en.wikipedia.org/wiki/Embrace,_extend,_and_extinguis... Step 1: Getting SQL and a few other things running in Linux. That was in 2016. Step 2: WSL and WSL 2 eBPF puts them at something like step 2.2.5
- jaywalk 5y ago
  This seems like the exact opposite of EEE. Care to explain your thinking?
  - hkt 5y ago
    One assumes the idea is to make windows have feature parity with linux, then try to leverage network effects for features windows has but linux doesn't, such that the "extinguish" step will follow the "extend" step.
    - jaywalk 5y ago
      That's called "being better than the competition" and not really what EEE was meant to describe.
      - bztzt 5y ago
        I'm not sure there's any crisply definable bright line separating them. Generally embrace/extend strategies do involve trying to make extensions that some people will consider valuable.
  - zhengyi13 5y ago
    If you're thinking in EEE terms, this is Embrace. Extend and Extinguish come later, once you've got a preponderance of developers on your platform.
  - 1970-01-01 5y ago
    Linux is cited as an example in the link: https://en.wikipedia.org/wiki/Embrace,_extend,_and_extinguish#cite_note-27 https://en.wikipedia.org/wiki/Embrace,_extend,_and_extinguis...
the_only_law 5y ago
I admittedly have only an extremely cursory knowledge of these sort of technologies, but how does eBPF compare to NDIS filters and WFP filters? Biggest reason I could imagine for this is easier portability of existing eBPF applications.
- aneutron 5y ago
  That's exactly what I thought. Because WFP are powerful, NDIS is bonkers (it's what what used by Internet Download Manager and Kaspersky for their features), so the only use case is maybe YARA-style eBPF rulesets.
- tptacek 5y ago
  You can do a lot more in NDIS and probably more in WFP than you can in eBPF, if only because eBPF has strict limits on the kinds of loops you can express (they have to be verifiably bounded). Packet-processing BPF programs are tightly bound to Linux kernel APIs, many of which won't be ported here, so I think write-once-run-anywhere is unlikely to be an attribute of this. Importantly: eBPF has at this point not much to do with packets; it's a generic kernel and userland instrumentation layer. Most new BPF code is written to monitor local program runtimes, not to look at packets. If Windows adopts XDP, we might get to a point where
  - joestringer 5y ago
    The packet-processing BPF programs are less tightly bound to Linux kernel APIs than you might think. Even in Linux, there has been motivation to make the APIs more generic to support different kernel hooks for packets, in particular XDP which doesn't operate on the standard internal packet buffer representation (skbuff).
    - tptacek 5y ago
      Microsoft doesn't support XDP, do they?(And you can only use XDP on Linux in certain circumstances). The clsbpf stuff is all pretty heavily tied to skbs. Also, even in an XDP program, you're still likely to use a bunch of perf stuff, which is again pretty Linux-specific.
zwad3 5y ago
Is there any analogue of seccomp in windows that can be used with BPF?
- tptacek 5y ago
  Windows already has a native system call filter, doesn't it?
  - roblabla 5y ago
    I don't think so? I actually looked for such functionality recently and couldn't find anything. Kaspersky uses a hypervisor to hook syscalls[0] in order to provide such functionality. There's also DTrace for Windows[1], but that requires being enabled through bcdedit which is a bit... meh. [0]: https://github.com/iPower/KasperskyHook https://github.com/iPower/KasperskyHook [1]: https://docs.microsoft.com/en-us/windows-hardware/drivers/devtest/dtrace#installing-dtrace-under-windows https://docs.microsoft.com/en-us/windows-hardware/drivers/de...
    - tptacek 5y ago
      I'm thinking of the win32k.sys filter, which, my Windows-literate friends inform me, only blocks a subset (a gnarly subset, but a subset still) of the total kernel attack surface; it's not a general-purpose filter.
tptacek 5y ago
A fun detail of this work is that it uses a formal-methods-based verifier (designed outside of Microsoft) that accepts a wider range of programs than does the Linux verifier, which is itself kind of nightmare fuel. https://vbpf.github.io/assets/prevail-paper.pdf https://vbpf.github.io/assets/prevail-paper.pdf The verifier in this paper also has some biting limitations; for instance, you can't resize a packet in it, because they don't account for pointer invalidation. I wonder whether they've since implemented these verifier features, since they'd be problematic for compatibility otherwise. Additionally, the PREVAIL paper explicitly doesn't verify program termination, which is kind of a dealbreaker for kernel BPF.
- salawat 5y ago
  >Additionally, the PREVAIL paper explicitly doesn't verify program termination, which is kind of a dealbreaker for kernel BPF. I'd be more alarmed if someone had solved the Halting Problem and I hadn't heard about it, to be honest.
  - sanxiyn 5y ago
    Microsoft did termination proof for Windows drivers in 2006. Termination proof does not require solving halting problem. Termination proofs for systems code (PLDI 2006): https://dl.acm.org/doi/10.1145/1133255.1134029 https://dl.acm.org/doi/10.1145/1133255.1134029
    - salawat 5y ago
      As mentioned in a sibling reply, there goes my week. Thanks for the reference! Time to learn something I apparently did not know!
      - whizzter 5y ago
        I think you can reason it like this. Assuming certain constraints (what you can define bounded analysis rules for) you can prove that things terminate. These rules things are usually far more advanced than we suspect initially, but they're not still not fully unbounded.(And we can often come up with more rules to make things practical even if unwieldy) Deciding termination for an arbitrarily large program in a turing complete system CAN require no less than execution of such a program thus we CANNOT guarantee to be able to verify termination for EVERY program. (This is the halting problem) I spent a lot of time on my thesis learning about type inference systems and as many such systems for uncooperative languages(JS,Python,etc) need to do abstract interpretation they fall into the same kind of issues and this is why practical systems use JIT compilation instead of AOT even if there are exceptions (Shed Skin Python is quite impressive for example).
GordonS 5y ago
Wow, I'm really stoked to see this! This could be a game-changer for the infosec community in particular - now, if you want to get into internals, such as tracing file system and registry calls, you've got to write drivers. And drivers are very tricky to write, and it's very easy to miss corner cases - which can result in the dreaded BSOD. Plus, drivers need to go through a verification and signing process by Microsoft. Having access to that capability from user-mode, without having to write drivers... that would be amazing.
- dd21223 5y ago
  It is really great to see that eBPF is causing an industry wide change instead of just a Linux one!
- Someone1234 5y ago
  Everything you described is already available on Windows in userland via filters (FileSystemWatcher/ReadDirectoryChangesW, RegNotifyChangeKeyValue, et al). Obviously you can only monitor whatever lives in your security context. eBPF looks cool since it is a VM, that could have access to more kernel structures (network?), but solving the issues you described is largely a solved problem on the platform.
  - GordonS 5y ago
    Nope, with FileSystemWatcher (and the Windows APIs it uses under the hood) you get minimal information - only using a driver can you get contextual information, such as the handle of the thread, process and user that made the call. And RegNotifyChangeKeyValue is only useful for watching a single, specific value - if you monitor a tree, it doesn't even tell you what changed, only that something matching the filter did. And as with file system changes, if you want to know which thread/process/user made the change, you need to use a driver.
- dataflow 5y ago
  If all you want is user-mode tracing, there's ETW. Where you run into problems is that it doesn't let you block on calls, etc.
hs86 5y ago
Here is a recent comment from Dropbox engineers about the state of tracing tooling on Linux compared to Windows: https://dropbox.tech/infrastructure/boosting-dropbox-upload-speed#debugging-with-microsoft https://dropbox.tech/infrastructure/boosting-dropbox-upload-... Is their assessment correct? If so, how comes that we got DTrace in 2019 and now eBPF ported to Windows? Are they trying to consolidate all tooling into one platform?
- muststopmyths 5y ago
  ETW (What netsh trace in the post uses underneath) has been around for 20+ years in Windows and is extremely powerful. However, Microsoft has always neglected the usability side of it. It's a bitch to set up and hard to interpret. Documentation is there, but scattered and hard to find easily. Most people here have probably seen Bruce Dawson's performance analysis and debugging posts. WPA/WPR are probably the only user-friendly (ish) ETW applications and even those are not that easy to use. So yeah, Windows has had powerful tracing and inspection tools for a while, but few people know how to use them well. As far as why they are doing DTrace and eBPF, I think both of those fill some holes that ETW doesn't. Mainly dynamic tracing and instrumentation, but I am sure there are other advantages I am not thinking of as well. They could have come up with another hopelessly complicated Microsoftie framework to do the same things, but it's probably a good thing that they didn't.
  - sanxiyn 5y ago
    ETW is the topic of the post titled "The Worst API Ever Made" and the title is well deserved. If ETW tooling is lacking, it's because API is so terrible nobody wants to write tooling. https://caseymuratori.com/blog_0025 https://caseymuratori.com/blog_0025
    - malkia 5y ago
      There are some good articles in here (recently added) - https://devblogs.microsoft.com/performance-diagnostics/ https://devblogs.microsoft.com/performance-diagnostics/ - but @randomascii's stuff was priceless (I'm still using UIforETW as it's friendliest little ui to setup xperf). I'm still struggling with WPA though (from loading/waiting to load symbols, to sometimes getting right there lost - but often it's helpful). Visual Studio's own Performance Profiler (Alt+F2, Ctrl+Alt+F2) also uses ETW - just more streamlined, better UI - but once it collects a bit too much data, it just can't handle it.
throwaway29303 5y ago
Pretty cool, but let's not fool ourselves. Recall history or why Microsoft is doing this. This is a long-term strategy. https://en.wikipedia.org/wiki/Embrace,_extend,_and_extinguish https://en.wikipedia.org/wiki/Embrace,_extend,_and_extinguis...
- jrockway 5y ago
  Has Microsoft been doing a lot of embracing, extending, and extinguishing lately? I guess Typescript kind of embraces and extends Javascript, but I think every frontend engineer will be doing a happy dance if they manage to extinguish it. Not quite 1990s Microsoft's approach.
gigatexal 5y ago
Microsoft is really being bold about adopting open tech to improve either it’s bottom line (Azure now runs more Linux VMs than Windows ones) and now with eBPF. Truly a new and different Microsoft.
- pjmlp 5y ago
  On top of HyperV, a Windows tech stack.
brendangregg 5y ago
Great news, I'm looking forward to analyzing performance on Windows with BPF! Given that PerfView and WPA also got flame graphs it'll start to feel like home. :-)
- srcmap 5y ago
  From the architecture diagram, it looks like just network ebpf only. Does anyone know if it can profile file/disk io type activities?
  - dblohm7 5y ago
    They make it pretty clear from the post that its support is currently limited, and that they plan on adding more hooks in the future.

eBPF on Windows

294 points by praseodym 5y ago

coolspot 5y ago

And that is, kids, how MS Windows piece-by-piece was slowly transformed into a Linux distro.

etaioinshrdlu 5y ago
I recently spun up a Windows EC2 server instance in aws, running the latest Windows, and was not impressed. The entire system is not very well set up for operation via SSH. SSH is not even enabled by default on ec2. You have to Remote Desktop in, run a bunch of commands to enable it, and even then I could never figure out how to get authorized_keys to work. Once you have SSH running, you can run commands via command prompt or powershell. Powershell is apparently pretty modern and well designed, but it's quite confusing if you're used to Unix. A few applications I tried to install came as MSI's and they could not be installed via the command line, requiring the graphical interface. It felt like a bad way to run a server. You want fast and lightweight remote access on a server, and you want scriptability. Windows was simply worse in this area. Even if Windows was 100% compatible with Linux apps (including the modern CUDA deep learning stack...), I don't know why one would ever want to run Windows on a server. It lost the mindshare war, and I really see why. It really is that bad of an experience. The main reason I've heard people use Windows server is because they are forced to because of app compatibility (no tech startup would find themselves in this position), or the preference of maintaining a server with clicking in the GUI (again, not a place most of us here find ourselves). Anyways, I'd love to hear from someone, anyone, who likes Windows server more than Linux. The point of this comment is also to state the opinion that Windows is currently a very bad Linux distro.
- bostonsre 5y ago
  I don't use it in production but yea, I've read that automation for windows servers is mainly geared toward management with powershell. It's been a while since I've used it and it's definitely a lot different than bash, but it seemed petty elegant when I used it last. I like objected oriented programming, so having an object oriented shell was pretty neat.
  - vladvasiliu 5y ago
    But PowerShell feels so clunky and slow to me (disclosure: I'm used to Linux). If you want object-oriented automation tools, Python works much better on Linux. Granted, it's not a shell, but if we're talking automation, then it doesn't really matter. I think the big issue for Windows is that most "admins" aren't in the habit of automating things, or at most they'll write a basic bat file. They still expect to click around their GUIs, so they'll be fairly reticent to install the Windows Server Core version [0]. "To be able to intervene in case something happens". The main issue with this is that they often aren't aware of possibilities offered only via PowerShell[1]. There's also the issue that when they look things over in the GUI, they won't see the configuration that's only visible through PowerShell. --- [0] Windows Server Core comes without a GUI, but it can be managed remotely with the usual tools. However, not all server roles work on it. Remote Desktop Gateway is one such example, even though it doesn't have any "desktop" functionality. [1] For example setting up split-view DNS. This is possible since Windows 2016, but only via PowerShell, and it's impossible to know from the GUI that it's activated. Also, this configuration doesn't replicate through ActiveDirectory.
- nont 5y ago
  I think mostly they're forced to use Windows server because of their proprietary software like outlook, active directory, iis etc. It was maybe easier to find people to support Windows servers... Just click here here and here to install.
  - cptskippy 5y ago
    All of the modern GUI is just a skin over powershell. Most server functionality/configuration isn't exposed through the GUI.
- recursive 5y ago
  I don't have much experience with Linux. All our customers (health care, mostly US) have Windows servers. Some of them have some Linux also, but that seems to be a minority. We need to be able to ship software they can run. Most of our development is windows-based.
- Uehreka 5y ago
  In my experience, even MSIs can be run from the command line, but the amount of effort required to look this up and write a clean script is definitely way more than the equivalent Linux sysadmin task because Windows is just not expecting users to work this way at all. My gripe with installing things on Linux is that every project assumes install means “build from source”, and it’s often annoyingly hard to find the exact yum/apt-get name for something (do I need a PPA? is the name thing, libthing, libthing-dev, libthing2, or something else?) But this is a much smaller issue than anything I run into with Windows Server.
- EvanAnderson 5y ago
  MSIs, if put together well, easily install silently from the command line. msiexec /qn /i msi-name.msi Edit (now that I have more time): All the tooling to make Windows command-line manageable is there "out of the box". In a former "life" I staged a Windows Server 2008 R2 install that was remotely manageable with SSH (using a third-party SSH server) "out of the box". Similar to a "kickstart" w/ the Anaconda installer, you're talking about chaining scripts to run after the installation to enable desired functionality. It wasn't much different than automating a Linux install. If you enable the serial console you can do some nice command-line management of Windows VMs through your hypervisor's virtual serial port functionality too. I enjoy having the serial console open because I've been able to diagnose "hung" Windows machines (that is, not responding to GUI logon attempts) thru the console. It's very handy.
  - malkia 5y ago
    and /norestart too
- e12e 5y ago
  I seem to recall authorized_keys works as expected; it goes into %HOME%\.ssh\authorized_keys (I'm not sure/can't recall if there are some similar permission requirements). Anyway, that's eg: c:\users\mcgyver\.ssh\authorized_keys And ssh needs a restart. That said, windows remote management is traditionally via WMI, not powershell. But PS has come a long way, and ssh is a sane transport. And it can function as a tunnel for WMI, RDP and PS (ssh is easier to use for key based auth, disabling password auth).
  - mook 5y ago
    If you're a member of the administrators group, by default it uses something out of C:\ProgramData instead. https://docs.microsoft.com/en-us/windows-server/administration/openssh/openssh_server_configuration#authorizedkeysfile https://docs.microsoft.com/en-us/windows-server/administrati...
    - e12e 5y ago
      But of course, you're not running as Administrator, right? That's why we have runas... :)
- nullify88 5y ago
  Its about 5 lines of powershell to install and start OpenSSH. Maybe a few more if you want to automatically inject public keys. Well within possibilities of a user_data script that runs on the first boot of an AWS instance. My main pain with OpenSSH on Windows was that its an older 7.7 Windows port that gets installed. However 8.1 is finally bundled as part of the recent KB5001391 and fixes some annoying bugs.
- jaywalk 5y ago
  I like Windows Server more than Linux. I administer both of them, although I am more familiar with Windows Server which definitely plays a big part in my preference. But I think that's going to be the biggest part of anyone's preference. Everything you complain about Windows Server not supporting is actually supported, but it was your unfamiliarity that was ultimately the issue. "Linux guy who is unfamiliar with Windows Server prefers Linux" isn't particularly shocking news.
  - EvanAnderson 5y ago
    I'm probably pretty similar in experience to you. I "grew up" with Linux (and Xenix) in the early 90s, not being able to afford to buy Windows Server. I transitioned over to Windows Server around NT 3.51 and now handle both. All the tooling for headless / remotely-manageable Windows is "in there" in 2008 and later. Prior versions required more jumping thru hoops to manage via command line, but Resource Kit tools helped.
    - jaywalk 5y ago
      My server administration journey actually began with a pirated copy of Windows 2000 Advanced Server. And I don't feel bad for it, because having started very young with Windows Server has directly led to many tens/hundreds of thousands of dollars in licensing fees paid to Microsoft at my direction :)
      - EvanAnderson 5y ago
        You missed a lot of tedious reboots by starting your experience with the Windows NT family w/ Windows 2000. Windows NT 4.0 was a breath of fresh air, as far as the UI went, compared to NT 3.51, but Windows NT didn't really come into its own, in my opinion, until Windows 2000. The tedious "re-apply the service pack and reboot" or "you thought about changing a setting, so therefore reboot" scenarios were nearly removed with Windows 2000. Having plug 'n play and hot-plug hardware work well was great, too.
        5y ago
- kwdc 5y ago
  I won't comment on the eBPF or WSL or even the Windows as linux debate, but I used to do some Windows admin. Some clarifications are in order. MSIs CAN be installed on servers with no gui required. Or much more commonly via powershell automation. Completely remotely and I'm not referring to remote desktop either. I know its fun to complain about Windows, but really, if you're not using powershell and the right tools such as a Windows Domain, you're kind of like some kind of debian linux admin that refuses to use bash scripts / debian apt / ssh etc plus chef or similar management. It really is that kind of comparison you're making. What would you say about a linux admin that refused to use SSH or any scripting? You'd wonder why they are complaining. Most of Windows Server comes alive when you join the machine to a domain and it gets access to actual infrastructure rather than some lone isolated server. This is to the point where you don't even need to log into the machine via remote desktop or ssh at all. In fact you manage the server from a completely different Windows machine with an admin account within the domain. You can then run the powershell script on your management computer and things happen on the target computer(s). I haven't even mentioned the multiple other tools you get as part of the machines being joined to a domain. Or you can deploy the script to the machine and execute it there. Still no remote desktop required.
  - EvanAnderson 5y ago
    I can't imagine how anybody manages any number of Windows machines w/o a domain. Even just an NT 4.0-style "downlevel" domain (hosted with Samba, if need be) made life so much easier "back in the day". Once Active Directory came out I never looked back. Group Policy makes so many Windows administration tasks reproducible and automated. I keep meaning, year after year, to take at Samba and see how its Active Directory emulation functionality has progressed. I never end up getting around to it. Presumably, since so much of Active Directory is really just leveraging LDAP and a funky proprietary schema, it has progressed significantly. I've had a hard time figuring out how to position Desired State Configuration. It feels like an appeal to Linux sysadmins used to Puppet, Chef, et al. I can already do everything it can do with Group Policy, and I'd want an Active Directory domain for single-sign-on anyway. I have a hard time coming up with a use case for learning all new tooling when I've already got AD and Group Policy.
- agumonkey 5y ago
  > Powershell is apparently pretty modern and well designed, but it's quite confusing if you're used to Unix. bash is pretty confusing eve if you're used to unix -- Greg's
  - majkinetor 5y ago
    Bash is simply outdated. It grown organically and thats what you got. Now you have zillion scripts and situation is even worst then with cobol. PowerShell was made by Unix people after 2+ decades of bash experience. It solved hundreds of things. Yet people complain it didn't solve few or it takes a bit more to load (no, verbosity = RTFM). You can't ever please I guess.
    - agumonkey 5y ago
      were powershell designers from the unix world ? I didn't know that I don't mind bash warts, history is what it is, I don't sell powershell .. but someone saying powershell is confusing has serious jaws. powershell is really nice and only if you fancy sed/grep everything then you'll consider bash* superior
      - majkinetor 5y ago
        sed/grep/awk have nothing to do with bash tho and you can run them equally easy on Windows (cinst sed grep awk) if you like outdated reg ex.
        agumonkey 5y ago
        yes they're not bash but they're an integral part of bash scripts pipe processing
        majkinetor 5y ago
        They are "integral" only because shell is lacking those functions. On some places tools are not there (docker image for example). Some people complain about size of the dotNet runtime, but if you install all those tools bash "needs" for comfy usage, you can equally easy install dotNet.
- rasputnik6502 5y ago
  I kinda like Windows server, mostly for the stability - both of the runtime and of functionality. Compared to Linux it requires less attention, i can concentrate more on software and less on the system: after two years of not touching i can still find configuration options in more or less same places. With Linux i have to re-learn quite a lot after not paying attention for a while. Missing good ssh/command line but not badly, I don't really need advanced shell scripting.
encryptluks2 5y ago
They implement Linux functionality enough to suck at it and then the Windows users that try it thinks it is Linux that sucks and not Windows.
- cptskippy 5y ago
  The Linux functionality is intended to allow Windows developers to develop for Linux and deploy to Linux.
  - encryptluks2 5y ago
    Compiling on WSL2 would have terrible performance. WSL2 is just there to try to entice companies to say their developers shouldn't need to move to Linux or Mac. No one is seriously developing in WSL.
    - kube-system 5y ago
      Why would they need WSL2 to do that? AD and Office were and still are plenty enough to keep MS on corporate PCs.
    - cptskippy 5y ago
      It's not really targeted at compiling either. Think line of business apps on web technology like .NET Core, PHP, Python, etc... With WSL you can setup some docker containers or do some orchestration with Kubernetes and push it off to the cloud with minimal effort. If you really want a full Linux environment you can spin up a VM in Hyper-V. WSL just makes it easier to do things where a VM is a hassle.
      - encryptluks2 5y ago
        Docker containers don't run in WSL.
        jhomedall 5y ago
tptacek 5y ago
There's a lot of interesting stuff in this post, including a different architecture for verifying eBPF programs, so it's frustrating to see a bullshitting comment at the top of the thread. I don't like Windows either, but the virality of the eBPF (and dtrace) model is an interesting story that tells us stuff about what operating systems are going to look like 10 years from now, and we need to do better than comments like this.
pjmlp 5y ago
On the contrary, that is how Microsoft fixes the big mistake of not having kept POSIX subsytem around and improved it throughout the last 25 years.
criddell 5y ago
I thought they agreed to not compete in the Unix world when they sold Xenix? Is it because SCO is dead that they can jump right into Linux?
dang 5y ago
"Eschew flamebait. Avoid unrelated controversies and generic tangents." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html Comments like this one take threads in predictable, uninteresting directions, and—what's worse—they often get upvoted, accumulating mass at the top of the thread and choking out the specific interesting stuff. "Windows v Linux" is a classic example of a generic black hole, sucking passing spaceships into a state from which no light can emerge: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Adang%20black%20hole&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que.... To be fair, this is a co-creation between the comment and the upvotes, and the upvotes are more to blame. But the idea of that guideline is to refrain from introducing black holes in the first place, so the spaceship can poke around something interesting in the vicinity without collapsing, screaming, into an irreversible fate. Related explanation here if anyone wants more: https://news.ycombinator.com/item?id=26894739 https://news.ycombinator.com/item?id=26894739. Note the point about diffs—that's key. Diffs are what's interesting!
- coolspot 5y ago
  Oops, sorry about that! Intention was to highlight how MS, that fought with *nix tooth and nail, now they are adopting more and more ideas from *nix, basically moving towards being best enterprise Linux.
  - pjmlp 5y ago
    You mean the company that used to have the best UNIX for PCs during the early 80's (Xenix), and a POSIX subsystem for NT/2000 for getting governement contracts?

bostonsre 5y ago

I wonder what kind of performance this gets when compared with running eBPF since eBPF was written with performance in mind from the start.

PoignardAzur 5y ago
What?
- bostonsre 5y ago
  BPF or berkely packet filter was written to be a faster replacement of tcpdump. People saw that it was pretty neat and started using it for non-tcp dump like stuff and it became extended BPF/eBPF. I would guess that running eBPF on Windows would be a lot slower, but it would be interesting to see a performance comparison.
  - tptacek 5y ago
    Point of order: BPF wasn't written simply for tcpdump; it's part of a line of research on using PL runtimes to configure and operate networking stacks; so, right after McCanne's BPF paper, you get MPF, which is proposed to do all of demux for Mach.
    - bostonsre 5y ago
      Neat, did not know that.
      - tptacek 5y ago
        A long post I wrote about this stuff, taking the history back to the Xerox Alto: https://fly.io/blog/bpf-xdp-packet-filters-and-udp/ https://fly.io/blog/bpf-xdp-packet-filters-and-udp/
tptacek 5y ago
They're doing native code generation, like Linux.

morelisp 5y ago

I see we're now firmly on the "extend" step.

therein 5y ago
Sounds like this time around they are going with embrace, import, extinguish.
staticassertion 5y ago
No, this would just be embrace, they have not extended the ebpf capabilities. It would be "extend" if there were one way comparability, which doesn't seem to be the case.
- hinkley 5y ago
  If they patent ways to analyze more complex eBPF programs that would be extend in a very literal sense of the term.
  - staticassertion 5y ago
    I don't get what you're trying to convey here. Yes, there are hypothetical situations we could talk about?

mad_vill 5y ago

I wonder how long until I can run cilium on my mixed node kubernetes clusters!

1970-01-01 5y ago

See https://en.wikipedia.org/wiki/Embrace,_extend,_and_extinguish https://en.wikipedia.org/wiki/Embrace,_extend,_and_extinguis... Step 1: Getting SQL and a few other things running in Linux. That was in 2016. Step 2: WSL and WSL 2 eBPF puts them at something like step 2.2.5

jaywalk 5y ago
This seems like the exact opposite of EEE. Care to explain your thinking?
- hkt 5y ago
  One assumes the idea is to make windows have feature parity with linux, then try to leverage network effects for features windows has but linux doesn't, such that the "extinguish" step will follow the "extend" step.
  - jaywalk 5y ago
    That's called "being better than the competition" and not really what EEE was meant to describe.
    - bztzt 5y ago
      I'm not sure there's any crisply definable bright line separating them. Generally embrace/extend strategies do involve trying to make extensions that some people will consider valuable.
- zhengyi13 5y ago
  If you're thinking in EEE terms, this is Embrace. Extend and Extinguish come later, once you've got a preponderance of developers on your platform.
- 1970-01-01 5y ago
  Linux is cited as an example in the link: https://en.wikipedia.org/wiki/Embrace,_extend,_and_extinguish#cite_note-27 https://en.wikipedia.org/wiki/Embrace,_extend,_and_extinguis...

the_only_law 5y ago

I admittedly have only an extremely cursory knowledge of these sort of technologies, but how does eBPF compare to NDIS filters and WFP filters? Biggest reason I could imagine for this is easier portability of existing eBPF applications.

aneutron 5y ago
That's exactly what I thought. Because WFP are powerful, NDIS is bonkers (it's what what used by Internet Download Manager and Kaspersky for their features), so the only use case is maybe YARA-style eBPF rulesets.
tptacek 5y ago
You can do a lot more in NDIS and probably more in WFP than you can in eBPF, if only because eBPF has strict limits on the kinds of loops you can express (they have to be verifiably bounded). Packet-processing BPF programs are tightly bound to Linux kernel APIs, many of which won't be ported here, so I think write-once-run-anywhere is unlikely to be an attribute of this. Importantly: eBPF has at this point not much to do with packets; it's a generic kernel and userland instrumentation layer. Most new BPF code is written to monitor local program runtimes, not to look at packets. If Windows adopts XDP, we might get to a point where
- joestringer 5y ago
  The packet-processing BPF programs are less tightly bound to Linux kernel APIs than you might think. Even in Linux, there has been motivation to make the APIs more generic to support different kernel hooks for packets, in particular XDP which doesn't operate on the standard internal packet buffer representation (skbuff).
  - tptacek 5y ago
    Microsoft doesn't support XDP, do they?(And you can only use XDP on Linux in certain circumstances). The clsbpf stuff is all pretty heavily tied to skbs. Also, even in an XDP program, you're still likely to use a bunch of perf stuff, which is again pretty Linux-specific.

zwad3 5y ago

Is there any analogue of seccomp in windows that can be used with BPF?

tptacek 5y ago
Windows already has a native system call filter, doesn't it?
- roblabla 5y ago
  I don't think so? I actually looked for such functionality recently and couldn't find anything. Kaspersky uses a hypervisor to hook syscalls[0] in order to provide such functionality. There's also DTrace for Windows[1], but that requires being enabled through bcdedit which is a bit... meh. [0]: https://github.com/iPower/KasperskyHook https://github.com/iPower/KasperskyHook [1]: https://docs.microsoft.com/en-us/windows-hardware/drivers/devtest/dtrace#installing-dtrace-under-windows https://docs.microsoft.com/en-us/windows-hardware/drivers/de...
  - tptacek 5y ago
    I'm thinking of the win32k.sys filter, which, my Windows-literate friends inform me, only blocks a subset (a gnarly subset, but a subset still) of the total kernel attack surface; it's not a general-purpose filter.

tptacek 5y ago

A fun detail of this work is that it uses a formal-methods-based verifier (designed outside of Microsoft) that accepts a wider range of programs than does the Linux verifier, which is itself kind of nightmare fuel. https://vbpf.github.io/assets/prevail-paper.pdf https://vbpf.github.io/assets/prevail-paper.pdf The verifier in this paper also has some biting limitations; for instance, you can't resize a packet in it, because they don't account for pointer invalidation. I wonder whether they've since implemented these verifier features, since they'd be problematic for compatibility otherwise. Additionally, the PREVAIL paper explicitly doesn't verify program termination, which is kind of a dealbreaker for kernel BPF.

salawat 5y ago
>Additionally, the PREVAIL paper explicitly doesn't verify program termination, which is kind of a dealbreaker for kernel BPF. I'd be more alarmed if someone had solved the Halting Problem and I hadn't heard about it, to be honest.
- sanxiyn 5y ago
  Microsoft did termination proof for Windows drivers in 2006. Termination proof does not require solving halting problem. Termination proofs for systems code (PLDI 2006): https://dl.acm.org/doi/10.1145/1133255.1134029 https://dl.acm.org/doi/10.1145/1133255.1134029
  - salawat 5y ago
    As mentioned in a sibling reply, there goes my week. Thanks for the reference! Time to learn something I apparently did not know!
    - whizzter 5y ago
      I think you can reason it like this. Assuming certain constraints (what you can define bounded analysis rules for) you can prove that things terminate. These rules things are usually far more advanced than we suspect initially, but they're not still not fully unbounded.(And we can often come up with more rules to make things practical even if unwieldy) Deciding termination for an arbitrarily large program in a turing complete system CAN require no less than execution of such a program thus we CANNOT guarantee to be able to verify termination for EVERY program. (This is the halting problem) I spent a lot of time on my thesis learning about type inference systems and as many such systems for uncooperative languages(JS,Python,etc) need to do abstract interpretation they fall into the same kind of issues and this is why practical systems use JIT compilation instead of AOT even if there are exceptions (Shed Skin Python is quite impressive for example).

GordonS 5y ago

Wow, I'm really stoked to see this! This could be a game-changer for the infosec community in particular - now, if you want to get into internals, such as tracing file system and registry calls, you've got to write drivers. And drivers are very tricky to write, and it's very easy to miss corner cases - which can result in the dreaded BSOD. Plus, drivers need to go through a verification and signing process by Microsoft. Having access to that capability from user-mode, without having to write drivers... that would be amazing.

dd21223 5y ago
It is really great to see that eBPF is causing an industry wide change instead of just a Linux one!
Someone1234 5y ago
Everything you described is already available on Windows in userland via filters (FileSystemWatcher/ReadDirectoryChangesW, RegNotifyChangeKeyValue, et al). Obviously you can only monitor whatever lives in your security context. eBPF looks cool since it is a VM, that could have access to more kernel structures (network?), but solving the issues you described is largely a solved problem on the platform.
- GordonS 5y ago
  Nope, with FileSystemWatcher (and the Windows APIs it uses under the hood) you get minimal information - only using a driver can you get contextual information, such as the handle of the thread, process and user that made the call. And RegNotifyChangeKeyValue is only useful for watching a single, specific value - if you monitor a tree, it doesn't even tell you what changed, only that something matching the filter did. And as with file system changes, if you want to know which thread/process/user made the change, you need to use a driver.
dataflow 5y ago
If all you want is user-mode tracing, there's ETW. Where you run into problems is that it doesn't let you block on calls, etc.

hs86 5y ago

Here is a recent comment from Dropbox engineers about the state of tracing tooling on Linux compared to Windows: https://dropbox.tech/infrastructure/boosting-dropbox-upload-speed#debugging-with-microsoft https://dropbox.tech/infrastructure/boosting-dropbox-upload-... Is their assessment correct? If so, how comes that we got DTrace in 2019 and now eBPF ported to Windows? Are they trying to consolidate all tooling into one platform?

muststopmyths 5y ago
ETW (What netsh trace in the post uses underneath) has been around for 20+ years in Windows and is extremely powerful. However, Microsoft has always neglected the usability side of it. It's a bitch to set up and hard to interpret. Documentation is there, but scattered and hard to find easily. Most people here have probably seen Bruce Dawson's performance analysis and debugging posts. WPA/WPR are probably the only user-friendly (ish) ETW applications and even those are not that easy to use. So yeah, Windows has had powerful tracing and inspection tools for a while, but few people know how to use them well. As far as why they are doing DTrace and eBPF, I think both of those fill some holes that ETW doesn't. Mainly dynamic tracing and instrumentation, but I am sure there are other advantages I am not thinking of as well. They could have come up with another hopelessly complicated Microsoftie framework to do the same things, but it's probably a good thing that they didn't.
- sanxiyn 5y ago
  ETW is the topic of the post titled "The Worst API Ever Made" and the title is well deserved. If ETW tooling is lacking, it's because API is so terrible nobody wants to write tooling. https://caseymuratori.com/blog_0025 https://caseymuratori.com/blog_0025
  - malkia 5y ago
    There are some good articles in here (recently added) - https://devblogs.microsoft.com/performance-diagnostics/ https://devblogs.microsoft.com/performance-diagnostics/ - but @randomascii's stuff was priceless (I'm still using UIforETW as it's friendliest little ui to setup xperf). I'm still struggling with WPA though (from loading/waiting to load symbols, to sometimes getting right there lost - but often it's helpful). Visual Studio's own Performance Profiler (Alt+F2, Ctrl+Alt+F2) also uses ETW - just more streamlined, better UI - but once it collects a bit too much data, it just can't handle it.

throwaway29303 5y ago

Pretty cool, but let's not fool ourselves. Recall history or why Microsoft is doing this. This is a long-term strategy. https://en.wikipedia.org/wiki/Embrace,_extend,_and_extinguish https://en.wikipedia.org/wiki/Embrace,_extend,_and_extinguis...

jrockway 5y ago
Has Microsoft been doing a lot of embracing, extending, and extinguishing lately? I guess Typescript kind of embraces and extends Javascript, but I think every frontend engineer will be doing a happy dance if they manage to extinguish it. Not quite 1990s Microsoft's approach.

gigatexal 5y ago

Microsoft is really being bold about adopting open tech to improve either it’s bottom line (Azure now runs more Linux VMs than Windows ones) and now with eBPF. Truly a new and different Microsoft.

pjmlp 5y ago
On top of HyperV, a Windows tech stack.

brendangregg 5y ago

Great news, I'm looking forward to analyzing performance on Windows with BPF! Given that PerfView and WPA also got flame graphs it'll start to feel like home. :-)

srcmap 5y ago
From the architecture diagram, it looks like just network ebpf only. Does anyone know if it can profile file/disk io type activities?
- dblohm7 5y ago
  They make it pretty clear from the post that its support is currently limited, and that they plan on adding more hooks in the future.