11 ms·
PCILeech
- peter_d_sherman 7y agoRelated: https://github.com/ufrisk/pcileech-fpga https://github.com/ufrisk/pcileech-fpga
- jjoonathan 7y agoHow widely deployed are IOMMUs these days? I thought they became a standard thing a few years ago.
- wtallis 7y agoIntel used IOMMU support in their CPUs for product segmentation until Thunderbolt came around. It's now pretty widely supported in hardware, but seldom enabled by default at both the motherboard firmware level and OS level.
- drewg123 7y agoThe problem is that, used properly, IOMMUs are horribly expensive. Consider a NIC driver where you're mapping an outgoing packet for DMA. What used to essentially be a virtual to physical translation becomes a virt to phys + entering the phys in the iommu + removing the mapping when the transmit is complete. This is expensive for hardware and software reasons. At one point I benchmarked a 100g setup on linux, and with the IOMMU enabled, we lost about 90% of the bandwidth and most of the CPU time was spent in lock contention over the red-black tree that managed the IOMMU tables. This was 5-ish years ago, so perhaps things have gotten better. So that makes people want to just enable the IOMMU for SR-IOV (and full device) pass-thru to VMs. This is cheaper, since you just set the mapping up when you allocate phys mem for the guest, and tear them down when freeing phys mem. MacOS used to use a really cool trick where they pre-mapped all mbufs into the IOMMU. That made network traffic transmit and receive comparatively fast. However,it also prevented lots of optimizations that modern operating systems use for zero-copy IO (like attaching pages from sendfile directly to mbufs, similer to skb_frags).
- emmericp 7y agoA problem on the hardware side is that Intel's IOMMU TLB is tiny (64 entries), so using huge pages for all DMA-accessible memory is absolutely required to get a good performance out of it. We've done some benchmarks here: https://www.net.in.tum.de/fileadmin/bibtex/publications/papers/ixy-writing-user-space-network-drivers.pdf https://www.net.in.tum.de/fileadmin/bibtex/publications/pape... (Figure 9 on page 10) Only a very basic benchmark, working on more...
- drewg123 7y agoNice paper, and thanks for the reminder of how small the IOMMU tlb is. We never hit this because we were testing full-sized packets (and really bigger, because of TSO) and hit host IOMMU management overheads at ~100K to 200k TSO sends/sec.
- emmericp 7y agoInteresting, did you use huge pages? I think ~100k to 200k TSO "packets" per second should be doable with the IOMMU. But I guess it depends where the data is coming from. Could be one of the odd cases where copying data is faster than doing zero-copy, e.g., just copy everything into the same small set of small-ish buffers to keep the number of pages that need to be present in the IOMMU small?
- hedora 7y agoThis sounds like a driver bug / misconfiguration. Once the IOMMU is set up, there isn’t any host side management to be done (other than managing the IOMMU TLB, but unless it’s thrashing, that’s a no-op on the fast path). At most, each kernel driver has to do an extra addition to map its physical I/O offset to the one exposed to the bus by the IOMMU. With huge pages, there’s approximately one offset per driver, so it lives in cache, probably next to other driver state.
- ajross 7y agoFWIW: 64 entries isn't particularly small for a dTLB, IIRC that's exactly the size on current Intel cores. The real problem is that device DMA, unlike software behavior, is distressingly non-local. The device will stream out a packet or storage block and then never touch that memory again (or not for a very long time -- memory buffers are huge relative to bandwidth on these things). The TLB just doesn't do you much good.
- acdha 7y agoThey did - for example, any supported macOS or iOS device hasn’t been vulnerable to this for years. You can disable that on macOS but not iOS.
- blibble 7y agoI've had intel_iommu=on in my boot cmdline on my personal machines (ones I do my banking on!) for 5 years without issue (board+CPU for all machines both support it, dmesg confirms it is indeed on)
- hedora 7y agoFrom the article: > Does not work if the OS uses the IOMMU/VT-d. This is the default on macOS (unless disabled in recovery mode). Windows 10 with Virtualization based security features enabled does not work fully - this is however not the default setting in Windows 10 or Linux. For PCI (shared bus) the device could just spoof packets, but for PCIe, there are dedicated lanes for each device. I wonder if a physical MITM device is practical.
- _aleph2c_ 7y agoAn interview with Ulf Fritz: https://www.youtube.com/watch?v=MIfY8g73xms&feature=emb_rel_end https://www.youtube.com/watch?v=MIfY8g73xms&feature=emb_rel_...
- DrRobinson 7y agoAnother interview with him: https://www.youtube.com/watch?v=W5Yb3q9iJao https://www.youtube.com/watch?v=W5Yb3q9iJao
- hoistbypetard 7y agoDoes DMA over PCIe work using USB gadget mode with a Linux device? i.e. could a Pi be used easily and inexpensively to build an acquisition device for this? Edit: Bleh. Nevermind. I saw this photo: https://gist.githubusercontent.com/ufrisk/c5ba7b360335a13bbac2515e5e7bb9d7/raw/2df37be67047e19ea2c3f73be67a0ba06fea203d/_gh_mbp.jpg https://gist.githubusercontent.com/ufrisk/c5ba7b360335a13bba... with a pcie adapter connected over what looked like USB3 and forgot that it's thunderbolt on the macbook. I was not quite to the middle of my first cup of coffee when I asked that.
- q3k 7y agoWell USB is not PCIe and doesn't do DMA, so no.
- hoistbypetard 7y agoAren't most of the devices on the linked page doing PCIe over USB3? Edit: Bleh. Nevermind. I saw this photo: https://gist.githubusercontent.com/ufrisk/c5ba7b360335a13bbac2515e5e7bb9d7/raw/2df37be67047e19ea2c3f73be67a0ba06fea203d/_gh_mbp.jpg https://gist.githubusercontent.com/ufrisk/c5ba7b360335a13bba... with a pcie adapter connected over what looked like USB3 and forgot that it's thunderbolt on the macbook. I was not quite to the middle of my first cup of coffee when I asked that.
- saagarjha 7y agoUSB 4!
- floatingatoll 7y agoTwo related links previously on HN (no comments): Introducing the Memory Process File System for PCILeech http://blog.frizk.net/2018/03/memory-process-file-system.html http://blog.frizk.net/2018/03/memory-process-file-system.htm... Using your BMC as a DMA device: Plugging PCILeech to HPE ILO 4 https://www.synacktiv.com/posts/exploit/using-your-bmc-as-a-dma-device-plugging-pcileech-to-hpe-ilo-4.html https://www.synacktiv.com/posts/exploit/using-your-bmc-as-a-...
- loeg 7y agoAnd from the other side of things, securing the host OS from devices, I found this article a few years ago interesting: Fuzzing PCI express: security in plaintext (2017) https://cloud.google.com/blog/products/gcp/fuzzing-pci-express-security-in-plaintext https://cloud.google.com/blog/products/gcp/fuzzing-pci-expre...
- MaupitiBlue 7y agoDoesn’t this change everything wrt cheating?
- tuvan 7y agoThese devices are apparently already used for cheating. Here are some blog posts from Riot Games and ESEA mentioning direct memory access devices as cheating vectors and preventative methods if you are interested. https://blog.esea.net/esea-hardware-cheats/ https://blog.esea.net/esea-hardware-cheats/ https://euw.leagueoflegends.com/en-gb/news/dev/dev-null-anti-cheat-kernel-driver/ https://euw.leagueoflegends.com/en-gb/news/dev/dev-null-anti...
- inetknght 7y agoI would imagine it's no more "change everything" than having root access to your own hardware.
- Namidairo 7y agoNot really. A few of the better anti-cheat products have had detection vectors for these for a while now, from simple things like detecting the driver to outright probing the device.
- Polylactic_acid 7y agoSurely you could combine a cheating chip with a legitimate device like a network card so to the OS it looks just like a network card.
- tezza 7y agoSo an Action Replay for x64 https://gamehacking.org/wiki/Action_Replay_(Amiga) https://gamehacking.org/wiki/Action_Replay_(Amiga)
- teddyh 7y agoAlso for PC: https://www.youtube.com/watch?v=usaioMbE8EQ https://www.youtube.com/watch?v=usaioMbE8EQ
- heeen2 7y agoWeren't these used in a famous counterstrike cheating scandal in Norway?
- VectorLock 7y agoI was just thinking "would this be an effective way to create undetectable cheats for video games?"
- landr0id 7y agoYep, they're also pretty popular with people who cheat in ESEA/FaceIT third-party ladders. One of the guys, ra1f, eventually outed himself [1]. The software-based anticheat is fairly decent, pushing people to hardware-based cheats built off of PCILeech to avoid detection. [1]: https://twitter.com/rra1f/status/1067518342595006466 https://twitter.com/rra1f/status/1067518342595006466
- anonsivalley652 7y agoRoughly 10 years ago, I knew a guy who sold various exfil/border-extraction products to state actors including FireWire dongles similar to Inception (they may even be effectively the same thing, IDK). He specialized in memory dumping attacks on several platforms including Windows, FreeBSD, Linux and macOS. The core problem is that buses should be authenticated, authorized, encrypted, and selective-/least-privileged channels. Exposing a memory and expansion bus to the outside world in the name of "convenience" is insane. A trusted set of components in the OS and in hardware should: 0. Be able to do a HwIDS checksumming of all firmwares to detect tampering. 1. Limit devices' ability to connect unless they are authorized by the user, much like a "hardware firewall" UI.. vaguely similar to say VMware Workstation/Fusion's dialog when plugging in a new USB device mixed with something like Little Snitch'es dialog for a process wanting to connect to a particular port. 2. Authenticate devices with public/private key certs that are burned in, a function where the device can answer challenge requests, and Signal protocol-like construction properly modified for PKI. Then, and only then, can a host talk securely to a device over an encrypted channel.
- hoistbypetard 7y agoWe never sold anything for it, but for some demos back around 2004/2005 we flashed some firewire iPods with custom firmware that performed the attack described here: https://web.archive.org/web/20071011191205/http://md.hudora.de/presentations/firewire/PacSec2004.pdf https://web.archive.org/web/20071011191205/http://md.hudora.... (To be clear, those slides aren't mine, but I can no longer find the firmware we based ours off of, and never did get permission to post source to our mods, which were never distributed.) I know a couple of our customers took entirely the wrong lesson from our demonstrations and banned mp3 players from their office buildings entirely afterwards :)
- anonsivalley652 7y agoHahaha. That's awesome. :) Speaking of things that look silly dept.: Many moons ago around 1998 in uni, I had an HP 48 graphing calculator which was both programmable and had a very powerful IR LED that was usually used for serial transfers between similar devices. It so happened that some enterprising soul made a customizable, preset and code-learning IR remote control app for it that worked, so I put the brand of the lecture hall's 4 TVs in it. From the back of the room, some 15 meters away, I subtly turned on all of the TVs with a phreak'n calculator. Disbelief, confusion and hilarity ensued.