4 ms·
The problem is that, used properly, IOMMUs are horribly expensive. Consider a NIC driver where you're mapping an outgoing packet for DMA. What used to essenti
by drewg123 7y ago
The problem is that, used properly, IOMMUs are horribly expensive.
Consider a NIC driver where you're mapping an outgoing packet for DMA. What used to essentially be a virtual to physical translation becomes a virt to phys + entering the phys in the iommu + removing the mapping when the transmit is complete. This is expensive for hardware and software reasons. At one point I benchmarked a 100g setup on linux, and with the IOMMU enabled, we lost about 90% of the bandwidth and most of the CPU time was spent in lock contention over the red-black tree that managed the IOMMU tables. This was 5-ish years ago, so perhaps things have gotten better.
So that makes people want to just enable the IOMMU for SR-IOV (and full device) pass-thru to VMs. This is cheaper, since you just set the mapping up when you allocate phys mem for the guest, and tear them down when freeing phys mem.
MacOS used to use a really cool trick where they pre-mapped all mbufs into the IOMMU. That made network traffic transmit and receive comparatively fast. However,it also prevented lots of optimizations that modern operating systems use for zero-copy IO (like attaching pages from sendfile directly to mbufs, similer to skb_frags).
- emmericp 7y agoA problem on the hardware side is that Intel's IOMMU TLB is tiny (64 entries), so using huge pages for all DMA-accessible memory is absolutely required to get a good performance out of it. We've done some benchmarks here: https://www.net.in.tum.de/fileadmin/bibtex/publications/papers/ixy-writing-user-space-network-drivers.pdf https://www.net.in.tum.de/fileadmin/bibtex/publications/pape... (Figure 9 on page 10) Only a very basic benchmark, working on more...
- drewg123 7y agoNice paper, and thanks for the reminder of how small the IOMMU tlb is. We never hit this because we were testing full-sized packets (and really bigger, because of TSO) and hit host IOMMU management overheads at ~100K to 200k TSO sends/sec.
- emmericp 7y agoInteresting, did you use huge pages? I think ~100k to 200k TSO "packets" per second should be doable with the IOMMU. But I guess it depends where the data is coming from. Could be one of the odd cases where copying data is faster than doing zero-copy, e.g., just copy everything into the same small set of small-ish buffers to keep the number of pages that need to be present in the IOMMU small?
- hedora 7y agoThis sounds like a driver bug / misconfiguration. Once the IOMMU is set up, there isn’t any host side management to be done (other than managing the IOMMU TLB, but unless it’s thrashing, that’s a no-op on the fast path). At most, each kernel driver has to do an extra addition to map its physical I/O offset to the one exposed to the bus by the IOMMU. With huge pages, there’s approximately one offset per driver, so it lives in cache, probably next to other driver state.
- ajross 7y agoFWIW: 64 entries isn't particularly small for a dTLB, IIRC that's exactly the size on current Intel cores. The real problem is that device DMA, unlike software behavior, is distressingly non-local. The device will stream out a packet or storage block and then never touch that memory again (or not for a very long time -- memory buffers are huge relative to bandwidth on these things). The TLB just doesn't do you much good.
- emmericp 7y agoThere's only one level of TLBs in the IOMMU. And that's 64 entries. Yeah, I think the dTLB is only 64 entries on Intel CPUs as well, but there's a second larger layer behind that, and an even larger third layer. IIRC it's a total of 4096 entries on recent Intel CPUs.
- the8472 7y agoCould p2p-dma for zero-copy (e.g. shoveling data from nvme to the nic) avoid a large fraction of the iommu overhead?
- waddlesplash 7y agoIdeally, one would only need to do this level of IOMMU for externally attached devices (i.e. Thunderbolt.) However, the problem is that the vast majority of kernel drivers in most OSes were not written with malicious devices in mind, so they often do things like e.g. putting kernel address space pointers in device-writeable areas. For instance, XHCI (USB3) has an "address" field in its buffer descriptor structures that often it allows drivers to put anything into, and it will copy that into the "transfer finished" message. Some USB3 drivers just put the kernel address space pointer of the bookkeeping record in there; or worse, allocate the bookkeeping records along with the physical buffers themselves, so a malicious device could manipulate those if it knew where to look.
- peterburkimsher 7y agoDoes the attack in its current form work over Thunderbolt? I'd really like to mount my RAM as a drive, and see the files that various programs have open (the waveform from Rogue Amoeba's Fission, among others). Using gore just gave me a 2GB raw file, and it's hard for me to spot files (JPG, PNG) in there.
- spacenick88 7y agoI think the most expensive isn't the address translation itself but TLB housekeeping when mappings get remapped or invalidated. This is especially true with virtualization where the hypervisor often needs to do extra work like (un)pinning guest pages, translating guest real addresses to host real and reissuing TLB flushes.
- the8472 7y ago> At one point I benchmarked a 100g setup on linux Datacenters have controlled physical access. I think IOMMU is far more important for anything with exposed thunderbolt ports (including the upcoming USB4). So Laptops, smartphones, workstations can still benefit from it even if it's currently not viable for cloud server-class workloads.