6 ms·

Using the IOMMU for Safe and Secure User Space Drivers [pdf]

65 points by emmericp 8y ago

drewg123 8y ago
I actually went straight to one of their references (https://gianniantichi.github.io/files/papers/pciebench.pdf https://gianniantichi.github.io/files/papers/pciebench.pdf) which was an amazing find. I should keep up more with the academic conferences, I guess. The most interesting / surprising thing about this paper is how damning it is for the Intel IOMMU.
- emmericp 8y ago
  Yeah, this paper is amazing; anyone considering using the IOMMU or interested in performance at the PCIe level should read it. We can also confirm their result about the IOMMU TLB size of only 64 on the CPUs we have here. (Yes, the TLB size is completely undocumented and there are no performance counters...)
snvzz 8y ago
This is best complemented with a multiserver, microkernel architecture. With an IOMMU, and with near everything including drivers running in userspace, all of this can be achieved: https://en.wikipedia.org/wiki/MINIX_3#Reliability_policies https://en.wikipedia.org/wiki/MINIX_3#Reliability_policies With a monolithic kernel such as Linux, the reliability benefits of running some drivers in userspace are negligible, as the TCB is still millions of LoCs with no proofs. Fuchsia, Genode and HelenOS are relatively active, promising open source operating systems with a microkernel, multi-server architecture and drivers running as unprivileged processes.
- 0815test 8y ago
  If comprehensive static proofs are in the picture, there's little benefit to running trusted code in user mode. It's just adding a protection boundary to something that could already be proven as safe via static analysis. IO-MMU however could still be useful since it somewhat obviates the need to trust the device itself, and devices are much harder to characterize properly.
  - snvzz 8y ago
    >If comprehensive static proofs are in the picture, there's little benefit to running trusted code in user mode. It's just adding a protection boundary to something that could already be proven as safe via static analysis. Try seeing it the other way around. It minimizes what needs proving (it isn't cheap) by having the microkernel (small and proven) enforce separation. Capabilities (not to be confused with POSIX capabilities, which are something else) do help massively here. Refer to the Genode Book[0] introduction chapter for an introduction to capabilities. In the driver example, running the driver unprivileged means the damage a faulty driver can be contained. In combination with a iommu, it can be made so that the hardware can only talk to the driver, to complete the protection. This is only possible if the kernel itself can be trusted, which would mean the kernel has to be proven, and that's unrealistic unless it's a microkernel, as it is a very costly process that doesn't even scale linearly with LoC. The alternative is to make all drivers part of the TCB, which isn't realistic for the same reason. Unfortunately, on top of this, driver code quality is known to be particularly bad. [0] https://genode.org/ https://genode.org/
ilovecaching 8y ago
This paper doesn't cover BPF or XDP, which is the future of high speed networking in the kernel. If you need to do user space processing and XDP can't meet your requirements, you can now use AF_XDP. DPDK and other user space networking was always a way of circumventing the kernel, now we have kernel approved methods for line speed network processing. XDP programs are actually safer than using Rust, because XDP programs are statically analyzed byte code that is guaranteed to finish executing, among other checks.
- emmericp 8y ago
  XDP is completely orthogonal to this work and would not have been a useful performance comparison. "Using AF_XDP" vs. "using a user space driver with IOMMU" is just apples vs. oranges. The goal we are trying to achieve in this project here is to show that drivers can (and should) be written in better languages to improve security and safety. Note that one of the drivers in the thesis is written in Rust. And regarding XDP being safer than Rust: Yes of course. But it's also very limiting; you can't write a driver in eBPF. (It currently just prohibits jumps with negative offsets but there's some ongoing work to allow for at least some bounded loops). We are interesting in making drivers themselves safer, not applications building on top of them. (As advisor of the thesis I however agree that XDP should have been mentioned; I'm not super happy with the length of the thesis but quite happy with the implementations)
  - ilovecaching 8y ago
    XDP is already being used to replace DPDK and IPVS use cases, so I don't see how its Apples to Oranges. It's generally not a good idea to circumvent the kernel, and in this case the kernel developers are providing better and better tooling for high speed networking to the point that user space driver implementations are no longer attractive compared to using the XDP ingress hook points. In industry I see rapid adoption of XDP (BPF in general is THE topic in Linux right now), and it's enabling all sorts of fascinating new use cases. I would be very interested to see a paper on iommu perf from the XDP perspective.
    - 8y ago
ilovecaching 8y ago
This paper doesn't cover BPF or XDP, which is the future of high speed networking in the kernel. If you need to do user space processing and XDP can't meet your requirements, you can now use AF_XDP. DPDK was always a way of circumventing the kernel, now we have kernel approved methods for line speed network processing.
sitkack 8y ago
I love where this work is going and I really enjoyed your ixy project. I see no mention of Dune [0] in the bibliography. Given the narrow focus of this research I think it is required. Once processors are tuned for cloud workloads, they will have the affordances that will make full blown operating systems optional. [0] http://dune.scs.stanford.edu/belay:dune.pdf http://dune.scs.stanford.edu/belay:dune.pdf

65 points by emmericp 8y ago

I actually went straight to one of their references (https://gianniantichi.github.io/files/papers/pciebench.pdf https://gianniantichi.github.io/files/papers/pciebench.pdf) which was an amazing find. I should keep up more with the academic conferences, I guess. The most interesting / surprising thing about this paper is how damning it is for the Intel IOMMU.

emmericp 8y ago
Yeah, this paper is amazing; anyone considering using the IOMMU or interested in performance at the PCIe level should read it. We can also confirm their result about the IOMMU TLB size of only 64 on the CPUs we have here. (Yes, the TLB size is completely undocumented and there are no performance counters...)

snvzz 8y ago

This is best complemented with a multiserver, microkernel architecture. With an IOMMU, and with near everything including drivers running in userspace, all of this can be achieved: https://en.wikipedia.org/wiki/MINIX_3#Reliability_policies https://en.wikipedia.org/wiki/MINIX_3#Reliability_policies With a monolithic kernel such as Linux, the reliability benefits of running some drivers in userspace are negligible, as the TCB is still millions of LoCs with no proofs. Fuchsia, Genode and HelenOS are relatively active, promising open source operating systems with a microkernel, multi-server architecture and drivers running as unprivileged processes.

0815test 8y ago
If comprehensive static proofs are in the picture, there's little benefit to running trusted code in user mode. It's just adding a protection boundary to something that could already be proven as safe via static analysis. IO-MMU however could still be useful since it somewhat obviates the need to trust the device itself, and devices are much harder to characterize properly.
- snvzz 8y ago
  >If comprehensive static proofs are in the picture, there's little benefit to running trusted code in user mode. It's just adding a protection boundary to something that could already be proven as safe via static analysis. Try seeing it the other way around. It minimizes what needs proving (it isn't cheap) by having the microkernel (small and proven) enforce separation. Capabilities (not to be confused with POSIX capabilities, which are something else) do help massively here. Refer to the Genode Book[0] introduction chapter for an introduction to capabilities. In the driver example, running the driver unprivileged means the damage a faulty driver can be contained. In combination with a iommu, it can be made so that the hardware can only talk to the driver, to complete the protection. This is only possible if the kernel itself can be trusted, which would mean the kernel has to be proven, and that's unrealistic unless it's a microkernel, as it is a very costly process that doesn't even scale linearly with LoC. The alternative is to make all drivers part of the TCB, which isn't realistic for the same reason. Unfortunately, on top of this, driver code quality is known to be particularly bad. [0] https://genode.org/ https://genode.org/

ilovecaching 8y ago

This paper doesn't cover BPF or XDP, which is the future of high speed networking in the kernel. If you need to do user space processing and XDP can't meet your requirements, you can now use AF_XDP. DPDK and other user space networking was always a way of circumventing the kernel, now we have kernel approved methods for line speed network processing. XDP programs are actually safer than using Rust, because XDP programs are statically analyzed byte code that is guaranteed to finish executing, among other checks.

emmericp 8y ago
XDP is completely orthogonal to this work and would not have been a useful performance comparison. "Using AF_XDP" vs. "using a user space driver with IOMMU" is just apples vs. oranges. The goal we are trying to achieve in this project here is to show that drivers can (and should) be written in better languages to improve security and safety. Note that one of the drivers in the thesis is written in Rust. And regarding XDP being safer than Rust: Yes of course. But it's also very limiting; you can't write a driver in eBPF. (It currently just prohibits jumps with negative offsets but there's some ongoing work to allow for at least some bounded loops). We are interesting in making drivers themselves safer, not applications building on top of them. (As advisor of the thesis I however agree that XDP should have been mentioned; I'm not super happy with the length of the thesis but quite happy with the implementations)
- ilovecaching 8y ago
  XDP is already being used to replace DPDK and IPVS use cases, so I don't see how its Apples to Oranges. It's generally not a good idea to circumvent the kernel, and in this case the kernel developers are providing better and better tooling for high speed networking to the point that user space driver implementations are no longer attractive compared to using the XDP ingress hook points. In industry I see rapid adoption of XDP (BPF in general is THE topic in Linux right now), and it's enabling all sorts of fascinating new use cases. I would be very interested to see a paper on iommu perf from the XDP perspective.
  - 8y ago

ilovecaching 8y ago

sitkack 8y ago

I love where this work is going and I really enjoyed your ixy project. I see no mention of Dune [0] in the bibliography. Given the narrow focus of this research I think it is required. Once processors are tuned for cloud workloads, they will have the affordances that will make full blown operating systems optional. [0] http://dune.scs.stanford.edu/belay:dune.pdf http://dune.scs.stanford.edu/belay:dune.pdf