6 ms·
Linux kernel framework for PCIe device emulation, in userspace
- tiernano 9mo agoHmmm.... Wondering if this could be eventually used to emulate a PCIe card using another device, like a RaspberryPi or something more powerful... Thinking the idea of a card you could stick in a machine, anything from a 1x to 16x slot, that emulates a network card (you could run VPN or other stuff on the card and offload it from the host) or storage (running something with enough power to run ZFS and a few disks, and show to the host as a single disk, allowing ZFS on devices that would not support it). but this is probably not something easy...
- xerxes901 9mo agoSomething like the stm32mp2 series of MCUs can run Linux and act as a PCIe endpoint you can control from a kernel module on the MCU. So you can program an arbitrary PCIe device that way (although it won’t be setting any speed records, and I think the PHY might be limited to PCIe 1x)
- jdub 9mo ago(Ha, nice to see Jon Corbet's name on the PCI Endpoint documentation...)
- tiernano 9mo agointeresting... x1 would too slow for large amounts of storage, but as a test, a couple small SSDs could potentially be workable... sounds like im doing some digging...
- cakehonolulu 9mo agoIf there's any particular feature you feel you are missing on PCIem or anything, feel free to open an Issue and I'll look into it ;)
- jacquesm 9mo agoThere are many workloads that would not be able to saturate even an x1 link, it all depends on how much of the processing can be done internally to whatever lives on the other side of that link. Raw storage and layer-to-layer communications in AI applications are probably the worst cases but there are many more that are substantially better than that.
- hsbauauvhabzb 9mo ago… or pcie over ethernet ;)
- topspin 9mo agoThat has a name: ExpEther[1], and likely more than one. pciem does mean you could do this with software. [1] https://www.expether.org/products.html https://www.expether.org/products.html
- pjc50 9mo ago> emulate a PCIe card using another device The other existing solution to this is FPGA cards: https://www.fpgadeveloper.com/list-of-fpga-dev-boards-for-pcie/ https://www.fpgadeveloper.com/list-of-fpga-dev-boards-for-pc... - note the wide spread in price. You then also have to deal with FPGA tooling. The benefit is much better timing.
- cakehonolulu 9mo agoIndeed, and even then, there's some sw-hw-codesign stuff that kinda helps you do what PCIem does but it's usually really pricey; so I kinda thought it'd be a good thing to have for free. PCIe prototyping is usually not something super straightforward if you don't want to pay hefty sums IME.
- immibis 9mo agoThe "DMA cards" used for video game cheating are generic PCIe cards and (at least the one I got) comes with open documentation (schematics, example projects etc).
- the_biot 9mo agoWhat's this? Hardware specifically for game cheating? Got any links?
- selectodude 9mo agoIf you search “DMA card”, there’s a lot of DMA cards all over the internet.
- idiotsecant 9mo agoDirect Memory Access (DMA) via PCI-e bypasses anti-cheat in the OS because the OS doesn't see the call to read or write the memory. There's no process to spy on, weird drivers, system calls, etc. You can imagine that maybe the anticheat could detect writes that perform a cheat by this method, but it has zero chance of detecting a wallhack style cheat that just reads memory. This is getting to be less relevant with modern OSs, though. Window 11 has IOMMU which only allows DMA to a given memory region defined per device. I think it should be impossible to do this on win11.
- cakehonolulu 9mo agoHi! Author here! You can technically offload the transactions the real driver on your host does to wherever you want really. PCI is very delay-tolerant and it usually negotiates with the device so I see not much of an issue doing that proven that you can efficiently and performantly manage the throughput throughout the architecture. The thing that kinda makes PCIem special is that you are pretty much free to do whatever you want with the accesses the driver does, you have total freedom. I have made a simple NVME controller (With a 1GB drive I basically malloc'd) which pops up on the local PCI bus (And the regular Linux's nvme block driver attaches to it just fine). You can format it, mount it, create files, folders... it's kinda neat. I also have a simple dumb rasteriser that I made inside QEMU that I wanted to write a driver for, but since it doesn't exist, I used PCIem to help me redirect the driver writes to the QEMU instance hosting the card (Thus was able to run software-rendered DOOM, OpenGL 1.X-based Quake and Half-Life ports).
- jacquesm 9mo agoFantastic tool, thank you for making this it is one of those things that you never knew you needed until someone took the time to put it together.
- gigatexal 9mo agoThis is really interesting. Could it be used to carve up a host GPU for use in a guest VM?
- cakehonolulu 9mo agoAs in, getting the PCIem shim to show up on a VM (Like, passthrough)? If that's what you're asking for, then; it's something being explored currently. Main challenges come from the subsystem that has to "unbind" the device from the host and do the reconfiguration (IOMMU, interrupt routing... and whatnot). But from my initial gatherings, it doesn't look like an impossible task.
- fc417fc802 9mo ago> carve up Passthru or time sharing? The latter is difficult because you need something to manage the timeslices and enforce process isolation. I'm no expert but I understand it to be somewhere between nontrivial and not realistic without GPU vendor cooperation. Note that the GPU vendors all deliberately include this feature as part of their market segmentation.
- Palomides 9mo agosome ARM chips can do PCIe endpoint mode, and the kernel has support for pretending to be an nvme ssd https://docs.kernel.org/nvme/nvme-pci-endpoint-target.html https://docs.kernel.org/nvme/nvme-pci-endpoint-target.html
- justsomehnguy 9mo agoAlready done https://mikrotik.com/product/ccr2004_1g_2xs_pcie https://mikrotik.com/product/ccr2004_1g_2xs_pcie and G-RAID
- immibis 9mo agoI recently bought a DMA cheating card because it's secretly just an FPGA PCIe card. Haven't tried to play around with it yet. Seems unlikely you'd emulate a real PCIe card in software because PCIe is pretty high-speed.
- wmf 9mo agoThis is what DPUs are for.
- MisterTea 9mo agoThis kind of stuff is stupid easy on an OS like Plan 9 where you speak a single protocol: 9P. Ethernet devices are abstracted and served by the kernel as a file system explained in ether(3). Since it's all 9P the system doesn't care where the server is running; could be a local in-kernel/user-space server or remote server over ANY 2-way link including TCP, IL, PCIe link, RS232 port, SPI, USB, etc. This means you can mount individual pieces of hardware or networking stacks like ip(3), any 9P server, from other machines to a processes local namespace. Per-process name spaces let you customize the processes view of the file system and hence all its children allowing you to customize each and every programs resource view. There is interest in getting 9front running on the Octeon chips. This would allow one to run anything they want on an Octeon card (Plan 9 cross platform is first class) so one could boot the card using the hosts root file system, write and test a program on the host, change the objtype env variable to mips/arm, build the binary for the Octeon and then run it on the Octeon using rcpu (like running a command remotely via ssh.) All you need is a working kernel on the Octeon and a host kernel driver and the rest is out of the box.
- 3PS 9mo agoThis is also the case with Google Fuchsia, just replace 9P with FIDL. I'm really hoping Fuchsia doesn't end up just being vaporware since it has made some very interesting technical decisions (often borrowing from Plan 9, NixOS, and others.)
- asdefghyk 9mo agoCould add one or more (reprograble?) FPGA's for extra? processing power OR reconfiguration ease to such a card ...... I've often wondered why such a card (with FPGA) is not available for retro? computer emulation or simulation ??
- hhh 9mo agothis is what dma cards do
- unsnap_biceps 9mo agoI ordered a pair of Orange PI 5+'s to work on playing with programming a PCIe device, but haven't made the time to get it working yet. https://blog.reds.ch/?p=1759 https://blog.reds.ch/?p=1759 and https://blog.reds.ch/?p=1813 https://blog.reds.ch/?p=1813 is what inspired me to play with it.
- tiernano 9mo agoohh now thats cool! Thanks for the links!
- Surac 9mo agothat is a huge win if you are developing drivers or even real hardware. it allows to iterate on protokols just with the press of a button
- cakehonolulu 9mo agoIndeed, the project has gone through a few iterations already (It was first a monolithic kernel module that required a secondary module to call into the API and whatnot). I've went towards a more userspace-friendly usage mainly so that you can iterate your changes much, much faster. Creating the synthetic PCI device is as easy as opening the userspace shim you program, it'll then appear on your bus. When you want to test new changes, you close the shim normally (Effectively removing it from the bus) and you can do this process as many times as needed.
- LarsKrimi 9mo agoLatching on to this thread, but can you make as simple as possible of an example? Something like just a single BAR with a register that printfs whatever is written
- cakehonolulu 9mo agoHi! I do have some rudimentary docs on which I made a simple device for example pruposes: https://cakehonolulu.github.io/docs/pciem/simple_device_walkthrough.html https://cakehonolulu.github.io/docs/pciem/simple_device_walk... Hopefully this is what you're searching for!
- LarsKrimi 9mo agoHi, thanks. That's almost it. The remaining problem is just how to tie it together (where do I put the handle_mmio_read pointer or which event should it be handled in?) PCIEM_EVENT_MMIO_READ is defined but not used anywhere in the codebase
- 9mo ago
- throwaway132448 9mo agoTangential question: PCIe is a pretty future-proof technology to learn/invest in, right? As in, it is very unlikely to become obsolete in the next 5-10 years (like USB)?
- neocron 9mo agoMight as well be replaced by optical connectors next years, but who knows in advance. Currently there is no competition
- tiernano 9mo agoeven though it would be optical, it still is using PCIe protocols in the background...
- embedding-shape 9mo agoHow could you possibly know exactly what protocol they'd be using for the potential future optical PCIe connection? Your guess is as good as anyone's, no?
- p_l 9mo agoProbably because optical PCI-E is an old thing by now. In fact, "zero~th generation" of thunderbolt used optical link, too. Also both thunderbolt and DisplayPort reuse a lot of common elements from PCI-E
- bobmcnamara 9mo agoPCIe is still using PCI protocol just over serdes
- pjc50 9mo agoHmm. What's the current maths on distance vs edge rate vs transceiver latency vs power consumption on when that would be a benefit? Not to mention how much of a pain it is to have good optical connectors. I wouldn't expect that to be mainstream until after optical networking becomes more common, and for consumer hardware that's very rare (apart from their modem).
- deleted 9mo ago[deleted]
- deleted 9mo ago[deleted]
- agent013 9mo agoI've been burned before by driver bugs that only manifested under very specific timing conditions or malformed responses from the device, tnx
- cakehonolulu 9mo agoAnytime, hopefully it fits your needs and helps you not spend more time than needed tracing issues like this. Thanks for the comment!
- JoshTriplett 9mo agoAny plans to upstream the kernel-side support?
- cakehonolulu 9mo agoI'd love to! Sure sounds like the natural next step for this.
- petabyt 9mo agovhci-hcd for USB has been so useful for usb development. Especially for testing usb driver code in CI.
- krupan 9mo agoSo just to be clear, you have to boot up the physical machine with a kernel command-line argument to reserve some RAM for this to work? And the amount of RAM you reserve is for BAR memory? If you wanted multiple PCIem devices (can you do that?) you'd need to reserve RAM for each of them?
- cakehonolulu 9mo agoHi! That's correct. We need a way to have a chunk of what Linux calls "Reserved" memory for the virtual BAR trick. Currently, PCIem only thinks about a single device (Since I first needed to have something that worked in order to check how feasible this all was), but there's planned support for multiple devices that can share a "Reserved" memory pool dynamically so you can have multiple BARs for multiple devices.
- iamoutoftouch 9mo agoHow is that better than emulating the device in QEMU or with something like libvfio-user (which also works on top of QEMU)?
- cakehonolulu 9mo agoI feel like libfvio-user is a cool project and works perfectly fine, that is, if you want to have the device on the host's userspace but exposed to a VM (QEMU, in this case). PCIem kinda does that, but it's down a level; in terms of, it basically pops the device on your host PCI bus, which lets real, unmodified drivers to interact with the userspace implementation of your card, no QEMU, no VM, no hypervisors. Not saying that you can then, for instance, forward all the accesses to QEMU (Some people/orgs already have their cards defined in QEMU so it'd be a bit pointless to redefine the same stuff over and over, right?) so they're free to basically glue their QEMU stuff to PCIem in case they want to try the driver directly on the host but maintaining the functional emulation on QEMU. PCIem takes care of abstracting the accesses and whatnot with an API that tries to mimick that the cool people over at KVM do.
- sedatk 9mo agoThat's pretty much the Linux equivalent of Device Simulation Framework we had for Windows back in 2000's. In the presentation below, only the USB capabilities of it is discussed, but it was able to simulate PCI devices too. https://download.microsoft.com/download/5/b/9/5b97017b-e28a-4bae-ba48-174cf47d23cd/dev098_wh06.ppt https://download.microsoft.com/download/5/b/9/5b97017b-e28a-...
- brcmthrowaway 9mo agoHow would I do this under macOS?
- cakehonolulu 9mo agoUnfortunately not with PCIem... I don't know how the XNU kernel does PCIe stuff; maybe something can be done there with a Kext module but no idea.
- _lunix 9mo agovery interesting work! I've been exploring a different idea on the side, using SPDK+libvfio-user [0] to emulate PCIe devices inside QEMU, which doesn't require a kernel module but it's a bit less flexible than this approach. [0] https://movementarian.org/blog/posts/2025-08-27-vfio-user-client-in-qemu/ https://movementarian.org/blog/posts/2025-08-27-vfio-user-cl...
- cakehonolulu 9mo agoHighly interesting! I kinda wanted not to rely on QEMU as a default "end" for the emulation (As in, I want the end user to be able to choose whatever userspace shim/transport layer/thing they want), but for some of my tests I did forward accesses to QEMU itself (And worked wonders). Thanks for that link! Super cool stuff!