5 ms·
Writing an NVMe Driver in Rust [pdf]
- tormeh 2y agoExtremely impressive. Does anyone know if performance is likely to decrease as more features are implemented? Because if not, this is a winner.
- gdiamos 2y agoThat is a bachelor’s thesis - someone should hire that student Clear writing and ideas
- eisbaw 2y agoLet him at least finish M.Sc. first. Then he can get dissapointed about the industry a bit later.
- rvense 2y agoGet them out now before they find out the truth about academia...
- dailykoder 2y agoA lot of bachelor thesis from TUM are very very impressive. Back in the day when Paul Emmerich[1] held his talks at the CCC about the user-space network drivers, I looked through a few of them and a good portion of them were "only" bachelor thesis'. I felt very bad when I compare them to my bachelor thesis. But on the other hand, I do have a friend now that is doing her masters at TUM and she says they sometimes have extreme difficulties getting their topics approved, because the TUM wants to stay "elite" and the students are under immense pressure. But hats off to the NVMe thesis. Well done! [1] https://www.net.in.tum.de/members/emmericp/ https://www.net.in.tum.de/members/emmericp/
- ddritzenhoff 2y agoCurrently a master's student at TUM, and I agree with your friend -- it's a painful journey haha.
- koverstreet 2y agoRust is a total gamechanger, and it's probably the thing that excites me the most about the future of kernel development.
- daghamm 2y agoIn what sense is it a game changer? A lot of memory and thread safety guarantees in Rust are only applicable to user space. Removing that, there are still some great things in Rust (e.g. enums ergonomics) but also a few questionable things.
- pornel 2y agoThis is a common misconception that if the kernel needs direct hardware access and such, then it all becomes unsafe. Rust programming style is building safe (zero-cost) abstractions on top of unsafe primitives, turning all other code into safe "glue" code. If you design for it, you can have a lot of "boring" code even in a kernel. A kernel will have more unsafe primitives to implement, but the safe/unsafe division still helps testing the unsafe parts, and still prevents bugs caused by misuse of these APIs. The type system that gives safety in userspace still exists in the kernel space. So even if the allocators and threads are different, you still have the type system tools to write safer APIs for them.
- tialaramex 2y agoYes. Because it's popular and large the Linux kernel has problems where abstractions leak and a subtle implementation detail causes problems because you were supposed to just know that certain APIs don't do quite what it seems like they'd do from the name. Rust's culture says you must mark abstractions that leak safety as unsafe. If this Rust function named "make_doodad" is labelled safe, it is not OK that I could run it without a doodad_manager, and yet in this case it blows up. Either somehow require me to prove I have a doodad_manager, or, mark it unsafe and document the requirement, or re-design the function so that it checks for a doodad_manager and fails cleanly when one is not present. In some cases you might decide all three are needed: make_doodad_with_manager(&manager) -> Doodad, unsafe make_doodad_unchecked() -> Doodad and make_doodad() -> Result<Doodad,Problem>
- drv 2y agoIt's cool to see more systems code written in Rust! I also previously worked on SPDK, so it was neat to see it being chosen as a point of comparison. However, I was waiting for the touted memory safety to be mentioned beyond the introduction, but it never really came up again. I was hoping for the paper to make a stronger argument for memory-safe languages like Rust, something like "our driver did not have bugs X, Y, and Z, which were found in other drivers, because the compiler caught them". Additionally, in a userspace device driver that is given control of a piece of hardware that can do DMA, like an NVMe controller, the most critical memory safety feature is an IOMMU, which the driver covered by the paper does not enable; no amount of memory safety in the driver code itself matters when the hardware can be programmed to read or write anywhere in the physical address space, including memory belonging to other processes or even the kernel, from totally "safe" (in Rust semantics) code. While the driver from the paper may certainly have a "simplified API and less code", I don't expect much of this to be related to the implementation language; it's comparing a clean-sheet minimal design to a project that has been around for a while and has had additional features incrementally added to it over time, making the older codebase inevitably larger and more complex. This doesn't seem like a particularly surprising result or an endorsement of a particular language, though it perhaps does indicate that it would be useful to start from scratch now and again just to see what the minimum viable system can look like. I certainly would have liked to rewrite it in Rust, but that wasn't really feasible. :) In any case, it's great to see proof that a Rust driver can have comparable performance to one written in C, since it will hopefully encourage new code to be written in a nicer language than C. I definitely don't miss having to deal with manual memory management and chasing down use-after-frees now that I write Rust instead of C. (As a side note, I'd encourage anyone thinking of using a userspace storage driver on Linux to check out io_uring first before going all in; if io_uring had existed before SPDK, I don't know that SPDK would have been written, given that io_uring gets you most of the way there performance-wise and integrates nicely with the rest of the kernel. A userspace driver has its uses, but I would consider it to be a last resort after exhausting all other options, since you have to reinvent all of the other functionality normally provided by the kernel like I/O scheduling, filesystems, encryption, etc., not just the NVMe driver itself. That is, assuming the io_uring security issues get resolved over time, and I expect they will.)
- 2y ago
- bengl3rt 2y agoI only read the paper (and the code in the paper) but not the complete source code, so maybe this would become clearer if I had, but... Does Rust fundamentally guarantee that if you make a struct, its fields will lay out in memory in the order that you defined them? Can it be used to interact with APIs (really ABIs) who expect a C struct (or pointer to one)? I think my main frustration with stuff like Go and Swift in this case is that their structs are not binary-compatible with C structs in this way because they rearrange things to be better aligned/packed/whatever.
- Hemospectrum 2y ago> Does Rust fundamentally guarantee that if you make a struct, its fields will lay out in memory in the order that you defined them? Can it be used to interact with APIs (really ABIs) who expect a C struct (or pointer to one)? You have to specify this behavior with #[repr(C)]. Otherwise, the compiler will rearrange fields to try to optimize packing and alignment.
- mirashii 2y agoOr, more precisely, reserves the right to do so in a future version. rustc does not currently do this. There are some compiler flags to randomize the struct layout just to make sure that you’re not implicitly depending on the order. https://doc.rust-lang.org/reference/type-layout.html https://doc.rust-lang.org/reference/type-layout.html https://github.com/rust-lang/compiler-team/issues/457 https://github.com/rust-lang/compiler-team/issues/457
- tialaramex 2y ago> Or, more precisely, reserves the right to do so in a future version. rustc does not currently do this Does not currently do what ? Rust certainly will re-arrange the layout of a default repr(Rust) struct to make it smaller, for example: https://rust.godbolt.org/z/7KsqvnE9o https://rust.godbolt.org/z/7KsqvnE9o [Edited to provide a nicer Godbolt example which compares the two layout strategies]
- anonymousDan 2y agoOne question that is unclear to me - what are the disadvantages of a user space driver like SPDK (in comparison to a kernel implementation)? It prevents multiplexing the device? The API is more complex to program?
- ongy 2y agoIt's not protected by the copy-left properties of the kernel D: For the more technical points: The kernel needs to have it available in a somewhat complex way to be able to mount the drive. I.e. init becomes a lot more involved with an initram that first needs to load the driver (potentially loopback it?) and the mount actual root. To some degree there can also be issues around syscall boundaries. I.e. the usual monolith vs. microkernel. I haven't checked the API they hook into to provide the device to other components. But it likely requires the kernel to jump back into userspace in various "hot-ish" paths for IO.
- ongy 2y agoOk, it doesn't integrate with the system at all. This provides a library to access the device. I.e. it can only be used by a single consumer, that consumer doesn't get nice things like ther kernel's file systems or device mapper (raid/crypt/verity...) features. This can be fine, e.g. when the consumer is a database that just needs a block device, or something like ceph/minio etc. that provides the storage api to its consumers. It'd have to use something like NBD (~fuse for block devices) to actually integrate and then my previous post describes some of the downsides to that setup.
- moonshotideas 2y agoGiven the statement ‘we achieve SPDK-like throughput,’ I’m curious whether the performance is slightly worse than SPDK. If it is, do you have any comparison metrics for throughput?
- moooo99 2y agoThis is extremely impressive work, especially for a bachelor thesis
- Kydlaw 2y agoReally cool and impressive work! I'm more on the software (high level) side than on the hardware, but I always wondered how USBs and other ports were truly working. It is an area of knowledge that I'm shamefully lacking... Does anyone have nice resources to share like this one that focuses on a specific port/connection and implement a driver/reader/parser? I'd very like to learn more of this.
- nielsole 2y agoBen eater has videos on USB, SPI and rs232. Bitluni has a couple of easily consumable videos about VGA.
- virtue3 2y agosecond on ben eater; the USB one showing inputs on a keyboard and such with an oscilloscope blew my mind. I was like -> oh, it really just is a huge bus and there's A LOT happening.
- MisterTea 2y agoPlan 9 and C but there is some good info in these videos: https://www.youtube.com/@adventuresin9797/search?query=usb https://www.youtube.com/@adventuresin9797/search?query=usb
- ajross 2y ago"Big" hardware like a USB3 controller or the NVMe controller in the linked article actually looks more like software than hardware anyway, to be truthful. The devices have their own processors and accept commands that look like a software protocol: you put a linked list of these structs/headers/commands into memory, which reference buffers at these addresses via pointer, then turn the device on by writing to this register over here. As the device finishes, it signals completion with a given struct by putting a message into this output buffer (or by setting its "active" field to false, maybe), etc... The only bits you need to worry about that looks like hardware are getting the correct physical addresses filled in, and on some platforms worring about memory ordering and/or cache management to be sure the device sees the same memory state you think it does.
- ajross 2y agoFor those curious: this is a userspace driver, written to open/mmap the PCI BAR via sysfs. It doesn't attempt to hook interrupts from the device and has to resort to polling for completion[1]. Also AFAICT there's really no attempt being made at parallel design: all the hardware interaction is handled by a single manager thread responding to queued commands from workers. That's a fine choice for simplicity and reliability but a very poor one for scalability in a storage backend that needs to be handling requests from filesystem drivers serving dozens or hundreds of CPUs. Basically: this is good work (great work for an undergraduate thesis). But it's very much "solving the easy part" and not really showing off Rust in any particularly impressive way. You can write very similar userspace "drivers" (and I have!) in Python. [1] Though on modern hardware designed to manage arbitrary scatter/gather command queues on its own, that's not really such a big deal. In performance situations the hardware will always have something to do anyway, and idle hardware can be sent a command synchronously. Fixing this amounts to a power optimization only.
- FullyFunctional 2y agoI didn't read the whole thing, but the author briefly mentions VFIO which would have given him access to interrupts so it would have been possible (but for flat-out performance, polling is the only thing fast enough). I wrote a small NVMe user-space driver using VFIO more than a decade ago this way. Coming from having virtualized ATAPI (SATA) and SCSI, NVMe was such a refreshingly excellent design.
- mgt19937 2y agoI used to think read is always faster than write in ssd. But in figure 5.3 and figure 5.4, it looks like in ssd, read iops is lower than write iops. When queue depth is low(like qd=1), random 4k read iops is far more less(14.5 kiops vs 128 kiops) than 4k random write iops. When queue depth is high, like qd=32, the read iops and write iops becomes similar. But read iops is still less than write iops.(436 kiops vs 608 kiops) I wonder why read is slower than write? Is it because ssd has a fast write cache, and it will finish the write request once the data is written into cache? Or it simply report that the data is written and actually write them in batch in background?