4 ms·
Ask HN: Looking for a resource on Linux kernel module development
I'd like to break into this area in my next role and I'm looking for resources to learn about this over the next few weeks so I can try to get a role in the area. My context is embedded linux.
EDIT: I don’t know what I don’t know so I’m looking for a “point in the right direction” and anything the fine people of HN think is a good resource
- jfdi 3y agoWhat kind of resource are you looking for? Have you already gotten started with your LLM of choice, and need something more?
- matt3210 3y agoI don’t know what I don’t know, so I need a “point in the right direction”.
- SanderNL 3y agoJust here to note that it is amazing I can just copy-paste the request and get a summary with all your guys’ suggestions. Don’t forget "Embedded Linux Primer" by Christopher Hallinan.
- synergy20 3y agoit's a good book, but it's 13 years old. in fact, most of linux kernel books are aged these days.
- yxre 3y agoI am also learning about the linux kernel, but not in an embedded context. Here are some good resources I have found. https://man7.org/tlpi/ https://man7.org/tlpi/ This book is considered the best overview of linux kernel interface. Its old, but still gives a broad coverage of topics. https://lkml.org/ https://lkml.org/ Browsing LKML is really helpful since it shows you where current development is happening and how kernel devs think. Some conversations get very advanced and nuanced. https://kernelnewbies.org/ https://kernelnewbies.org/ is a good starting place for starting to compile and deploying kernel builds Other than that, reading and tracing the source is probably the most productive thing to do for an experienced dev to do
- robert72127 3y agoGeneral intro to module development: https://sysprog21.github.io/lkmpg/ https://sysprog21.github.io/lkmpg/ https://linux-kernel-labs.github.io/refs/heads/master/so2/index.html https://linux-kernel-labs.github.io/refs/heads/master/so2/in... Drivers: https://www.amazon.com/Linux-Device-Drivers-Development-customized/dp/1785280007 https://www.amazon.com/Linux-Device-Drivers-Development-cust... https://www.oreilly.com/library/view/linux-device-drivers/0596005903/ https://www.oreilly.com/library/view/linux-device-drivers/05...
- matt3210 3y agoThanks a ton!
- scrollaway 3y agoYou could look for existing active current or ex kernel module developers and pay for a couple hours of their time to guide you through the weeds and give you early pointers. This is my approach when breaking into a new field if I have access to people.
- matt3210 3y agoGood advice, I'll keep an eye out for experts willing to teach
- convolvatron 3y agomail me at gmail
- Hackbraten 3y agoIf you prefer reading, you may want to look into Kaiwan N. Billimoria’s books: https://www.packtpub.com/authors/kaiwan-n-billimoria https://www.packtpub.com/authors/kaiwan-n-billimoria I’m halfway into their first Linux Kernel Programming book and like it so far.
- matt3210 3y agoGood suggestion, thanks!
- asicsp 3y agoSee also: Linux kernel and its insides (https://0xax.gitbooks.io/linux-insides/content/index.html https://0xax.gitbooks.io/linux-insides/content/index.html)
- matt3210 3y agoVery nice, thanks!
- beeburrt 3y agoMaybe Joseph Kong's books? Designing BSD Rootkits https://nostarch.com/rootkits.htm https://nostarch.com/rootkits.htm FreeBSD Device Drivers https://nostarch.com/bsddrivers.htm https://nostarch.com/bsddrivers.htm Playing Games With Kernel Memory ... FreeBSD Style http://phrack.org/issues/63/7.html#article http://phrack.org/issues/63/7.html#article
- matt3210 3y agoVery nice, a BSD prospective helps a lot!
- spacemadness 3y agoA complete tangent here, but is there any modern Phrack like zine? Other than Phrack of course which seems to have had a brief rebirth with their October 2021 release.
- mfrw 3y agoLinux Kernel Development by Robert Love is something I would encourage going through if you are unsure about what you don't know. Although it's based on 3.x kernel, it still is pretty relevant. Good Luck!
- matt3210 3y agoI'll need it, thanks!
- nottorp 3y agoSemi related piece of advice: Also get a random raspberry pi and a simple peripheral for it: some 1 wire temperature sensor or something on i2c. Then ignore the existing drivers and do your own. Do a character device and have a read from it trigger a read in kernel and copying of the data to user space, maybe in human readable form. Should be good practice. And for general embedded Linux development you also want to take a look at Yocto.
- matt3210 3y agoThis is great advice thanks!
- realo 3y agoYocto is fine, basically the "traditional" way of doing embedded systems (e.g pSOS or vxworks style) . However, I found great joy in using Nix for more modern development t of higher-end embedded devices. If your target platform is powerful enough to accommodate NixOS, you might want to give it a try.
- nottorp 3y agoNix seems to advertise itself as something to do reproducible builds with, which is damn nice. But what's the extra effort if your hw vendor doesn't support it? The examples on their homepage look more like cloud computing not embedded.
- smashed 3y agoYocto has made great progress into fully reproducible builds too, just fyi..
- realo 3y agoYou are totally right... Nix is cool for advanced CPUs (i.e. x86, armv7, aarch64, etc) and much less so for microcontrollers. But if your platform can run NixOS ... then ... let's see... With nix I have a file that exactly describes the git sources (pinnable on precise changesets if required) for all my components (including custom kernel patches, device drivers, etc). I use a nix config to create a fully GPLV3 compliant rootfs and another set of nix config files in a separate repo to create our device specific intellectual property, in a separate encrypted image different from the rootfs. The nix config files easily describes everything i need to fully support Secure boot, as well as code to unseal a TPM luks password used to decrypt the image that contains our intellectual property for that device (basically the application). All this is fully reproducible, so I can provide our clients with the nix config files for our rootfs, tell them to use nix and they can reproduce our exact rootfs image to run in the device , for GPLv3 compliance. Of course , then the device is "broken" , because our IP will not be decryptBle anymore. But the GpLv3 does not mandate full functionality when running foreign images... Now I am absolutely certain all this can be also done with Yocto... but with less simplicity ... less joy ... :) I used pSOS in the past, got burned when Integrated Systems was sold... then stayed away from proprietary OSes as much as I could. Used Debian with XEN later on... then Yocto in many products, then back to a Debian-like OS and now , finally, using Nix. It just works.
- oasisaimlessly 3y agoWith VFIO, there is increasingly little need for writing in-kernel device drivers. You can do everything in userspace with better DX and without having to deal with kernel API breakage. If you want to try out VFIO, you could try writing a userspace program that interfaces with some simple PCI device, like a serial card.
- matt3210 3y agoGood point, can you point me in the right direction for more info? Edit: A cursory glance suggests this is for connecting existing devices to virtualized guests, how does that replace kernel modules for hardware components?
- oasisaimlessly 3y agoPaaaing hardware to VMs is the "killer app" for VFIO, but VMs (e.g. QEMU) are just normal userspace processes. In other words, VFIO can delegate control of a device to any userspace process, but it's most commonly used for VMs. DPDK [1] is an example of a non-VM VFIO user. [1] https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html
- synergy20 3y agofor userspace drivers check out UIO. VFIO is still mostly about virtual machines, as a beginner I won't touch it for now. I will stay in the kernel for a while before doing userspace drivers.
- tux3 3y agoYou could try looking up the tasks of the Eudyptula Challenge. More than half the value was in sending mails and getting feedback, but it's still a great introduction that gradually gets you used to more and more complicated kernel dev There are solutions lying around on the internet for the various tasks, but I'd advise not looking. The kernel is sparsely documented (some areas, rigorously undocumented), so a bit of experience searching and figuring out how to get the information on your own is genuinely valuable
- bitcoinmoney 3y agoI think it’s not there anymore ?
- tux3 3y agoThe challenge is closed, but it's really just a series of emails with tasks for you to do. You can find a list of the tasks online, only there won't be someone to verify your solutions since the challenge is closed.
- smashed 3y agoSince your context is embedded Linux, I would recommend following Raspbian development for general purpose SBC embedded Linux or OpenWRT for smaller, more specialized devices. Both systems have code bases and build systems that will build root filesystem and kernels for a wide range of devices. By following development of a device (aka target) that interests you, you will see how drivers are stabilized with various patches applied over time.
- matt3210 3y agoGood advice, thanks!
- wilburm 3y agoSince you’re in embedded Linux, I’d suggest some of the bootlin learning material. Something like an i2c or spi driver might be good. QEMU is nice for learning because you can more easily debug kernel code with gdb.
- matt3210 3y agoGood advice, thanks!
- cushychicken 3y agoBootlin’s training for embedded Linux and driver development is second to none.
- Roark66 3y agoI second advice to focus on embedded systems, but I'd stay away from raspberry pi. I'd much prefer something you could debug easily from outside. There are plenty of arm chips one can debug with openocd and gdb. Sometimes being able to debug a running kernel from the outside can make a huge difference when troubleshooting. If also choose some class of device and get to know it. Many Linux drivers are very similar across a class of devices. For example CSI cameras on top of V4L. Thankfully the code in the Linux kernel is very readable. Also it is well documented online. I definitely recommend a good ide too.
- cvccvroomvroom 3y agoThere's a book on the topic. 978-0672329463
- westurner 3y ago- "The Linux Kernel Module Programming Guide" (2023) https://news.ycombinator.com/item?id=35782630 https://news.ycombinator.com/item?id=35782630 - Still trying to find the How To Build a Formally Verified Kernel Module from Zero tut Edit: - /? Kernel module https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=%20kernel%20module&sort=byPopularity&type=story https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... - "Linux Kernel Module written in Scratch (a visual programming language for kids)" (2022) https://news.ycombinator.com/item?id=31921996 https://news.ycombinator.com/item?id=31921996 (EduBlocks does Scratch with Python blocks and/or text code) - "How to Discover and Prevent Linux Kernel Zero-day Exploit using Formal Verification" (2021) [w/ Coq] http://digamma.ai/blog/discover-prevent-linux-kernel-zero-day-exploit-using-formal-verification/ http://digamma.ai/blog/discover-prevent-linux-kernel-zero-da... https://news.ycombinator.com/item?id=31617335 https://news.ycombinator.com/item?id=31617335