10 ms·

Does anyone have a story or two of a time you’ve created a kernel module to solve a problem? I would be interested in hearing real world use cases.

samtho 9y ago
I wrote one that decodes Wiegand[0] off of GPIO. I ended up not using it because epoll-ing the pins is easier to use from a userspace perspective and fast enough to capture the transmission. I loosely based the kernel module off of one that was written for another microcontroller[1]. [0]: https://en.wikipedia.org/wiki/Wiegand_interface https://en.wikipedia.org/wiki/Wiegand_interface [1]: https://github.com/rascalmicro/wiegand-linux https://github.com/rascalmicro/wiegand-linux
- ianhowson 9y ago
  I did the same but kept it in-kernel to meet Wiegand timings. I had (horrible) another driver which could hang the system for long enough to lose bits. I was in Australia for that work too...
deleted 9y ago
[deleted]
sly010 9y ago
I didn't end up writing the kernel module, so this might be an anti use case: I was working on an embedded system and needed to have fast I2C access (mostly just wanted very small latency bc it's an instrument). I2C from Linux userspace (using ioctls) adds a lot of overhead. I started looking into kernel modules but after a day of research I found out that you can access hardware registers from userspace using mmap("/dev/mem") which is even faster than a kernel module. Edit:typos
- zeusk 9y ago
  > mmap("/dev/mem") but that would only work if some kernel module or the kernel itself hasn't mapped that IO space, right?
  - sly010 9y ago
    I don't think mmap stops you from mapping the same memory twice, nor does linux. In my case I didn't even disable the system I2C driver. If course if I tried to use it I would get io errors and if I am not careful I can crash the kernel, but that's what you get for poking around. This was a raspberry pi.
    - zeusk 9y ago
      Yeah, that's what I meant since i2c is a serial protocol and most drivers assume mutual exclusion on device registers.
megous 9y ago
Usually to control on-SoC peripherals/IP blocks or to add support for other external devices (that's what kernel's for, mostly, anyway) - camera sensor, I2C controlled regulator, thermal sensor,... story? no. :D I just wanted to have a nice non-amd64 SoC based desktop, and some things were missing at the time.
alxlaz 9y ago
Most Linux device drivers are kernel modules, so I suppose most of the stories I have would be of the form "I needed to write a device driver for X" :-). Userspace drivers, while certainly in existence, are not very popular in Linux land. Other interesting stories would include: - Instrumenting certain filesystem operations (all modules share the same memory space; it's possible for one module to take a sneak peak at another's internal structures). This was back before dtrace & friends were a (useful) thing. - Real-time processing of instrumentation samples. Doing it in the kernel allowed us to avoid costly back-and-forth between user and kernel memory -- but we only did relatively simple processing, such as scaling, channel muxing/demuxing and the like. If you find yourself thinking about doing kernelside programming because carrying samples to and from userspace is too expensive, you should probably review your design first.
- craftyguy 9y ago
  > Userspace drivers, while certainly in existence, are not very popular in Linux land Actually, that's not true at all! Just look at Mesa3D and any number of GL proprietary blobs from graphics vendors. Userspace drivers are actually quite common, they just aren't what folks think of immediately when they think about 'drivers'.
aseipp 9y ago
Yes, I wrote one a long time ago that exposed basic counters from the ARMv7 PMU to userspace -- where they are normally not accessible unless used through perf_events. This is because perf wasn't available (I think the shitty old vendor BSP I was using couldn't support it for some reason) and it's relatively costly (perf_event_open is a whole syscall + another syscall to `read` off the event info) -- I just wanted some basic cycle timings for some cryptography code I wrote to see what worked and what didn't. Even though it has some annoying gotchas (such as the fact ARM cores can sleep/frequency scale on demand with no forewarning, meaning cycles aren't always the most precise units of measurement), and is very simple -- this thing ended up being mildly popular. Even though I wrote it years ago, someone from CERN recently emailed me to say they happily used it for work, and someone from Samsung ported it to ARMv8 for me... (I should dust off my boards one of these days and clean it up again, perhaps! People still email me about it.)
mbrumlow 9y ago
I wrote one so I could keep my job... I work at a company that provides backups for both Linux and windows. The entire concept was around block level backups. You could just open up the block device and copy the data directly but it would quickly become out of sync by the time you finished copying it. We did not want to require LVM to be able to utilize snapshots to solve the sync problem. On top of that we had a strong requirement of being able to delete data from the backup. This resulted in me learning how to build a kernel module and then slowly over about 6 months creating a kernel driver that allowed us to take point in time snapshots of any mounted block device with any fs sitting on top. Other requirements also dictated that we keep track of every block that changed on the target block device after the initial backup (or full block scan, after reboot). I wish I could release the source but my employers would not like that :( So at least for me learning how to write kernel modules and digging in to some of the lower stuff has keep me gainfully employed over the years. It is still in use on about 250k to 300k servers today (it fluctuates). The hardest part was not writing the module, but getting others interested in it enough so I don't have to be the sole maintainer. I like working on all parts of the product and don't want to just be the "kernel guy". One other time I wrote a very poorly done network block device driver in about 8 hours. You can find it here https://github.com/mbrumlow/nbs https://github.com/mbrumlow/nbs -- Note I am not proud of this code, it was something I did really quick, wanted it on hand to show to a perspective employer -- I did not get the job, I am also fairly sure they did not even look at the driver, so I don't think the crappy code there affected me.
- throwaway613834 9y ago
  So you implemented something like Windows's volume snapshots/shadow copies? (Why couldn't you use the existing feature?) EDIT: Thanks for both replies! :)
  - aseipp 9y ago
    Pretty much. But Windows snapshots aren't writeable after you take them, which is a major downer and a major necessary feature. (Having this feature lets people do things like "Take snapshot and delete everything except *.lic files, under C:\Users, on every backup" and you can just issue deletes to the shadow volume after it is consistently created, and intercept the new deletions, like normal. Tracking deletions like this lets you more easily know what you should/should not back up -- only back up the allocated blocks in the shadow copy.) So, there was a thing that added this feature to the Windows kernel in the product to make this work. Aside from the Linux stuff, which was totally separate. But if you don't need the writing capability, Shadow Copies are good enough, sure.. (Source: I used to work with the guy who made the above post)
ryandrake 9y ago
This was a super-long time ago, but a kernel driver was needed for video capture back in the 90's, to control devices across I2C, etc. At the time I had a Matrox video card which did analog capture, decoding and TV-in, and so wrote the first few iterations of a kernel driver[1] to get it to work. It since got adopted by a great maintainer. Nowadays I'd guess nobody is using it because the hardware is ancient. 1: http://www.cs.brandeis.edu/~eddie/mga4linux/ http://www.cs.brandeis.edu/~eddie/mga4linux/
invernomut0 9y ago
Some years ago I was working on an embedded platform with two processors connected via ethernet, one handling data and most of the application load, the second one handling some automotive buses (CANbus and ARCnet). I wrote a kernel driver to expose raw data coming from the ARCnet to userspace applications in the most transparent way. So I labeled data coming from the second cpu with an unused ethertype, I changed the ethernet driver to inspect the ether type of the incoming data and send it up inside the network stack as if it was received from a certain fake ARCnet device. The fake ARCnet device was a dummy module that allowed userspace to receive that data via normal sockets. Probably it could be done in a better way but it was quite simple and it worked well.
andrewcchen 9y ago
I modified a kernel module to use the wifi enable/disable button on my hp laptop as a normal button. For some reason the bios sends an acpi event when the button is pressed, instead of just having it as a normal keyboard key, so there is a kernel driver to handle it.
alecdibble 9y ago
A few years ago, I worked at a company that had hardware peripherals conntected to a master CPU card running linux through RS-232. One of the peripherals was a power supply that supplied power to another peripheral. Since they were architected as Slaves, the peripherals couldn't directly talk to each other, they had to get the messages relayed through the master CPU. This whole communication process had to happen within a small time frame, something like 50 ms. A kernel module handled the communication, decoded the packets, and automatically relayed peripheral messages to other peripherals. The kernel module made it easier to achieve stable and real-time operation within the time constraint. When writing the same function in userspace, there was absolutely no guarantee that messages would be sent or received in time.
zb3 9y ago
I wrote one so that I could set hardware breakpoints and use them to extract decrypted data from an app which was heavily protected against debugging. The thing was that the app (running as an unprivleged user) had no way of knowing what I was doing in the kernel (uprobes can be used for this too providing the code doesn't checksum itself). But I guess reverse engineering is not a common use case for writing linux kernel modules though :)
- anitil 9y ago
  Maybe not common, but super interesting if you want to share more.....
  - zb3 9y ago
    I guess that kernel modules are not commonly used for RE because in general there are better tools that support emulation, symbolic execution, dynamic instrumentation and so on (such as qira, vtrace, angr, Intel Pin and more, there are many articles about them). I just wanted something simple and portable to extract/modify registers/memory pointed to by registers/unwind stack when a given instruction was executed, while being as fast and transparent as possible, so I wrote a kernel module... and while it worked(hook mmap via kprobes to get base address + register_user_hw_breakpoints + printk to get output), using it was alot of manual work, and I'm sure tools I listed above could be more efficient. Also since there was no emulation, I had to spoof opening /dev/ urandom, which you don't have to do if you're using qira It'D be great if My small module Could be published As is, but I'd need to strip all addresses/filenames, and also add more proper locking (but yes, I want to publish it anyway).
woodrowbarlow 9y ago
i was working on an embedded device with only RAM and a flash chip for the software blob. i wrote a kernel module that allowed us to keep logs in RAM even across a soft reboot (i.e., if the device crashes or does a software update). it basically reserved a chunk of 1M at the top of RAM (using the same physical address each time). there was a checksum system that allowed it to tell during boot whether the data is still valid or if the bits had decayed. i've also done a couple i2c drivers for temperature sensors or LED controllers. kernel work is fun.
stevekemp 9y ago
I've written simple Linux Security Modules, as learning experiences, but also to help tighten up a bunch of virtual machines. The one that I'm most proud of delegates decisions on whether executables can be run to .. userspace. Which is simultaneously evil and brilliant. i.e. User "foo" tries to run "/tmp/exploit" and the kernel executes "/sbin/can-exec foo /tmp/exploit". If the return code of that is zero then the execution is permitted otherwise it is denied. This gives you ample opportunity to log all executables, perform SHA1-hash checks of contents, or deny executables to staff-members outside business hours. There's a lot of scope for site-specific things, creativity is the limit!
- bincat 9y ago
  I think it would be much appreciated if this software could be released as open source. How did the user space redirection handle potentially high spikes of execve's? And btw, thanks for all the years of debian-administration.org.
ship_it 9y ago
Not for Linux per se but I've been developing a variant of CAN(+FD) kernel extension for XNU/OSX. Some things are manageable through userland so not everything is done in kernel but extending berkeley sockets is really fun and I'd suggest anyone interested to try it.
lloeki 9y ago
Hardly a problem per se but I was bored on a long train trip (7+ hours) and I just happened to have (on purpose) a Xbox One controller around, so I wired it in, hacked around on userland with libusb, then wrote a kext for the thing to appear as a proper HID device on OS X. Two lessons learned: 1. dealing with tangible things for once is incredibly satisfying. 2. next time, download the fscking docs instead of relying on an intermittent 3G connection. https://github.com/lloeki/xbox_one_controller/ https://github.com/lloeki/xbox_one_controller/
souprock 9y ago
My real-world use case is a hypervisor. That turns out to be some pretty wild code. Intel's VMX documentation is not well-organized or complete, processor models vary, the hardware has bugs, and some things we have to do aren't even officially supported by Intel. We have to mess with page tables. We have a huge block of assembly code. It all has to run perfectly. In case that doesn't make you want to run away screaming, I put a post in the "Who is hiring?" article: https://news.ycombinator.com/item?id=15829518 https://news.ycombinator.com/item?id=15829518

samtho 9y ago

I wrote one that decodes Wiegand[0] off of GPIO. I ended up not using it because epoll-ing the pins is easier to use from a userspace perspective and fast enough to capture the transmission. I loosely based the kernel module off of one that was written for another microcontroller[1]. [0]: https://en.wikipedia.org/wiki/Wiegand_interface https://en.wikipedia.org/wiki/Wiegand_interface [1]: https://github.com/rascalmicro/wiegand-linux https://github.com/rascalmicro/wiegand-linux

ianhowson 9y ago
I did the same but kept it in-kernel to meet Wiegand timings. I had (horrible) another driver which could hang the system for long enough to lose bits. I was in Australia for that work too...

deleted 9y ago

[deleted]

sly010 9y ago

I didn't end up writing the kernel module, so this might be an anti use case: I was working on an embedded system and needed to have fast I2C access (mostly just wanted very small latency bc it's an instrument). I2C from Linux userspace (using ioctls) adds a lot of overhead. I started looking into kernel modules but after a day of research I found out that you can access hardware registers from userspace using mmap("/dev/mem") which is even faster than a kernel module. Edit:typos

zeusk 9y ago
> mmap("/dev/mem") but that would only work if some kernel module or the kernel itself hasn't mapped that IO space, right?
- sly010 9y ago
  I don't think mmap stops you from mapping the same memory twice, nor does linux. In my case I didn't even disable the system I2C driver. If course if I tried to use it I would get io errors and if I am not careful I can crash the kernel, but that's what you get for poking around. This was a raspberry pi.
  - zeusk 9y ago
    Yeah, that's what I meant since i2c is a serial protocol and most drivers assume mutual exclusion on device registers.

megous 9y ago

Usually to control on-SoC peripherals/IP blocks or to add support for other external devices (that's what kernel's for, mostly, anyway) - camera sensor, I2C controlled regulator, thermal sensor,... story? no. :D I just wanted to have a nice non-amd64 SoC based desktop, and some things were missing at the time.

alxlaz 9y ago

Most Linux device drivers are kernel modules, so I suppose most of the stories I have would be of the form "I needed to write a device driver for X" :-). Userspace drivers, while certainly in existence, are not very popular in Linux land. Other interesting stories would include: - Instrumenting certain filesystem operations (all modules share the same memory space; it's possible for one module to take a sneak peak at another's internal structures). This was back before dtrace & friends were a (useful) thing. - Real-time processing of instrumentation samples. Doing it in the kernel allowed us to avoid costly back-and-forth between user and kernel memory -- but we only did relatively simple processing, such as scaling, channel muxing/demuxing and the like. If you find yourself thinking about doing kernelside programming because carrying samples to and from userspace is too expensive, you should probably review your design first.

craftyguy 9y ago
> Userspace drivers, while certainly in existence, are not very popular in Linux land Actually, that's not true at all! Just look at Mesa3D and any number of GL proprietary blobs from graphics vendors. Userspace drivers are actually quite common, they just aren't what folks think of immediately when they think about 'drivers'.

aseipp 9y ago

Yes, I wrote one a long time ago that exposed basic counters from the ARMv7 PMU to userspace -- where they are normally not accessible unless used through perf_events. This is because perf wasn't available (I think the shitty old vendor BSP I was using couldn't support it for some reason) and it's relatively costly (perf_event_open is a whole syscall + another syscall to `read` off the event info) -- I just wanted some basic cycle timings for some cryptography code I wrote to see what worked and what didn't. Even though it has some annoying gotchas (such as the fact ARM cores can sleep/frequency scale on demand with no forewarning, meaning cycles aren't always the most precise units of measurement), and is very simple -- this thing ended up being mildly popular. Even though I wrote it years ago, someone from CERN recently emailed me to say they happily used it for work, and someone from Samsung ported it to ARMv8 for me... (I should dust off my boards one of these days and clean it up again, perhaps! People still email me about it.)

mbrumlow 9y ago

I wrote one so I could keep my job... I work at a company that provides backups for both Linux and windows. The entire concept was around block level backups. You could just open up the block device and copy the data directly but it would quickly become out of sync by the time you finished copying it. We did not want to require LVM to be able to utilize snapshots to solve the sync problem. On top of that we had a strong requirement of being able to delete data from the backup. This resulted in me learning how to build a kernel module and then slowly over about 6 months creating a kernel driver that allowed us to take point in time snapshots of any mounted block device with any fs sitting on top. Other requirements also dictated that we keep track of every block that changed on the target block device after the initial backup (or full block scan, after reboot). I wish I could release the source but my employers would not like that :( So at least for me learning how to write kernel modules and digging in to some of the lower stuff has keep me gainfully employed over the years. It is still in use on about 250k to 300k servers today (it fluctuates). The hardest part was not writing the module, but getting others interested in it enough so I don't have to be the sole maintainer. I like working on all parts of the product and don't want to just be the "kernel guy". One other time I wrote a very poorly done network block device driver in about 8 hours. You can find it here https://github.com/mbrumlow/nbs https://github.com/mbrumlow/nbs -- Note I am not proud of this code, it was something I did really quick, wanted it on hand to show to a perspective employer -- I did not get the job, I am also fairly sure they did not even look at the driver, so I don't think the crappy code there affected me.

throwaway613834 9y ago
So you implemented something like Windows's volume snapshots/shadow copies? (Why couldn't you use the existing feature?) EDIT: Thanks for both replies! :)
- aseipp 9y ago
  Pretty much. But Windows snapshots aren't writeable after you take them, which is a major downer and a major necessary feature. (Having this feature lets people do things like "Take snapshot and delete everything except *.lic files, under C:\Users, on every backup" and you can just issue deletes to the shadow volume after it is consistently created, and intercept the new deletions, like normal. Tracking deletions like this lets you more easily know what you should/should not back up -- only back up the allocated blocks in the shadow copy.) So, there was a thing that added this feature to the Windows kernel in the product to make this work. Aside from the Linux stuff, which was totally separate. But if you don't need the writing capability, Shadow Copies are good enough, sure.. (Source: I used to work with the guy who made the above post)

ryandrake 9y ago

This was a super-long time ago, but a kernel driver was needed for video capture back in the 90's, to control devices across I2C, etc. At the time I had a Matrox video card which did analog capture, decoding and TV-in, and so wrote the first few iterations of a kernel driver[1] to get it to work. It since got adopted by a great maintainer. Nowadays I'd guess nobody is using it because the hardware is ancient. 1: http://www.cs.brandeis.edu/~eddie/mga4linux/ http://www.cs.brandeis.edu/~eddie/mga4linux/

invernomut0 9y ago

Some years ago I was working on an embedded platform with two processors connected via ethernet, one handling data and most of the application load, the second one handling some automotive buses (CANbus and ARCnet). I wrote a kernel driver to expose raw data coming from the ARCnet to userspace applications in the most transparent way. So I labeled data coming from the second cpu with an unused ethertype, I changed the ethernet driver to inspect the ether type of the incoming data and send it up inside the network stack as if it was received from a certain fake ARCnet device. The fake ARCnet device was a dummy module that allowed userspace to receive that data via normal sockets. Probably it could be done in a better way but it was quite simple and it worked well.

andrewcchen 9y ago

I modified a kernel module to use the wifi enable/disable button on my hp laptop as a normal button. For some reason the bios sends an acpi event when the button is pressed, instead of just having it as a normal keyboard key, so there is a kernel driver to handle it.

alecdibble 9y ago

A few years ago, I worked at a company that had hardware peripherals conntected to a master CPU card running linux through RS-232. One of the peripherals was a power supply that supplied power to another peripheral. Since they were architected as Slaves, the peripherals couldn't directly talk to each other, they had to get the messages relayed through the master CPU. This whole communication process had to happen within a small time frame, something like 50 ms. A kernel module handled the communication, decoded the packets, and automatically relayed peripheral messages to other peripherals. The kernel module made it easier to achieve stable and real-time operation within the time constraint. When writing the same function in userspace, there was absolutely no guarantee that messages would be sent or received in time.

zb3 9y ago

I wrote one so that I could set hardware breakpoints and use them to extract decrypted data from an app which was heavily protected against debugging. The thing was that the app (running as an unprivleged user) had no way of knowing what I was doing in the kernel (uprobes can be used for this too providing the code doesn't checksum itself). But I guess reverse engineering is not a common use case for writing linux kernel modules though :)

anitil 9y ago
Maybe not common, but super interesting if you want to share more.....
- zb3 9y ago
  I guess that kernel modules are not commonly used for RE because in general there are better tools that support emulation, symbolic execution, dynamic instrumentation and so on (such as qira, vtrace, angr, Intel Pin and more, there are many articles about them). I just wanted something simple and portable to extract/modify registers/memory pointed to by registers/unwind stack when a given instruction was executed, while being as fast and transparent as possible, so I wrote a kernel module... and while it worked(hook mmap via kprobes to get base address + register_user_hw_breakpoints + printk to get output), using it was alot of manual work, and I'm sure tools I listed above could be more efficient. Also since there was no emulation, I had to spoof opening /dev/ urandom, which you don't have to do if you're using qira It'D be great if My small module Could be published As is, but I'd need to strip all addresses/filenames, and also add more proper locking (but yes, I want to publish it anyway).

woodrowbarlow 9y ago

i was working on an embedded device with only RAM and a flash chip for the software blob. i wrote a kernel module that allowed us to keep logs in RAM even across a soft reboot (i.e., if the device crashes or does a software update). it basically reserved a chunk of 1M at the top of RAM (using the same physical address each time). there was a checksum system that allowed it to tell during boot whether the data is still valid or if the bits had decayed. i've also done a couple i2c drivers for temperature sensors or LED controllers. kernel work is fun.

stevekemp 9y ago

I've written simple Linux Security Modules, as learning experiences, but also to help tighten up a bunch of virtual machines. The one that I'm most proud of delegates decisions on whether executables can be run to .. userspace. Which is simultaneously evil and brilliant. i.e. User "foo" tries to run "/tmp/exploit" and the kernel executes "/sbin/can-exec foo /tmp/exploit". If the return code of that is zero then the execution is permitted otherwise it is denied. This gives you ample opportunity to log all executables, perform SHA1-hash checks of contents, or deny executables to staff-members outside business hours. There's a lot of scope for site-specific things, creativity is the limit!

bincat 9y ago
I think it would be much appreciated if this software could be released as open source. How did the user space redirection handle potentially high spikes of execve's? And btw, thanks for all the years of debian-administration.org.

ship_it 9y ago

Not for Linux per se but I've been developing a variant of CAN(+FD) kernel extension for XNU/OSX. Some things are manageable through userland so not everything is done in kernel but extending berkeley sockets is really fun and I'd suggest anyone interested to try it.

lloeki 9y ago

Hardly a problem per se but I was bored on a long train trip (7+ hours) and I just happened to have (on purpose) a Xbox One controller around, so I wired it in, hacked around on userland with libusb, then wrote a kext for the thing to appear as a proper HID device on OS X. Two lessons learned: 1. dealing with tangible things for once is incredibly satisfying. 2. next time, download the fscking docs instead of relying on an intermittent 3G connection. https://github.com/lloeki/xbox_one_controller/ https://github.com/lloeki/xbox_one_controller/

souprock 9y ago

My real-world use case is a hypervisor. That turns out to be some pretty wild code. Intel's VMX documentation is not well-organized or complete, processor models vary, the hardware has bugs, and some things we have to do aren't even officially supported by Intel. We have to mess with page tables. We have a huge block of assembly code. It all has to run perfectly. In case that doesn't make you want to run away screaming, I put a post in the "Who is hiring?" article: https://news.ycombinator.com/item?id=15829518 https://news.ycombinator.com/item?id=15829518