13 ms·

Writing a Simple Linux Kernel Module

429 points by daftpanda 9y ago

ramzyo 9y ago
>> A Linux kernel module is a piece of compiled binary code that is inserted directly into the Linux kernel, running at ring 0, the lowest and least protected ring of execution in the x86–64 processor. The author seems to be implying that rings are implemented at the processor level for x86-64 processors. If I’m interpreting the wording correctly that’s interesting! Coming from the ARM world I’d always thought that rings were an OS construct.
- bravo22 9y ago
  ARM has modes which is somewhat analogous to x86 rings.
- joosters 9y ago
  Most chips have different privilege levels... ARM processors have 'modes' like User, FIQ, IRQ and Supervisor (sort of the equivalent to ring 0). Different modes can have different access controls, e.g. varying access to memory. Edit: See http://www.heyrick.co.uk/armwiki/Processor_modes http://www.heyrick.co.uk/armwiki/Processor_modes
  - ramzyo 9y ago
    Right - out of curiosity do you know if the Linux kernel uses the term "ring", or some other terminology to map to the underlying hardware implementation, be they rings for x86 or modes for ARM? Maybe it's just "privilege level" or something similar?
    - JoshTriplett 9y ago
      The Linux kernel just talks about "kernel mode" and "user mode" (or "userspace"). Those then map to ring 0 and ring 3 on x86, or other mechanisms on other platforms.
      - ramzyo 9y ago
        Cool, thanks!
    - carussell 9y ago
      Gustavo Duartes has a really great blog[1] about kernels and low-level development, with Linux on x86/amd64 being the reference. Here[2] is the one where he goes into CPU rings and kernel- versus user-mode. 1. http://duartes.org/gustavo/blog/archives/ http://duartes.org/gustavo/blog/archives/ 2. http://duartes.org/gustavo/blog/post/cpu-rings-privilege-and-protection/ http://duartes.org/gustavo/blog/post/cpu-rings-privilege-and...
- topspin 9y ago
  The genesis of hardware protected modes in x86 devices go back to 1982 and the 80286.
chatmasta 9y ago
Does anyone have a story or two of a time you’ve created a kernel module to solve a problem? I would be interested in hearing real world use cases.
- samtho 9y ago
  I wrote one that decodes Wiegand[0] off of GPIO. I ended up not using it because epoll-ing the pins is easier to use from a userspace perspective and fast enough to capture the transmission. I loosely based the kernel module off of one that was written for another microcontroller[1]. [0]: https://en.wikipedia.org/wiki/Wiegand_interface https://en.wikipedia.org/wiki/Wiegand_interface [1]: https://github.com/rascalmicro/wiegand-linux https://github.com/rascalmicro/wiegand-linux
  - ianhowson 9y ago
    I did the same but kept it in-kernel to meet Wiegand timings. I had (horrible) another driver which could hang the system for long enough to lose bits. I was in Australia for that work too...
- deleted 9y ago
  [deleted]
- sly010 9y ago
  I didn't end up writing the kernel module, so this might be an anti use case: I was working on an embedded system and needed to have fast I2C access (mostly just wanted very small latency bc it's an instrument). I2C from Linux userspace (using ioctls) adds a lot of overhead. I started looking into kernel modules but after a day of research I found out that you can access hardware registers from userspace using mmap("/dev/mem") which is even faster than a kernel module. Edit:typos
  - zeusk 9y ago
    > mmap("/dev/mem") but that would only work if some kernel module or the kernel itself hasn't mapped that IO space, right?
dottrap 9y ago
"And finally, you’ll need to know at least some C. The C++ runtime is far too large for the kernel, so writing bare metal C is essential." That line reminded me that NetBSD added Lua for writing kernel modules. (https://news.ycombinator.com/item?id=6562611 https://news.ycombinator.com/item?id=6562611) Anybody have any experiences to share from this?
- sigjuice 9y ago
  If you look at the NetBSD kernel source https://github.com/NetBSD/src.git https://github.com/NetBSD/src.git, it doesn't look like there are any real and substantial Lua kernel modules. $ find sys -name '*.lua' sys/modules/examples/luahello/luahello.lua sys/modules/examples/luareadhappy/happy.lua sys/modules/lua/infinite.lua sys/modules/lua/test.lua sys/modules/luasystm/test.lua
- alxlaz 9y ago
  There are plenty of operating systems that use C++ on the kernel side of things. Some things, like exceptions, are frowned upon and rarely used -- if at all -- but there is no shortage of C++ kernel code. Just not in Linux. When it comes to Linux, one could say that most reasons to avoid it are historical, but this does not quite paint the awkward truth -- namely that, for most of the kernel's lifetime (since back in 1991), C++ compilers simply did not have the level of maturity and stability across the breadth of platforms that Linux required. Linus Torvalds' stance on this matter is pretty well-known: http://harmful.cat-v.org/software/c++/linus http://harmful.cat-v.org/software/c++/linus . Today, when x86-64 and ARM are the only two families that you need to care about in the following ten years or so (maybe RISC-V but I rather doubt it), it probably makes sense to look at C++ for operating systems work, but the runtime is certainly heavier than back when Linus was writing about it, too. A modern C++ compiler has a lot of baggage; C++ was huge back in 1998, now it's bloody massive. IMHO, all the reasons why you would want to use C++ (templating support without resorting to strange hacks, useful pointer semantics and so on) are reasonably well-served by cleaner languages with less hefty runtimes, like Rust. What these alternatives do lack is the amazing level of commercial support that C++ has.
  - 9y ago
fmela 9y ago
In the device_read function, len — ; should be len --;
- megous 9y ago
  Probably a CMS thing. printk is probably also discouraged, these days. pr_warn/info/..() or dev_warn/info/err...() should be used. Also for anyone writing kernel code, this is indispensable: http://elixir.free-electrons.com/linux/latest/source http://elixir.free-electrons.com/linux/latest/source
  - uep 9y ago
    It doesn't exactly apply here since the author doesn't seem to be allocating resources dynamically, but another thing that's "newer" and less widely used are the devm_ family of functions. Nowadays, lots of code could be using devm_kmalloc, devm_ioremap, etc which will release the resources automatically when the driver detaches from the device.
- dpedu 9y ago
  Copy-paste trap!
deleted 9y ago
[deleted]
djohnston 9y ago
Kernel newb w/ a question on this. When you register the device with register_chrdev you also have to create the file in userspace with mknod ?
- JoshTriplett 9y ago
  You never need to use mknod yourself anymore. Based on the metadata you provide to the kernel, the kernel will create the device itself in devtmpfs, and even set appropriate baseline permissions (e.g. root-only, world-readable, world-writable). udev will then handle any additional permissions you might need, such as ACLs or groups.
  - anitil 9y ago
    Out of historical interest, why was it necessary before? Was it an intentional separation between userspace/kernel responsibilities?
    - JoshTriplett 9y ago
      A few reasons. Prior to the existence of udev, all devices had to be created with mknod and have permissions set; a script called "MAKEDEV" did that. That didn't handle dynamically created devices, so (skipping "hotplug" and an old attempt called "devfs" which was filled with race conditions) the kernel started sending events to userspace for dynamic devices, and udev created those. Then, it made sense to "coldplug" all the existing devices and have udev create those, to use the same unified path for everything, and use udev to set groups and permissions. But that did take some time at boot time, and it meant that you needed either udev or a manually created /dev for tiny embedded systems. devtmpfs handled that use case, automatically creating devices from the kernel. But then, if the kernel knows how to create devices, why not have it do so all the time? So, udev jettisoned the code to create devices itself, and started requiring devtmpfs.
spapas82 9y ago
Replying after checking my notes from the University (NTUA): Before around 14 years (~ 2004) one of the excersies we had in Operatin Systems Lab was to actually implement a char device driver for Linux called ... lunix as a kernel module. The actual device was just a buch of bytes in memory. These times were so happy - implementing a device driver seemed like owning the world for a student! If anybody wants I'd be happy to put the source code in github - mainly for historical reasons because I'm not sure that the code will be still working today.
- magpi3 9y ago
  There is a similar exercise in "Linux Device Drivers." The first driver they ask you to write is a char device that runs in memory called "scull." Here is a link to the free book (warning it's a pdf): http://free-electrons.com/doc/books/ldd3.pdf http://free-electrons.com/doc/books/ldd3.pdf And here is a link to a github project with all exercises updated for the most recent kernel: https://github.com/martinezjavier/ldd3 https://github.com/martinezjavier/ldd3
  - spapas82 9y ago
    Ha ha "scull" ringed a bell and after taking a look at the cover of the book you mention (it's not in the pdf you provide but can be found if you search for images for Linux Device Drivers oreilly) and see the horse ... I totally remembered the book! It was one of the refereces I used while implementing the module and it was a really good and comprehensive book - totally agree with the recommendation :)
- abstractbeliefs 9y ago
  Please do. One of the issues for tfa is how little it actually covers. While it quite fairly says module development is more like writing an API than an application, it then proceeds to entirely ignore any hooks other than init and exit!
  - spapas82 9y ago
    I'll try adding it tomorrow to github (it's a little late here right now). In the meantime, I'd like to also recommend the book "Linux Device Drivers" that fellow commenter magpi3 mentioned.
qsdf38100 9y ago
Kernel code does not operate at "incredible speed". This guy doesn't know what he is talking about.
- protomikron 9y ago
  That's actually true. There might be some more initialization overhead if you exec a process (the libc stuff), but non-kernel (i.e. user space) code is not slower per se.
- madez 9y ago
  You are correct about the speed, but the short response combined with the unjustifiedly overly general attack of the authors knowledge is unnecessary.
  - qsdf38100 9y ago
    This is somewhat unnecessary, but this article is spreading false claim about kernel mode "incredible speed" and this should be denounced.
- Laaas 9y ago
  https://stackoverflow.com/questions/36992702/what-is-the-performance-impact-of-virtual-memory-relative-to-direct-mapped-memor https://stackoverflow.com/questions/36992702/what-is-the-per... At least virtual memory impacts performance, which is one thing user space code uses
  - qsdf38100 9y ago
    Kernel code also uses virtual memory. There is no such thing as using "physical memory" unless you are running in real mode. CPU is switched to protected mode during boot.
- AdamJacobMuller 9y ago
  It depends on what you're doing. Eliminating context switches can be hugely valuable. I agree the author should have been more nuanced, but, he's not flat-out wrong.
vagab0nd 9y ago
netcat released their album as a linux kernel module [0]. Browsing their code might be a more fun way to learn how to write a kernel module with a real "use case". 0: https://github.com/usrbinnc/netcat-cpi-kernel-module https://github.com/usrbinnc/netcat-cpi-kernel-module

429 points by daftpanda 9y ago

>> A Linux kernel module is a piece of compiled binary code that is inserted directly into the Linux kernel, running at ring 0, the lowest and least protected ring of execution in the x86–64 processor. The author seems to be implying that rings are implemented at the processor level for x86-64 processors. If I’m interpreting the wording correctly that’s interesting! Coming from the ARM world I’d always thought that rings were an OS construct.

bravo22 9y ago
ARM has modes which is somewhat analogous to x86 rings.
joosters 9y ago
Most chips have different privilege levels... ARM processors have 'modes' like User, FIQ, IRQ and Supervisor (sort of the equivalent to ring 0). Different modes can have different access controls, e.g. varying access to memory. Edit: See http://www.heyrick.co.uk/armwiki/Processor_modes http://www.heyrick.co.uk/armwiki/Processor_modes
- ramzyo 9y ago
  Right - out of curiosity do you know if the Linux kernel uses the term "ring", or some other terminology to map to the underlying hardware implementation, be they rings for x86 or modes for ARM? Maybe it's just "privilege level" or something similar?
  - JoshTriplett 9y ago
    The Linux kernel just talks about "kernel mode" and "user mode" (or "userspace"). Those then map to ring 0 and ring 3 on x86, or other mechanisms on other platforms.
    - ramzyo 9y ago
      Cool, thanks!
  - carussell 9y ago
    Gustavo Duartes has a really great blog[1] about kernels and low-level development, with Linux on x86/amd64 being the reference. Here[2] is the one where he goes into CPU rings and kernel- versus user-mode. 1. http://duartes.org/gustavo/blog/archives/ http://duartes.org/gustavo/blog/archives/ 2. http://duartes.org/gustavo/blog/post/cpu-rings-privilege-and-protection/ http://duartes.org/gustavo/blog/post/cpu-rings-privilege-and...
topspin 9y ago
The genesis of hardware protected modes in x86 devices go back to 1982 and the 80286.

chatmasta 9y ago

Does anyone have a story or two of a time you’ve created a kernel module to solve a problem? I would be interested in hearing real world use cases.

samtho 9y ago
I wrote one that decodes Wiegand[0] off of GPIO. I ended up not using it because epoll-ing the pins is easier to use from a userspace perspective and fast enough to capture the transmission. I loosely based the kernel module off of one that was written for another microcontroller[1]. [0]: https://en.wikipedia.org/wiki/Wiegand_interface https://en.wikipedia.org/wiki/Wiegand_interface [1]: https://github.com/rascalmicro/wiegand-linux https://github.com/rascalmicro/wiegand-linux
- ianhowson 9y ago
  I did the same but kept it in-kernel to meet Wiegand timings. I had (horrible) another driver which could hang the system for long enough to lose bits. I was in Australia for that work too...
deleted 9y ago
[deleted]
sly010 9y ago
I didn't end up writing the kernel module, so this might be an anti use case: I was working on an embedded system and needed to have fast I2C access (mostly just wanted very small latency bc it's an instrument). I2C from Linux userspace (using ioctls) adds a lot of overhead. I started looking into kernel modules but after a day of research I found out that you can access hardware registers from userspace using mmap("/dev/mem") which is even faster than a kernel module. Edit:typos
- zeusk 9y ago
  > mmap("/dev/mem") but that would only work if some kernel module or the kernel itself hasn't mapped that IO space, right?

dottrap 9y ago

"And finally, you’ll need to know at least some C. The C++ runtime is far too large for the kernel, so writing bare metal C is essential." That line reminded me that NetBSD added Lua for writing kernel modules. (https://news.ycombinator.com/item?id=6562611 https://news.ycombinator.com/item?id=6562611) Anybody have any experiences to share from this?

sigjuice 9y ago
If you look at the NetBSD kernel source https://github.com/NetBSD/src.git https://github.com/NetBSD/src.git, it doesn't look like there are any real and substantial Lua kernel modules. $ find sys -name '*.lua' sys/modules/examples/luahello/luahello.lua sys/modules/examples/luareadhappy/happy.lua sys/modules/lua/infinite.lua sys/modules/lua/test.lua sys/modules/luasystm/test.lua
alxlaz 9y ago
There are plenty of operating systems that use C++ on the kernel side of things. Some things, like exceptions, are frowned upon and rarely used -- if at all -- but there is no shortage of C++ kernel code. Just not in Linux. When it comes to Linux, one could say that most reasons to avoid it are historical, but this does not quite paint the awkward truth -- namely that, for most of the kernel's lifetime (since back in 1991), C++ compilers simply did not have the level of maturity and stability across the breadth of platforms that Linux required. Linus Torvalds' stance on this matter is pretty well-known: http://harmful.cat-v.org/software/c++/linus http://harmful.cat-v.org/software/c++/linus . Today, when x86-64 and ARM are the only two families that you need to care about in the following ten years or so (maybe RISC-V but I rather doubt it), it probably makes sense to look at C++ for operating systems work, but the runtime is certainly heavier than back when Linus was writing about it, too. A modern C++ compiler has a lot of baggage; C++ was huge back in 1998, now it's bloody massive. IMHO, all the reasons why you would want to use C++ (templating support without resorting to strange hacks, useful pointer semantics and so on) are reasonably well-served by cleaner languages with less hefty runtimes, like Rust. What these alternatives do lack is the amazing level of commercial support that C++ has.
- 9y ago

fmela 9y ago

In the device_read function, len — ; should be len --;

megous 9y ago
Probably a CMS thing. printk is probably also discouraged, these days. pr_warn/info/..() or dev_warn/info/err...() should be used. Also for anyone writing kernel code, this is indispensable: http://elixir.free-electrons.com/linux/latest/source http://elixir.free-electrons.com/linux/latest/source
- uep 9y ago
  It doesn't exactly apply here since the author doesn't seem to be allocating resources dynamically, but another thing that's "newer" and less widely used are the devm_ family of functions. Nowadays, lots of code could be using devm_kmalloc, devm_ioremap, etc which will release the resources automatically when the driver detaches from the device.
dpedu 9y ago
Copy-paste trap!

deleted 9y ago

[deleted]

djohnston 9y ago

Kernel newb w/ a question on this. When you register the device with register_chrdev you also have to create the file in userspace with mknod ?

JoshTriplett 9y ago
You never need to use mknod yourself anymore. Based on the metadata you provide to the kernel, the kernel will create the device itself in devtmpfs, and even set appropriate baseline permissions (e.g. root-only, world-readable, world-writable). udev will then handle any additional permissions you might need, such as ACLs or groups.
- anitil 9y ago
  Out of historical interest, why was it necessary before? Was it an intentional separation between userspace/kernel responsibilities?
  - JoshTriplett 9y ago
    A few reasons. Prior to the existence of udev, all devices had to be created with mknod and have permissions set; a script called "MAKEDEV" did that. That didn't handle dynamically created devices, so (skipping "hotplug" and an old attempt called "devfs" which was filled with race conditions) the kernel started sending events to userspace for dynamic devices, and udev created those. Then, it made sense to "coldplug" all the existing devices and have udev create those, to use the same unified path for everything, and use udev to set groups and permissions. But that did take some time at boot time, and it meant that you needed either udev or a manually created /dev for tiny embedded systems. devtmpfs handled that use case, automatically creating devices from the kernel. But then, if the kernel knows how to create devices, why not have it do so all the time? So, udev jettisoned the code to create devices itself, and started requiring devtmpfs.

spapas82 9y ago

Replying after checking my notes from the University (NTUA): Before around 14 years (~ 2004) one of the excersies we had in Operatin Systems Lab was to actually implement a char device driver for Linux called ... lunix as a kernel module. The actual device was just a buch of bytes in memory. These times were so happy - implementing a device driver seemed like owning the world for a student! If anybody wants I'd be happy to put the source code in github - mainly for historical reasons because I'm not sure that the code will be still working today.

magpi3 9y ago
There is a similar exercise in "Linux Device Drivers." The first driver they ask you to write is a char device that runs in memory called "scull." Here is a link to the free book (warning it's a pdf): http://free-electrons.com/doc/books/ldd3.pdf http://free-electrons.com/doc/books/ldd3.pdf And here is a link to a github project with all exercises updated for the most recent kernel: https://github.com/martinezjavier/ldd3 https://github.com/martinezjavier/ldd3
- spapas82 9y ago
  Ha ha "scull" ringed a bell and after taking a look at the cover of the book you mention (it's not in the pdf you provide but can be found if you search for images for Linux Device Drivers oreilly) and see the horse ... I totally remembered the book! It was one of the refereces I used while implementing the module and it was a really good and comprehensive book - totally agree with the recommendation :)
abstractbeliefs 9y ago
Please do. One of the issues for tfa is how little it actually covers. While it quite fairly says module development is more like writing an API than an application, it then proceeds to entirely ignore any hooks other than init and exit!
- spapas82 9y ago
  I'll try adding it tomorrow to github (it's a little late here right now). In the meantime, I'd like to also recommend the book "Linux Device Drivers" that fellow commenter magpi3 mentioned.

qsdf38100 9y ago

Kernel code does not operate at "incredible speed". This guy doesn't know what he is talking about.

protomikron 9y ago
That's actually true. There might be some more initialization overhead if you exec a process (the libc stuff), but non-kernel (i.e. user space) code is not slower per se.
madez 9y ago
You are correct about the speed, but the short response combined with the unjustifiedly overly general attack of the authors knowledge is unnecessary.
- qsdf38100 9y ago
  This is somewhat unnecessary, but this article is spreading false claim about kernel mode "incredible speed" and this should be denounced.
Laaas 9y ago
https://stackoverflow.com/questions/36992702/what-is-the-performance-impact-of-virtual-memory-relative-to-direct-mapped-memor https://stackoverflow.com/questions/36992702/what-is-the-per... At least virtual memory impacts performance, which is one thing user space code uses
- qsdf38100 9y ago
  Kernel code also uses virtual memory. There is no such thing as using "physical memory" unless you are running in real mode. CPU is switched to protected mode during boot.
AdamJacobMuller 9y ago
It depends on what you're doing. Eliminating context switches can be hugely valuable. I agree the author should have been more nuanced, but, he's not flat-out wrong.

vagab0nd 9y ago

netcat released their album as a linux kernel module [0]. Browsing their code might be a more fun way to learn how to write a kernel module with a real "use case". 0: https://github.com/usrbinnc/netcat-cpi-kernel-module https://github.com/usrbinnc/netcat-cpi-kernel-module