11 ms·

Write your first Linux kernel module

287 points by benev 12y ago

pdevr 12y ago
This is a good starting point for anyone interested in the Eudyptula Challenge [1]. [1] http://eudyptula-challenge.org/ http://eudyptula-challenge.org/
- sp332 12y ago
  The idea for this came to us after a long night of drinking in which it was determined that if the Linux kernel was to survive, it would need new programmers to fix all of the bugs that were recently added after a long night of drinking. Sound thinking.
thegeomaster 12y ago
Maybe it's worth emphasizing that writing kernel code is a very responsible task, not to be taken lightly. Hackers coming from a userspace background tend to think it's just "userspace with a few quirks" which is grossly wrong: your code has to be completely free of bugs, as a dangling pointer, for instance, won't just send you a friendly SIGSEGV and terminate your ass, it will crash the whole system, leading to consequences for everyone who relies on its stability. Just imagine introducing a bug which crashes a Linux-based life support system. You could be responsible for endangering human lives. (I ageee that Linux isn't a sane choice here, but just for the sake of the argument.) Also, standards to get your pull requests merged into mainline Linux are insanely high, but even given that, you can never rely solely on maintainers' code review to catch your mistakes. Take kernel hacking seriously.
- akerl_ 12y ago
  That's putting it a bit harsh. Yes, if you plan on writing kernel code for your fleet of production systems, or for submission to mainline, you need to be very careful. That said, starting out, you work on a development system, you write a bunch of code, it'll have a lot of bugs, and you'll learn from them and from the process of fixing them. Saying "You could be responsible for endangering human lives" is quite overdramatic. If people only pushed code to the kernel that they knew 100% was "completely free of bugs", we'd not have a Linux kernel at all. I've had to roll out enough new kernels due to security issues just this year to know that.
  - thegeomaster 12y ago
    Yeah, on second thought, it probably sounds over-dramatic and all, but it would probably impose some valuable discipline for people who want to hack at the Linux kernel. (I've been programming for quite some time, but have never thought I am ready for going beyond userspace. Maybe I have it a little wrong, after all.)
    - pabb 12y ago
      I know you meant well, but to give such a stern warning and then admit that you haven't done any kernel-related work at all is frustrating. I've just been getting involved in kernel development for the past month, and it's pretty clear that the go-to taskss for novices is primarily writing device drivers or fixing bugs. I'm not sure what fraction of kernel developers get to work on the kernel core (CPU scheduling, boot/initialization, file systems, memory management), but I'm going to guess it's relatively small, or at least isolated compared to those working on drivers and other modules. Further, if the maintainers were so cautious as to want to limit contributions to the kernel, you wouldn't see initiatives like the aforementioned (and awesome) Eudyptula Challenge, or Greg KH's "Linux Kernel In a Nutshell", which he chose to give away for free in order to garner more interest in kernel dev. The state of the project right now is clearly weighted towards getting new involvement as opposed to discouraging contribution because of extremely rare and unlikely cases like the one you mentioned.
      - thegeomaster 12y ago
        I don't think you have to do kernel work to warn people they should be cautious when submitting to mainline. As a matter of fact, I've read the book you mentioned, alongside some others such as Understanding the Linux Kernel by Bovet and Cesati, and I've brushed against low-level kernel features such as memory-mapped I/O or epoll, so I'm at least aware of what might go wrong if you decide to write a kernel module just for fun and then issue a pull request against the original source tree. I don't think contribution should to be discouraged, but I do think that people who have no sense of responsibility should be kept away from the kernel source. Kernel development is not supposed to be for everyone, it's hard. Great expertise and technical mastery is needed to write good, quality and responsible code that is suitable for inclusion into a kernel that today powers so many different devices. I will admit, again, that my warning was probably too harsh, but I still stand firmly by the viewpoint that "casual" or careless kernel contribution should be a no-no. It's one thing to poke around on your dev system to find out how stuff works, but entirely another to write an official patch to a system that millions of people around the world rely on. EDIT: And when I say that "kernel development is hard", I mean that it's harder in the general case than writing most userspace code. It requires you to think of stuff you don't usually worry about (think of the all the race conditions that might occur) and let go of all the abstractions provided to you by the kernel, it's harder to debug than userspace programs, and you have to support a wide variety of hardware platforms, each with its own quirks. In fact, I haven't done any kernel development just because I was intimidated by how hard it is. I've tried writing a kernel from scratch (OSDev wiki was a great help) but it soon came to be a buggy and unmaintainable set of kludges that was barely doing any real work. EDIT #2: Why I can't reply to replies to this comment? Is there a conversation depth limit on HN?
- danford 12y ago
  >You could be responsible for endangering human lives. I'm pretty sure what ever devices people depend on to keep them alive would be running a modified version of the Linux kernel with anything unnecessary stripped out and probably a lot of custom stuff. They wouldn't just download it from kernel.org and throw it on there. I would hope not at least.
  - thegeomaster 12y ago
    I agree. But I think kernel contributors should do their absolute best to ensure the code is as bug-free as possible. There are irresponsible sysadmins who'd use a Linux kernel where there is no place for a Linux kernel to be, and just because of that alone it would be a good idea to be paranoid when pushing to Linux mainline.
  - cnvogel 12y ago
    I wholeheartly agree with ❝devices people depend on to keep them alive would be running a -- version of the Linux kernel with anything unnecessary stripped out❞. But the part with a ❝modified version (...) and probably a lot of custom stuff❞ made be shudder. The Linux kernel is a very well maintained piece of software, and whenever, in the embedded world, I've seen modification or larger forks of the stock/mainline kernel, they always were of worse quality and had more bugs than what I (or everyone else) has gotten used to when running mainline code (as is customary on your normal x86 PCs). That's because code-review actually works in Linux, and while it makes it sometimes very hard to get new features in, the ones that make it are usually well vettet.
- userbinator 12y ago
  > it will crash the whole system As anyone who's programmed in "completely open" environments like C64, ZX, DOS, etc. will be accustomed to, reaching for the reset button becomes almost second-nature after the first few times you mess up, and then you start developing a more thoughtful approach than just "compile and run, see if it works" when you realise that waiting for the reboot takes a significant amount of time. Maybe this is some sort of counterargument to the trend towards "early and often" testing cycles... But I really think everyone should have attempted writing code for an environment with no memory protection and where bugs mean a reboot. It probably makes you a better programmer overall.
  - thegeomaster 12y ago
    I agree with you completely. We shouldn't live in the past and fail to embrace the present, but I think everyone would benefit from knowing what the past was like. Maybe it will make you develop quite a few healthy habits, after all :)
- cjbprime 12y ago
  Hm. I'm a Linux kernel subsystem maintainer, and I think this stereotype ("only extremely accomplished programmers should work on the kernel") probably hurts more than it helps. I'm not sure I've ever met someone who wanted to contribute to mainline and didn't understand the responsibility involved. What I think we see more of is excellent programmers who think they aren't good enough for the kernel, with the result that the average age for core kernel developers appears to be increasing all the time. It's not a great situation. I'm not saying my opinion is representative of all kernel maintainers, but I'm personally in favor of losing the "you must be this tall to work on the kernel" rhetoric.
  - _wmd 12y ago
    It's probably more to do with personality type than intelligence or anything else. It takes an extremely thick skin to handle the regular patch review slating dished out by most of the core team.. I simply don't want to deal with that shit for a hobby, and it'd take an impressive salary to make me deal with it professionally. There's being opinionated, then there's being an ass, and I think there's far too much of the latter on the kernel lists. If you look at the modern nerd, compared to the generation that birthed the kernel, they're more socially aware, fashionable animals that are more interested on working on sexy, visible things. Why spend your day dealing with bad attitudes and timing bugs when you can pick up $200k+ writing Ruby apps that enable sharing 5 second video clips? I doubt the kernel will ever be starved for new talent, just that the ratio of people willing to work on it or deal with the social norms it entails is shrinking all the time.
    - DigitalJack 12y ago
      I think a better quote would be: "There's being opinionated, there's being an ass, then there's Linus Torvalds."
      - Gracana 12y ago
        Don't worry though, he's not writing a three-paragraph rant calling you stupid, he's calling your idea stupid! Why do people take things so personally!? ...big ol' "/s" Maybe the lkml is not always so bad, but I'd rather not deal with it and so far I have been content playing with my own projects. I feel kind of silly complaining about it when so many people manage it just fine, but then I see another explosion of aggression and drama and think "how are people okay with this?" I can't imagine having to work in an environment like that.
- cnvogel 12y ago
  Not everyone writes code for a critical Linux-based life support system. Very often it's just people who want to have a 7-segment LED display show the number of unread emails, or get a soundcard running, something along these lines. And for them it's completely ok that there's a race-condition that might crash their machine if the kernel-module is unloaded just at the instant where the USB plug is removed -- that's something to learn, once the general grasp of how kernel-code with it's myriad of callbacks and hierarchies of subsystems has been reached. Most crashes also will be catched by the Kernel and will leave you with nothing more than a [42568.139488] BUG: unable to handle kernel NULL pointer dereference at 0000000000000004 [42568.139805] IP: [<ffffffffa003e008>] init_module+0x8/0x20 [my_dummy_module] [42568.140308] PGD 4450f067 PUD 4450e067 PMD 0 [42568.140882] Oops: 0002 [#1] PREEMPT SMP (...) in your dmesg, and maybe a dead "insmod" process in your process table. Make a note of the call trace, sync, reboot (maybe hard, using the magic sysrq key) and you are off for the next round of debugging. Try it, it's fun. Also there's kgdb which you can use when your bugs aren't that obvious. I never had serious data loss (maybe the freshly compiled .o files will have 0-size, if I was in too much of a hurry to insmod ./my_module.ko. And even if, it will teach you to have an easily restorable development machine. (I triggered the oops with...) int init_module(void) { int *ptr = 0x0000000000000004ULL; *ptr = 17; return 0; }
  - mschuster91 12y ago
    > Very often it's just people who want to have a 7-segment LED display show the number of unread emails, or get a soundcard running, something along these lines. 7-seg LEDs are best controlled by a microcontroller attached via USB or a USB-RS232 interface, as next to no one ships a parallel printer port. Soundcards these days usually all already have kernel drivers, I haven't heard of any modern chip without. And USB stuff is best developed in userspace with libusb, it's way easier...
Nursie 12y ago
Cool, definitely relevant to my interests. I've done various ARM/orion5x platform porting related stuff before but never a module...
chrisBob 12y ago
Do most kernel programmers start out working on their regular daily-use machine? My only experience is with an embedded linux environment, and I would probably recommend starting with something like a RPi so that your environment is easy to restore if you mess something up. I also recommend "Linux Device Drivers" from O'Reilly. http://lwn.net/Kernel/LDD3/ http://lwn.net/Kernel/LDD3/
- sophacles 12y ago
  Whenever I do kernel stuff, it happens in a VM, limited to a subset of the cores available on the machine. That way when I inevitably bork something, I can just forcibly stop it. Another trick I've used is making as much of my core logic as possible happen in functions that are called from the kernel hooks, rather than in the kernel hooks. Then you can test a ton of it in userland to handle basic logic and flow control bugs, finally porting the code to the module. To help with this, it sometimes is actually faster to copy some of the kernel routines you use to a different .c file and change things like kprintf and kmalloc for the userland versions. It isn't going to be a perfect port, but it helps get some of the complex stuff out of the way where you have your familiar userland tools. All that said - I've only dabbled in the kernel a little, there are probably better ways that someone can chime in on.
  - solarexplorer 12y ago
    Keeping the complex stuff out of the kernel is good advice. (and not just during development)
    - sophacles 12y ago
      True! Unfortunately it's sometimes unavoidable. The specific case I was remembering involved writing a qdisc... it was actually more complex to move that out of the kernel via nf_queue than to just do it in-kernel. (At least some parts of it.)
  - sitkack 12y ago
    Idea: sysctl/ioctl rest interface so one can drive the new kernel module from the host vm.
jvns 12y ago
This is so great! Writing toy kernel modules on your own is a super fun way to get into kernel programming. There's no danger of anything going wrong, especially if you develop inside a VM. My partner wrote a kernel module for a talk I gave that rickrolls your file system -- it overrides the 'open' system call so that any time you open a '.mp3' file, it plays Rick Astley instead. (this requires you to have Rick Astley already on your computer =)) The source code for that (and a few other toy kernel modules) is at https://github.com/jvns/kernel-module-fun https://github.com/jvns/kernel-module-fun The rickroll module (by Kamal Marhubi) is the most well commented and fun: https://github.com/jvns/kernel-module-fun/blob/master/rickroll.c https://github.com/jvns/kernel-module-fun/blob/master/rickro... I wrote a blog post with more about this kernel-programming-for-fun philosophy: http://jvns.ca/blog/2014/01/04/4-paths-to-being-a-kernel-hacker/ http://jvns.ca/blog/2014/01/04/4-paths-to-being-a-kernel-hac... Of course, writing code for the mainline kernel is a responsible task, but writing kernel code on your own machine that nobody will ever run in production is a great way to learn, and I think it's made me a better programmer. A few concrete ways I think it's made me a better programmer: - I now understand the interface between kernel space and userspace much better (I use strace all the time to debug my regular userspace code!) - realizing that the Linux kernel is just written in C and it's something I can read (and understand! and change!) was a big deal for me - understanding the basics of the underlying operating system helps me reason about my programs better - I now feel like if I ran into an actual kernel bug or performance problem, it's something I could conceivably attempt to fix and understand, instead of hiding under the bed and saying OH NO THE KERNEL IT IS TOO HARD
deutronium 12y ago
This is my first kernel module, which 'fakes' your system uptime: http://www.anfractuosity.com/projects/uptime/ http://www.anfractuosity.com/projects/uptime/ Personally I think people should be encouraged to hack around with the kernel, rather than be afraid of it, how else can we learn?
benjojo12 12y ago
I had great fun writing a silly toy kernel module a few weeks back, ( https://github.com/benjojo/dev_markov https://github.com/benjojo/dev_markov ) Would totally recommend anyone curious giving it a go.
Aardwolf 12y ago
VirtualBox uses a kernel module, and that is annoying because the linux kernel API changes with every update, which means its kernel module needs to be recompiled. Since I update my Linux quite often, that basically means: Everytime some friends call me to play a certain game online with each other, they need to be waiting a bit longer for me because I have yet again to recompile VirtualBox's kernel modules. Is there a way to convert something that requires kernel modules, to run fully in userspace? I'd be happy to lose some efficiency for this. Just a 100% reliable game to run my game would be perfect. Are there things you can do only in kernel modules and not in userspace, and does VB need this? In theory userspace can do everything you need, right? Graphics, networking, audio, ...
- aquayellow 12y ago
  Virtualbox is an out-of-tree module and hence needs to be compiled. The base package is GPLv2 and is compatible with the Linux kernel license, but I don't know if Virtualbox developers ever tried to "upstream" their module. This is probably the easiest way to solve your problem. Most Virtualization Software these days use virtualization hardware extensions (Intel VMX/AMD SVM) and so the "hypervisor code" needs to be able to execute privileged instructions, eg - you can't run VMON from ring 3/userspace.
  - justincormack 12y ago
    Apparently the virtualbox code is terrible, hence the lack of upstreaming...
- josteink 12y ago
  DKMS might help if you have access to the source. https://en.m.wikipedia.org/wiki/Dynamic_Kernel_Module_Support https://en.m.wikipedia.org/wiki/Dynamic_Kernel_Module_Suppor...
- jimmaswell 12y ago
  I'd just update the kernel less if i were you. There's not really much of a point updating the kernel all the time for normal use is there?
Myrmornis 12y ago
Really great article.
jhiesey 12y ago
I might be missing something, but the final version still looks susceptible to deadlock to me. Suppose the buffer is empty. Process A, which is reading, gets through the mutex_unlock() call and into wait_event_interruptible(), which seems to first internally check the condition. It is then switched out before it internally waits. Process B then runs all the way through, acquiring the mutex, doing its work, and runs wake_up_interruptible(), which won't do anything, because process A hasn't started waiting. Process B then releases the mutex and gets switched out. Process A then gets switched back in, and then immediately starts waiting, with no more events to wake it up. Why won't it just wait forever (deadlock)? Am I missing something? Do all processes occasionally get woken up without an explicit wake up?

287 points by benev 12y ago

pdevr 12y ago

This is a good starting point for anyone interested in the Eudyptula Challenge [1]. [1] http://eudyptula-challenge.org/ http://eudyptula-challenge.org/

sp332 12y ago
The idea for this came to us after a long night of drinking in which it was determined that if the Linux kernel was to survive, it would need new programmers to fix all of the bugs that were recently added after a long night of drinking. Sound thinking.

thegeomaster 12y ago

Maybe it's worth emphasizing that writing kernel code is a very responsible task, not to be taken lightly. Hackers coming from a userspace background tend to think it's just "userspace with a few quirks" which is grossly wrong: your code has to be completely free of bugs, as a dangling pointer, for instance, won't just send you a friendly SIGSEGV and terminate your ass, it will crash the whole system, leading to consequences for everyone who relies on its stability. Just imagine introducing a bug which crashes a Linux-based life support system. You could be responsible for endangering human lives. (I ageee that Linux isn't a sane choice here, but just for the sake of the argument.) Also, standards to get your pull requests merged into mainline Linux are insanely high, but even given that, you can never rely solely on maintainers' code review to catch your mistakes. Take kernel hacking seriously.

akerl_ 12y ago
That's putting it a bit harsh. Yes, if you plan on writing kernel code for your fleet of production systems, or for submission to mainline, you need to be very careful. That said, starting out, you work on a development system, you write a bunch of code, it'll have a lot of bugs, and you'll learn from them and from the process of fixing them. Saying "You could be responsible for endangering human lives" is quite overdramatic. If people only pushed code to the kernel that they knew 100% was "completely free of bugs", we'd not have a Linux kernel at all. I've had to roll out enough new kernels due to security issues just this year to know that.
- thegeomaster 12y ago
  Yeah, on second thought, it probably sounds over-dramatic and all, but it would probably impose some valuable discipline for people who want to hack at the Linux kernel. (I've been programming for quite some time, but have never thought I am ready for going beyond userspace. Maybe I have it a little wrong, after all.)
  - pabb 12y ago
    I know you meant well, but to give such a stern warning and then admit that you haven't done any kernel-related work at all is frustrating. I've just been getting involved in kernel development for the past month, and it's pretty clear that the go-to taskss for novices is primarily writing device drivers or fixing bugs. I'm not sure what fraction of kernel developers get to work on the kernel core (CPU scheduling, boot/initialization, file systems, memory management), but I'm going to guess it's relatively small, or at least isolated compared to those working on drivers and other modules. Further, if the maintainers were so cautious as to want to limit contributions to the kernel, you wouldn't see initiatives like the aforementioned (and awesome) Eudyptula Challenge, or Greg KH's "Linux Kernel In a Nutshell", which he chose to give away for free in order to garner more interest in kernel dev. The state of the project right now is clearly weighted towards getting new involvement as opposed to discouraging contribution because of extremely rare and unlikely cases like the one you mentioned.
    - thegeomaster 12y ago
      I don't think you have to do kernel work to warn people they should be cautious when submitting to mainline. As a matter of fact, I've read the book you mentioned, alongside some others such as Understanding the Linux Kernel by Bovet and Cesati, and I've brushed against low-level kernel features such as memory-mapped I/O or epoll, so I'm at least aware of what might go wrong if you decide to write a kernel module just for fun and then issue a pull request against the original source tree. I don't think contribution should to be discouraged, but I do think that people who have no sense of responsibility should be kept away from the kernel source. Kernel development is not supposed to be for everyone, it's hard. Great expertise and technical mastery is needed to write good, quality and responsible code that is suitable for inclusion into a kernel that today powers so many different devices. I will admit, again, that my warning was probably too harsh, but I still stand firmly by the viewpoint that "casual" or careless kernel contribution should be a no-no. It's one thing to poke around on your dev system to find out how stuff works, but entirely another to write an official patch to a system that millions of people around the world rely on. EDIT: And when I say that "kernel development is hard", I mean that it's harder in the general case than writing most userspace code. It requires you to think of stuff you don't usually worry about (think of the all the race conditions that might occur) and let go of all the abstractions provided to you by the kernel, it's harder to debug than userspace programs, and you have to support a wide variety of hardware platforms, each with its own quirks. In fact, I haven't done any kernel development just because I was intimidated by how hard it is. I've tried writing a kernel from scratch (OSDev wiki was a great help) but it soon came to be a buggy and unmaintainable set of kludges that was barely doing any real work. EDIT #2: Why I can't reply to replies to this comment? Is there a conversation depth limit on HN?
danford 12y ago
>You could be responsible for endangering human lives. I'm pretty sure what ever devices people depend on to keep them alive would be running a modified version of the Linux kernel with anything unnecessary stripped out and probably a lot of custom stuff. They wouldn't just download it from kernel.org and throw it on there. I would hope not at least.
- thegeomaster 12y ago
  I agree. But I think kernel contributors should do their absolute best to ensure the code is as bug-free as possible. There are irresponsible sysadmins who'd use a Linux kernel where there is no place for a Linux kernel to be, and just because of that alone it would be a good idea to be paranoid when pushing to Linux mainline.
- cnvogel 12y ago
  I wholeheartly agree with ❝devices people depend on to keep them alive would be running a -- version of the Linux kernel with anything unnecessary stripped out❞. But the part with a ❝modified version (...) and probably a lot of custom stuff❞ made be shudder. The Linux kernel is a very well maintained piece of software, and whenever, in the embedded world, I've seen modification or larger forks of the stock/mainline kernel, they always were of worse quality and had more bugs than what I (or everyone else) has gotten used to when running mainline code (as is customary on your normal x86 PCs). That's because code-review actually works in Linux, and while it makes it sometimes very hard to get new features in, the ones that make it are usually well vettet.
userbinator 12y ago
> it will crash the whole system As anyone who's programmed in "completely open" environments like C64, ZX, DOS, etc. will be accustomed to, reaching for the reset button becomes almost second-nature after the first few times you mess up, and then you start developing a more thoughtful approach than just "compile and run, see if it works" when you realise that waiting for the reboot takes a significant amount of time. Maybe this is some sort of counterargument to the trend towards "early and often" testing cycles... But I really think everyone should have attempted writing code for an environment with no memory protection and where bugs mean a reboot. It probably makes you a better programmer overall.
- thegeomaster 12y ago
  I agree with you completely. We shouldn't live in the past and fail to embrace the present, but I think everyone would benefit from knowing what the past was like. Maybe it will make you develop quite a few healthy habits, after all :)
cjbprime 12y ago
Hm. I'm a Linux kernel subsystem maintainer, and I think this stereotype ("only extremely accomplished programmers should work on the kernel") probably hurts more than it helps. I'm not sure I've ever met someone who wanted to contribute to mainline and didn't understand the responsibility involved. What I think we see more of is excellent programmers who think they aren't good enough for the kernel, with the result that the average age for core kernel developers appears to be increasing all the time. It's not a great situation. I'm not saying my opinion is representative of all kernel maintainers, but I'm personally in favor of losing the "you must be this tall to work on the kernel" rhetoric.
- _wmd 12y ago
  It's probably more to do with personality type than intelligence or anything else. It takes an extremely thick skin to handle the regular patch review slating dished out by most of the core team.. I simply don't want to deal with that shit for a hobby, and it'd take an impressive salary to make me deal with it professionally. There's being opinionated, then there's being an ass, and I think there's far too much of the latter on the kernel lists. If you look at the modern nerd, compared to the generation that birthed the kernel, they're more socially aware, fashionable animals that are more interested on working on sexy, visible things. Why spend your day dealing with bad attitudes and timing bugs when you can pick up $200k+ writing Ruby apps that enable sharing 5 second video clips? I doubt the kernel will ever be starved for new talent, just that the ratio of people willing to work on it or deal with the social norms it entails is shrinking all the time.
  - DigitalJack 12y ago
    I think a better quote would be: "There's being opinionated, there's being an ass, then there's Linus Torvalds."
    - Gracana 12y ago
      Don't worry though, he's not writing a three-paragraph rant calling you stupid, he's calling your idea stupid! Why do people take things so personally!? ...big ol' "/s" Maybe the lkml is not always so bad, but I'd rather not deal with it and so far I have been content playing with my own projects. I feel kind of silly complaining about it when so many people manage it just fine, but then I see another explosion of aggression and drama and think "how are people okay with this?" I can't imagine having to work in an environment like that.
cnvogel 12y ago
Not everyone writes code for a critical Linux-based life support system. Very often it's just people who want to have a 7-segment LED display show the number of unread emails, or get a soundcard running, something along these lines. And for them it's completely ok that there's a race-condition that might crash their machine if the kernel-module is unloaded just at the instant where the USB plug is removed -- that's something to learn, once the general grasp of how kernel-code with it's myriad of callbacks and hierarchies of subsystems has been reached. Most crashes also will be catched by the Kernel and will leave you with nothing more than a [42568.139488] BUG: unable to handle kernel NULL pointer dereference at 0000000000000004 [42568.139805] IP: [<ffffffffa003e008>] init_module+0x8/0x20 [my_dummy_module] [42568.140308] PGD 4450f067 PUD 4450e067 PMD 0 [42568.140882] Oops: 0002 [#1] PREEMPT SMP (...) in your dmesg, and maybe a dead "insmod" process in your process table. Make a note of the call trace, sync, reboot (maybe hard, using the magic sysrq key) and you are off for the next round of debugging. Try it, it's fun. Also there's kgdb which you can use when your bugs aren't that obvious. I never had serious data loss (maybe the freshly compiled .o files will have 0-size, if I was in too much of a hurry to insmod ./my_module.ko. And even if, it will teach you to have an easily restorable development machine. (I triggered the oops with...) int init_module(void) { int *ptr = 0x0000000000000004ULL; *ptr = 17; return 0; }
- mschuster91 12y ago
  > Very often it's just people who want to have a 7-segment LED display show the number of unread emails, or get a soundcard running, something along these lines. 7-seg LEDs are best controlled by a microcontroller attached via USB or a USB-RS232 interface, as next to no one ships a parallel printer port. Soundcards these days usually all already have kernel drivers, I haven't heard of any modern chip without. And USB stuff is best developed in userspace with libusb, it's way easier...

Nursie 12y ago

Cool, definitely relevant to my interests. I've done various ARM/orion5x platform porting related stuff before but never a module...

chrisBob 12y ago

Do most kernel programmers start out working on their regular daily-use machine? My only experience is with an embedded linux environment, and I would probably recommend starting with something like a RPi so that your environment is easy to restore if you mess something up. I also recommend "Linux Device Drivers" from O'Reilly. http://lwn.net/Kernel/LDD3/ http://lwn.net/Kernel/LDD3/

sophacles 12y ago
Whenever I do kernel stuff, it happens in a VM, limited to a subset of the cores available on the machine. That way when I inevitably bork something, I can just forcibly stop it. Another trick I've used is making as much of my core logic as possible happen in functions that are called from the kernel hooks, rather than in the kernel hooks. Then you can test a ton of it in userland to handle basic logic and flow control bugs, finally porting the code to the module. To help with this, it sometimes is actually faster to copy some of the kernel routines you use to a different .c file and change things like kprintf and kmalloc for the userland versions. It isn't going to be a perfect port, but it helps get some of the complex stuff out of the way where you have your familiar userland tools. All that said - I've only dabbled in the kernel a little, there are probably better ways that someone can chime in on.
- solarexplorer 12y ago
  Keeping the complex stuff out of the kernel is good advice. (and not just during development)
  - sophacles 12y ago
    True! Unfortunately it's sometimes unavoidable. The specific case I was remembering involved writing a qdisc... it was actually more complex to move that out of the kernel via nf_queue than to just do it in-kernel. (At least some parts of it.)
- sitkack 12y ago
  Idea: sysctl/ioctl rest interface so one can drive the new kernel module from the host vm.

jvns 12y ago

This is so great! Writing toy kernel modules on your own is a super fun way to get into kernel programming. There's no danger of anything going wrong, especially if you develop inside a VM. My partner wrote a kernel module for a talk I gave that rickrolls your file system -- it overrides the 'open' system call so that any time you open a '.mp3' file, it plays Rick Astley instead. (this requires you to have Rick Astley already on your computer =)) The source code for that (and a few other toy kernel modules) is at https://github.com/jvns/kernel-module-fun https://github.com/jvns/kernel-module-fun The rickroll module (by Kamal Marhubi) is the most well commented and fun: https://github.com/jvns/kernel-module-fun/blob/master/rickroll.c https://github.com/jvns/kernel-module-fun/blob/master/rickro... I wrote a blog post with more about this kernel-programming-for-fun philosophy: http://jvns.ca/blog/2014/01/04/4-paths-to-being-a-kernel-hacker/ http://jvns.ca/blog/2014/01/04/4-paths-to-being-a-kernel-hac... Of course, writing code for the mainline kernel is a responsible task, but writing kernel code on your own machine that nobody will ever run in production is a great way to learn, and I think it's made me a better programmer. A few concrete ways I think it's made me a better programmer: - I now understand the interface between kernel space and userspace much better (I use strace all the time to debug my regular userspace code!) - realizing that the Linux kernel is just written in C and it's something I can read (and understand! and change!) was a big deal for me - understanding the basics of the underlying operating system helps me reason about my programs better - I now feel like if I ran into an actual kernel bug or performance problem, it's something I could conceivably attempt to fix and understand, instead of hiding under the bed and saying OH NO THE KERNEL IT IS TOO HARD

deutronium 12y ago

This is my first kernel module, which 'fakes' your system uptime: http://www.anfractuosity.com/projects/uptime/ http://www.anfractuosity.com/projects/uptime/ Personally I think people should be encouraged to hack around with the kernel, rather than be afraid of it, how else can we learn?

benjojo12 12y ago

I had great fun writing a silly toy kernel module a few weeks back, ( https://github.com/benjojo/dev_markov https://github.com/benjojo/dev_markov ) Would totally recommend anyone curious giving it a go.

Aardwolf 12y ago

VirtualBox uses a kernel module, and that is annoying because the linux kernel API changes with every update, which means its kernel module needs to be recompiled. Since I update my Linux quite often, that basically means: Everytime some friends call me to play a certain game online with each other, they need to be waiting a bit longer for me because I have yet again to recompile VirtualBox's kernel modules. Is there a way to convert something that requires kernel modules, to run fully in userspace? I'd be happy to lose some efficiency for this. Just a 100% reliable game to run my game would be perfect. Are there things you can do only in kernel modules and not in userspace, and does VB need this? In theory userspace can do everything you need, right? Graphics, networking, audio, ...

aquayellow 12y ago
Virtualbox is an out-of-tree module and hence needs to be compiled. The base package is GPLv2 and is compatible with the Linux kernel license, but I don't know if Virtualbox developers ever tried to "upstream" their module. This is probably the easiest way to solve your problem. Most Virtualization Software these days use virtualization hardware extensions (Intel VMX/AMD SVM) and so the "hypervisor code" needs to be able to execute privileged instructions, eg - you can't run VMON from ring 3/userspace.
- justincormack 12y ago
  Apparently the virtualbox code is terrible, hence the lack of upstreaming...
josteink 12y ago
DKMS might help if you have access to the source. https://en.m.wikipedia.org/wiki/Dynamic_Kernel_Module_Support https://en.m.wikipedia.org/wiki/Dynamic_Kernel_Module_Suppor...
jimmaswell 12y ago
I'd just update the kernel less if i were you. There's not really much of a point updating the kernel all the time for normal use is there?

Myrmornis 12y ago

Really great article.

jhiesey 12y ago

I might be missing something, but the final version still looks susceptible to deadlock to me. Suppose the buffer is empty. Process A, which is reading, gets through the mutex_unlock() call and into wait_event_interruptible(), which seems to first internally check the condition. It is then switched out before it internally waits. Process B then runs all the way through, acquiring the mutex, doing its work, and runs wake_up_interruptible(), which won't do anything, because process A hasn't started waiting. Process B then releases the mutex and gets switched out. Process A then gets switched back in, and then immediately starts waiting, with no more events to wake it up. Why won't it just wait forever (deadlock)? Am I missing something? Do all processes occasionally get woken up without an explicit wake up?