7 ms·
Ideas for getting started in the Linux kernel
- voltagex_ 10y agoI second the recommentation for the Eudyptula Challenge although the "rule" about not asking/posting for help is probably why I have been on Challenge 5 for about a year.
- rrmm 10y agoLots of good suggestions on the list. I would add a few (maybe advanced ones) that I don't think I saw: * write a device driver for a real or virtual device; hook into the various kernel virtual filesystems that allow userspace interaction. * write a small network stack (for example a raw protocol on top of ethernet). * add a scheduling algorithm or mess with an existing one; analyze its performance impact. Lots of new-to-me tools in that list were interesting. Back in my day, we had to use a serial port console, and if we were fancy we would set it up so that you could remotely cycle the computer's power when it hung up (uphill both ways in the snow).
- voltagex_ 10y agoEverything old is new again - serial port debugging has been invaluable for me when I was messing with the Dreamplug, and then the Raspberry Pi. I think I'd add using a serial-based terminal to that list too. Embedded hardware can be infuriating, but fun.
- rkachowski 10y agoIs this still running? I've sent a few emails about joining the challenge around 2 months ago (after a multi year break) but have yet to receive even the automated response..
- tronje 10y agoDid you make sure to send non-HTML email?
- rkachowski 10y agoyeah - i'd gotten up to level 9 before. i just wanted to start again but didn't receive any response.
- TempleOSV409 10y agoStart with TempleOS. The CIA does not what you doing this.
- tayo42 10y agoI feel like this underestimates how hard it is to get started. I've started playing around with the kernel and making small modifications. It's not straight forward. For me just getting it complied in a vm was an ordeal. Most books are pretty old and if you go google for what ever you want you end up with a lot of very dense resources. Then you need to figure out if that article you found actually applied to whatever current kernel your using and what's changed since then. Ive been doing like hours of reading for a few line change here and then. And somewhat off topic. I think it will be interesting to see what happens to linux in the long term. Are that many new/young programers getting into kernel development? Id imagine there's not to many people getting into programing now that are writing c or working on the kernel. Who is going to replace the kernel developers of today? Or am I just totally off?
- ashitlerferad 10y agoYoung folks come through the GSoC/Outreachy programs: https://wiki.linuxfoundation.org/en/Google_Summer_of_Code https://wiki.linuxfoundation.org/en/Google_Summer_of_Code https://kernelnewbies.org/OutreachyIntro https://kernelnewbies.org/OutreachyIntro Lots of new folks come through companies hiring people to work on Linux.
- yeukhon 10y agoThat's basically the experience everyone has to go through with any new codebase. For example, when I worked on a couple Firefox bugs a few years ago, I hadn't touch C++ for several years. I did have to spend several hours just to get a hang of things. Luckily what I needed to change was not too challenging. Several dozen lines were added and overall the necessary functions were all . hg grep helped a lot to find what I want. If I were to implement a new driver from scratch, that would be really tough. Though since C and C++ aren't something I am very comfortable with (I haven't written a single line of and C and C++ for so long), it can be really tough especially during debugging.
- _pmf_ 10y ago> That's basically the experience everyone has to go through with any new codebase. Yes; for a codebase of comparable size and widespread usage, the Linux kernel is very, very easy to get into and very, very well documented. Go to Documentation/[your subsystem] and you'll get something useful, usually a reference on a driver you can base yours on (which is how practically all drivers are created). Nice ways to get started: - simple character device - SPI slave and SPI protocol driver
- amboar 10y agoAn approach that's worked well for me was to first hack qemu to add some new (emulated) hardware, then write the kernel drivers using qemu to test (followed by running on the hardware). That does leave you with two problems instead of one, but is quite flexible when you are trying to track down bugs and can increase your hack/compile/test velocity.
- andrey_utkin 10y agoWhich kinds of drivers you have developed this way? I guess it's not possible to test driver code unless probe() succeeds, which requires actual hardware to be in place. You say you had "emulated" hardware - what this means, exactly? Thanks in advance for explanations.
- _pmf_ 10y ago> I guess it's not possible to test driver code unless probe() succeeds, which requires actual hardware to be in place. Not the OP, but I think qemu and VirtualBox have APIs for implementing custom virtual hardware (similar to how the rest of the machine is emulated). To your kernel running in the VM, the emulated device appears as a real device.
- amboar 10y ago> You say you had "emulated" hardware - what this means, exactly? Software pretending it is hardware, for example QEMU: http://wiki.qemu.org/Main_Page http://wiki.qemu.org/Main_Page > Which kinds of drivers you have developed this way? As an example, this approach helped me develop a pin-controller driver[1]. I first added a new bare-bones SoC and machine to QEMU[2] to give me enough of a environment to start adding other models, such as the SoC's System Control Unit[3], which in-turn contains the registers the pin-controller driver pokes at. Using QEMU I could control the initial values in the registers and exercise the pin controller driver's implementation. > I guess it's not possible to test driver code unless probe() succeeds, which requires actual hardware to be in place. Not entirely - if you are writing the driver you can always pretend it succeeds! Ideally you want to get some gratification as soon as possible - if that means making some assumptions and nasty hacks, that's fine (as long as it won't smoke something!) Then start iterating to iron them out. At least, that's the kind of approach that works for me. [1] https://lkml.org/lkml/2016/7/20/69 https://lkml.org/lkml/2016/7/20/69 [2] https://lists.nongnu.org/archive/html/qemu-devel/2016-03/msg03667.html https://lists.nongnu.org/archive/html/qemu-devel/2016-03/msg... [3] https://lists.gnu.org/archive/html/qemu-devel/2016-06/msg07042.html https://lists.gnu.org/archive/html/qemu-devel/2016-06/msg070...
- a3_nm 10y agoAs for books about the Linux kernel, I'd recommend https://github.com/0xAX/linux-insides https://github.com/0xAX/linux-insides (which, oddly, didn't seem to be linked)
- bgilroy26 10y agoI would recommend Robert Love's Quora answers as an additional low barrier starting point http://qr.ae/TNK9UD http://qr.ae/TNK9UD
- copser 10y agoIs there any kernel that was written and maintained using python language? I assume that the C and C++ is a must to learn to this, but is it or we can work with karnel using python.
- jm547ster 10y agoBeside performance and safety concerns even a kernel written in C needs to to jump down to native assembly for certain parts, so I don't believe this would/could have been done
- datenwolf 10y agoTechnically there's nothing that forbids doing it. It would essentially boil down to port the Python interpreter / VM / Cython (pick your flavour) to run on the naked metal (hard, but not impossible; probably easier done with Lua), which would mostly involve memory management and all the low level stuff to get it running in the first place. Besides all the really hard stuff one had to solve, ironically mapping a pointer-less / array-less language like Python to a page-wise memory model would be rather easy; as soon as one does not have to care about that things like lists and dicts are contiguous in address space suddenly a lot of constraints you'd have in the Linux kernel would vanish.
- pjc50 10y agoI suppose there are systems that run "micropython" on the bare metal. But note that to work on Python itself often means working on or with CPython in C.
- thearn4 10y agoAs much as I love using Python for my day job, this is one place where it really doesn't fit. Learning OS development is actually a great reason to practice C.
- andrey_utkin 10y agoIt's not that getting started with Linux kernel is inherently hard. It's just what a lot of people want to have as "getting started with Linux kernel" (see motivation part of this article) is hard, but that's not about Linux kernel itself. Some people get this wrong and start thinking that Linux kernel or its community itself is the problem (Maybe kerneldevs and Linus himself are aggressive picky bullies unfriendly to newcomers? Maybe you should ditch C and rewrite kernel in better language? Maybe Linux development process is wrong and they should change it? Maybe Linux maintenance organization structure is bad? Maybe ...) There is also common frustration about friction at getting your patch accepted. But I believe in most cases maintainers are right and there are some issues to fix in any patch, so author has something to do on it more. Maintainers have reasons to care - they are who improve subsystems API and modify ALL drivers in the tree, so they'd better have all drivers as perfect as possible. > Are you interested in knowing how operating systems work in general? Do you want to know how parts of Linux specifically work? An undertaking which a lot of novices take for reasons unknown to me. I was always fine without this. I follow #kernelnewbies IRC channel (good source of help, BTW) and see which questions are being asked. Surprisingly many novices decide to start with understanding kernel codebase in its entirety. Such people should just understand that their aim is inherently hard. Any large codebase has non-obvious and/or complex things, a lot of indirection, etc. - I don't understand how this internal kernal mechanism works, please explain! - It's complicated for outright explanation. If in trouble, read the code. And just trust that it works. What are you trying to achieve? - I'm just studying how kernel works. > Is your hardware broken? Regarding device drivers, a lot of opportunities are missing because most of devices don't have datasheets good enough to allow third-side developer to make a driver. So you just have no way to know how to give commands to your piece of hardware. It's a shame, but this shame is not on Linux, it is on manufacturers. Even if you are stubborn enough to get in touch with manufacturer, you may end up with something like - The development team was dismissed N years ago and people don't work here anymore, so nobody to ask, but if you want NNNNNNN pieces order, we can make new research team for this. You can reverse-engineer something, of course, but as somebody having experience of reverse-engineering heavily obfuscated _sources_ of Linux v2.6 driver, I'd say the amount of wasted time and frustrations is tremendous. > Do you just want to make an Open Source contribution? Do you want a high five? Oh yeah, kernel is a fantastically reputable place to have open source contribution. Your bros will respect you as coolest guy ever, your girlfriend/boyfriend will praise you as superhero, hiring managers will want you. Because they don't know that contributing to kernel is not THAT hard.
- Annatar 10y agoFrom the article: I would like to dispel this notion once and for all: you do not have to have a special talent to work on the kernel unless you count dogged persistence and patience as a talent. He is correct, but in a completely wrong way: this is why GNU/Linux has never been, and never will be anything more than just "good enough" (and there goes my karma in a retaliatory action, but someone had to write it). Let me clarify what I mean by that: an operating system's kernel is the last stop in computer science, and requires formal education, lots and lots of formal education, coupled with lots and lots of experience, coupled with the most important ingredient, insight. That's why when one compares a kernel like Linux and illumos, illumos is light years more sophisticated: it was done by professionals, with formal education, and with deep, deep insights into the thorniest of issues in production at the largest companies in the world. Now, you could argue Linux has this type of exposure now, but what it does not have is 25+ years head start in this exposure, and it does not have the people. And that makes all the difference in the world. And when you read ideas like you do not have to have a special talent to work on the kernel, that kind of mentality speaks volumes of the product. And indeed, Linux is the product of such a mentality.
- hiphopyo 10y agoWhere does OpenBSD's kernel fit into all of this? Isn't that, in turn, lightyears ahead of illumos?
- X86BSD 10y agoI'm not sure in what way they would be light years ahead. But I will say the it's probably much easier to hack on the bsd kernels than Linux. Taking FreeBSD as an example, if for no other reason than the absurdly great documentation, tooling, and developer community.
- hiphopyo 10y agoFor one, Theo De Raadt Says FreeBSD Is Just Catching Up On Security: https://tech.slashdot.org/story/13/12/16/0121213/theo-de-raadt-says-freebsd-is-just-catching-up-on-security https://tech.slashdot.org/story/13/12/16/0121213/theo-de-raa... OpenBSD's documentation is also known to be a tad better than FreeBSD's.
- known 10y agoTODO http://www.google.com/search?q=todo&sitesearch=lxr.free-electrons.com/source&gws_rd=cr&ei=KdSyV7KdDYekjwPl6K6IBQ http://www.google.com/search?q=todo&sitesearch=lxr.free-elec...
- d33 10y agoOne thing I hadn't seen said here yet: Do we really need that many kernel developers? I'm getting the impression that a lot of dev power is wasted just to maintain drivers that should be produced by hardware vendors. The vendors won't do that because there's no stable ABI, which means that people need to reinvent the wheel. One could argue that it adds better quality control and all in all, we get open source implementations of what would instead be proprietary drivers, but in the end, complex drivers aren't good enough anyway (see Nouveau). As for quality control... many say that Linus's view on security is questionable. For one, I don't like the concept of pluggable /stackable security mechanisms (selinux, apparmor, etc) where one consistent system would do. It just feels cluttered.
- the_why_of_y 10y ago> The vendors won't do that because there's no stable ABI, which means that people need to reinvent the wheel. This is utter nonsense; I suggest you have a good look at a "git log" of some driver files source files in linux.git, here's just an example: https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/log/drivers/gpu/drm/amd https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.... Edit: In particular, there not being a stable ABI prevents vendors from re-inventing various wheels because they can just argue for and submit enhancements and changes to various core kernel subsystems so that they better match the requirements of their hardware / driver, instead of putting extra code into every driver like they have to do on other systems where they have to live with whatever ABI was declared stable by the OS vendor a decade ago.
- d33 10y ago@the_why_of_y: what should I note here? That some vendors do that within the source tree? Yes, some do, but many don't, including a fair share of smaller ones.
- the_why_of_y 10y agoThis sadly true, but I don't see how this is caused by lack of a stable ABI, more likely you'll see justifications such as "we need to protect our intellectual properteh", management at hardware vendors not understanding the development process, or just not caring about a niche market (in case of desktop/laptop hardware).