5 ms·

I am impressed with the level of linux knowledge in this thread. How do people become linux kernel hackers? Most of the developers I know (including myself) use

by zests 6y ago

I am impressed with the level of linux knowledge in this thread. How do people become linux kernel hackers? Most of the developers I know (including myself) use linux but have very little awareness beyond application level programming.

the8472 6y ago
You don't necessarily have to be a kernel hacker to be familiar with many of the features that the kernel provides. Just doing application debugging often requires to dig deeper until you hit some kernel balrogs. Container problems? Namespaces, Cgroups, ... Network problems? Netfilter, tc, lots of sysctl knobs, tcp algorithms (cue 1287947th thread on nagle/delayed acks/cork) Slow disk IO? Now you need to read up on syscalls and maybe find more efficient uses. Copy_file_range doesn't work as expected? Suddenly you're reading kernel release notes or source code.
marcosdumay 6y ago
> How do people become linux kernel hackers? Honestly, by hacking it. There's a famous book about Linux internals that I don't remember the name (but has "Linux" and "internals" on it). But I have never seen anybody doing it by reading a book (despite how excellent it can be). You just go change what you want or read the submodule you are interested in understanding, and use the book, site or whatever when you have a problem.
- bigyikes 6y ago
  I have never run into a problem that I thought needed to be solved in the kernel. What kinds of things have you wanted to change?
  - the_only_law 6y ago
    Conversely every problem I’ve run into in a kernel that I could potentially solve requires me to be an expert in 1-3 things outside of the kernel.
  - marcosdumay 6y ago
    The first time I went into it was to write a driver for a device in my undergrad. After that I've changed a driver here or there (never anything worth merging), and needed the documentation of the sound systems. It's not an easy thing, by any means. Just locating where you have to touch on the source tree is a problem that will lead you to plenty of books or sites. But don't try reading those before you have a problem to solve, you will lose time and drown in information. (By the way, I am assuming you know how syscalls work. If you don't, go study that before you start anything.)
  - cgh 6y ago
    In my case, which is probably typical, there was a bug in a device driver for some obscure thing we were using where I worked. So I had to dive into the world of kernel modules and fix it. I think a lot of kernel knowledge and development is commercially-driven, in this sense.
  - sdlion 6y ago
    Once I found myself reading about device module programming since the common USB-Serial device module (I forgot its name, cdc something) wasn't properly working for a Chinese multiserial port chip inside a GSM cluster modem (one USB device to multiple serial ports). I was attempting to hack away a simple example but I found the USB-Serial (more) generic driver intended for "test only" and... it just worked. Another reason for reading about IO calls, schedulers, etc? That "I'm still writing data into your USB flash drive even when the GUI says it finished 5 minutes ago" that I hate so much.
  - bloaf 6y ago
    I learned 90% of the things I know about operating systems by solving problems I myself caused.
  - sebcat 6y ago
    Apart from Linux hw support for things at work, I implemented a fairly simple pseudo-device for establishing TCP connections from a process in capability mode on FreeBSD. The device driver has support for a denylist to disallow connections to specific IP ranges. It has multiple syscalls wrapped into one ioctl, and sockets opened from the device always had TCP_NODELAY, O_CLOEXEC and SOCK_NONBLOCK set. Worked pretty well for its intended use case. https://github.com/sebcat/yans/blob/master/drivers/freebsd/tcpsrc/tcpsrc.4.txt https://github.com/sebcat/yans/blob/master/drivers/freebsd/t... https://github.com/sebcat/yans/blob/master/drivers/freebsd/tcpsrc/tcpsrc.c https://github.com/sebcat/yans/blob/master/drivers/freebsd/t...
- asicsp 6y ago
  >There's a famous book about Linux internals that I don't remember the name (but has "Linux" and "internals" on it) this one? https://0xax.gitbooks.io/linux-insides/content/index.html https://0xax.gitbooks.io/linux-insides/content/index.html though it says insides instead of internals
  - marcosdumay 6y ago
    Yes, this one. Thanks.
- alfiedotwtf 6y ago
  I think you were thinking “Linux Core Kernel” by Scott Maxwell. And yes, it’s an awesome book that’s copied the style of a same type of book that annotates the SVR4 kernel
deleted 6y ago
[deleted]
01100011 6y ago
For the most part, it's just software. If you have the time and the interest, you can learn it like anything else. At some level, it requires an awareness of how the hardware works(page tables/MMUs/IOMMUs, interrupts, SMP, NUMA, etc). I don't mean to downplay the investment, but if you're already an experienced software engineer you can get into it if it interests you. There is a different mindset among systems software programmers though. Reliability comes first, performance and functionality come second. It's a world away from hacking python scripts that only need to run once to perform their function.
gpanders 6y ago
I learned a TON about the Linux kernel through writing custom device drivers for FPGAs. Granted most of my experience is in the driver area and not in any of the subsystems, but even still I have a much better grasp of how the kernel operates now (and even more importantly, I know how to navigate it and how to find relevant documentation).
dboreham 6y ago
UTSL
amboar 6y ago
As others have said, hacking it, certainly. But if you're not up for that and would like something more passive, read LWN.net (and possibly subscribe!)
yobert 6y ago
I learned a lot by trying to make Go talk to ALSA without using any existing C interfaces. Just happy exploration goes a long ways!

by zests 6y ago

the8472 6y ago

You don't necessarily have to be a kernel hacker to be familiar with many of the features that the kernel provides. Just doing application debugging often requires to dig deeper until you hit some kernel balrogs. Container problems? Namespaces, Cgroups, ... Network problems? Netfilter, tc, lots of sysctl knobs, tcp algorithms (cue 1287947th thread on nagle/delayed acks/cork) Slow disk IO? Now you need to read up on syscalls and maybe find more efficient uses. Copy_file_range doesn't work as expected? Suddenly you're reading kernel release notes or source code.

marcosdumay 6y ago

> How do people become linux kernel hackers? Honestly, by hacking it. There's a famous book about Linux internals that I don't remember the name (but has "Linux" and "internals" on it). But I have never seen anybody doing it by reading a book (despite how excellent it can be). You just go change what you want or read the submodule you are interested in understanding, and use the book, site or whatever when you have a problem.

bigyikes 6y ago
I have never run into a problem that I thought needed to be solved in the kernel. What kinds of things have you wanted to change?
- the_only_law 6y ago
  Conversely every problem I’ve run into in a kernel that I could potentially solve requires me to be an expert in 1-3 things outside of the kernel.
- marcosdumay 6y ago
  The first time I went into it was to write a driver for a device in my undergrad. After that I've changed a driver here or there (never anything worth merging), and needed the documentation of the sound systems. It's not an easy thing, by any means. Just locating where you have to touch on the source tree is a problem that will lead you to plenty of books or sites. But don't try reading those before you have a problem to solve, you will lose time and drown in information. (By the way, I am assuming you know how syscalls work. If you don't, go study that before you start anything.)
- cgh 6y ago
  In my case, which is probably typical, there was a bug in a device driver for some obscure thing we were using where I worked. So I had to dive into the world of kernel modules and fix it. I think a lot of kernel knowledge and development is commercially-driven, in this sense.
- sdlion 6y ago
  Once I found myself reading about device module programming since the common USB-Serial device module (I forgot its name, cdc something) wasn't properly working for a Chinese multiserial port chip inside a GSM cluster modem (one USB device to multiple serial ports). I was attempting to hack away a simple example but I found the USB-Serial (more) generic driver intended for "test only" and... it just worked. Another reason for reading about IO calls, schedulers, etc? That "I'm still writing data into your USB flash drive even when the GUI says it finished 5 minutes ago" that I hate so much.
- bloaf 6y ago
  I learned 90% of the things I know about operating systems by solving problems I myself caused.
- sebcat 6y ago
  Apart from Linux hw support for things at work, I implemented a fairly simple pseudo-device for establishing TCP connections from a process in capability mode on FreeBSD. The device driver has support for a denylist to disallow connections to specific IP ranges. It has multiple syscalls wrapped into one ioctl, and sockets opened from the device always had TCP_NODELAY, O_CLOEXEC and SOCK_NONBLOCK set. Worked pretty well for its intended use case. https://github.com/sebcat/yans/blob/master/drivers/freebsd/tcpsrc/tcpsrc.4.txt https://github.com/sebcat/yans/blob/master/drivers/freebsd/t... https://github.com/sebcat/yans/blob/master/drivers/freebsd/tcpsrc/tcpsrc.c https://github.com/sebcat/yans/blob/master/drivers/freebsd/t...
asicsp 6y ago
>There's a famous book about Linux internals that I don't remember the name (but has "Linux" and "internals" on it) this one? https://0xax.gitbooks.io/linux-insides/content/index.html https://0xax.gitbooks.io/linux-insides/content/index.html though it says insides instead of internals
- marcosdumay 6y ago
  Yes, this one. Thanks.
alfiedotwtf 6y ago
I think you were thinking “Linux Core Kernel” by Scott Maxwell. And yes, it’s an awesome book that’s copied the style of a same type of book that annotates the SVR4 kernel

deleted 6y ago

[deleted]

01100011 6y ago

For the most part, it's just software. If you have the time and the interest, you can learn it like anything else. At some level, it requires an awareness of how the hardware works(page tables/MMUs/IOMMUs, interrupts, SMP, NUMA, etc). I don't mean to downplay the investment, but if you're already an experienced software engineer you can get into it if it interests you. There is a different mindset among systems software programmers though. Reliability comes first, performance and functionality come second. It's a world away from hacking python scripts that only need to run once to perform their function.

gpanders 6y ago

I learned a TON about the Linux kernel through writing custom device drivers for FPGAs. Granted most of my experience is in the driver area and not in any of the subsystems, but even still I have a much better grasp of how the kernel operates now (and even more importantly, I know how to navigate it and how to find relevant documentation).

dboreham 6y ago

UTSL

amboar 6y ago

As others have said, hacking it, certainly. But if you're not up for that and would like something more passive, read LWN.net (and possibly subscribe!)

yobert 6y ago

I learned a lot by trying to make Go talk to ALSA without using any existing C interfaces. Just happy exploration goes a long ways!