4 ms·

If MS or any FAANG wants to hire you for a kernel dev position, go for it; they clearly see a lot of potential in you that maybe you don't see yourself. I want

If MS or any FAANG wants to hire you for a kernel dev position, go for it; they clearly see a lot of potential in you that maybe you don't see yourself. I want to point out a couple of things that you should be prepared for: > 2. Debugging exclusively via metrics and logs, since I can't just attach a debugger to a running server. You often can't do this in kernel/OS development work either. Serial printf logging is often required. It can be a real "my tools to debug my tools are broken" slog. > 3. Designing systems in general. Some people love the challenge of distributed transactions, eventual consistency and all that jazz, but it just rubs my brain the wrong way. I'm not interested at all in that problem space [1]. I'm not sure I understand this issue. I mean, kernels have subsystems for doing stuff; you might need to design one someday? But it won't be dull-as-dishwater web technology stacks, it'll be you writing data structures directly in C/C++ or Rust if MS goes there.

ccrush 4y ago
Can you no longer use windbg to attach to a running kernel on a remote box, and debug the remote box from your development machine? I believe that's how kernel debugging is generally done.
- ace2358 4y ago
  That’s probably true, but sometimes you have to go one level deeper right? What happens when your debugger has crashed?
  - Arainach 4y ago
    In 8 years on the Windows team I can't recall the debugger ever crashing. If it does, you open the debugger again and reconnect to the target machine.
    - hyperrail 4y ago
      Heh, I wish that had ever been the case for me. Sometimes it feels like WinDbg freezes every other command. Plus, getting it to reconnect to a kd after it freezes and I kill it was such an exercise in frustration that if the target machine were my own repro box instead of a stress break remote, I would have just hard-reset the machine or (if it were a VM) restored it from snapshot.
- amluto 4y ago
  (I’m a Linux kernel dev) Tools like windbg probably work great until you are debugging the code that runs when windbg tries to take over. Or you’re debugging something sensitive to interrupts and it’s literally impossible to keep up with a tool like windbg. Or you a debugging something that overwrites windbg in memory because you have a corrupt pointer. Or you triple-fault the machine and it reboots with so much prejudice that windbg doesn’t have a chance. Or you’re poking at hardware that the debugger can’t usefully interact with. Etc. Basically, in the kernel, a lot of the things that a working kernel does to hold your hand aren’t available. So there’s a lot of time spent just thinking or adding logging statements or otherwise using low tech tools. (Or using high tech instrumentation or sanitizers!) Kernel development is great :)
  - DaiPlusPlus 4y ago
    I thought everyone doing OS/kernel dev would be running code in a VM they can pause, snapshot, and single-step all without a guest debugger?
    - xmodem 4y ago
      Virtual machines are used extensively by kernel developers but are not an adequate substitute for real hardware in a lot of situations. After all, one of a kernel's jobs is to abstract away differences in underlying hardware.
  - starfleet_bop 4y ago
    Does x86 have a JTAG like / hardware debugger?
    - vbitz 4y ago
      You can get something like JTAG from Intel with a NDA. https://designintools.intel.com/Silicon_View_Technology_Closed_Chassis_Adapter_p/itpxdpsvt.htm https://designintools.intel.com/Silicon_View_Technology_Clos... Most of the time you can do kernel dev with a serial port and printf though.
  - 4y ago
shultays 4y ago
Serial printf logging is often required This reminded me using "printk" in an incorrect place in an operating system course assignment. It was actually printk itself that was hanging the kernel!
maybekerneldev 4y ago
> > 3. Designing systems in general. Some people love the challenge of distributed transactions, eventual consistency and all that jazz, but it just rubs my brain the wrong way. I'm not interested at all in that problem space [1]. >I'm not sure I understand this issue. I mean, kernels have subsystems for doing stuff; you might need to design one someday? But it won't be dull-as-dishwater web technology stacks, it'll be you writing data structures directly in C/C++ or Rust if MS goes there. Ah, I meant designing distributed systems. Like, figuring out the difference services, storage requirements, databases, then planning out the infrastructure. It's really not my cup of tea. I do enjoy designing systems/subsystems/components as long as they're within a single host :)

ccrush 4y ago

Can you no longer use windbg to attach to a running kernel on a remote box, and debug the remote box from your development machine? I believe that's how kernel debugging is generally done.

ace2358 4y ago
That’s probably true, but sometimes you have to go one level deeper right? What happens when your debugger has crashed?
- Arainach 4y ago
  In 8 years on the Windows team I can't recall the debugger ever crashing. If it does, you open the debugger again and reconnect to the target machine.
  - hyperrail 4y ago
    Heh, I wish that had ever been the case for me. Sometimes it feels like WinDbg freezes every other command. Plus, getting it to reconnect to a kd after it freezes and I kill it was such an exercise in frustration that if the target machine were my own repro box instead of a stress break remote, I would have just hard-reset the machine or (if it were a VM) restored it from snapshot.
amluto 4y ago
(I’m a Linux kernel dev) Tools like windbg probably work great until you are debugging the code that runs when windbg tries to take over. Or you’re debugging something sensitive to interrupts and it’s literally impossible to keep up with a tool like windbg. Or you a debugging something that overwrites windbg in memory because you have a corrupt pointer. Or you triple-fault the machine and it reboots with so much prejudice that windbg doesn’t have a chance. Or you’re poking at hardware that the debugger can’t usefully interact with. Etc. Basically, in the kernel, a lot of the things that a working kernel does to hold your hand aren’t available. So there’s a lot of time spent just thinking or adding logging statements or otherwise using low tech tools. (Or using high tech instrumentation or sanitizers!) Kernel development is great :)
- DaiPlusPlus 4y ago
  I thought everyone doing OS/kernel dev would be running code in a VM they can pause, snapshot, and single-step all without a guest debugger?
  - xmodem 4y ago
    Virtual machines are used extensively by kernel developers but are not an adequate substitute for real hardware in a lot of situations. After all, one of a kernel's jobs is to abstract away differences in underlying hardware.
- starfleet_bop 4y ago
  Does x86 have a JTAG like / hardware debugger?
  - vbitz 4y ago
    You can get something like JTAG from Intel with a NDA. https://designintools.intel.com/Silicon_View_Technology_Closed_Chassis_Adapter_p/itpxdpsvt.htm https://designintools.intel.com/Silicon_View_Technology_Clos... Most of the time you can do kernel dev with a serial port and printf though.
- 4y ago

shultays 4y ago

Serial printf logging is often required This reminded me using "printk" in an incorrect place in an operating system course assignment. It was actually printk itself that was hanging the kernel!

maybekerneldev 4y ago

> > 3. Designing systems in general. Some people love the challenge of distributed transactions, eventual consistency and all that jazz, but it just rubs my brain the wrong way. I'm not interested at all in that problem space [1]. >I'm not sure I understand this issue. I mean, kernels have subsystems for doing stuff; you might need to design one someday? But it won't be dull-as-dishwater web technology stacks, it'll be you writing data structures directly in C/C++ or Rust if MS goes there. Ah, I meant designing distributed systems. Like, figuring out the difference services, storage requirements, databases, then planning out the infrastructure. It's really not my cup of tea. I do enjoy designing systems/subsystems/components as long as they're within a single host :)