14 ms·

A perennial complaint; Rob Pike had a trollier but similar version years back. I'm not nearly as optimistic that we have the resources to cope with the monstrou

A perennial complaint; Rob Pike had a trollier but similar version years back. I'm not nearly as optimistic that we have the resources to cope with the monstrous complexity of hardware now. An anecdote I like to share is that while I was TA'ing CMU's 15-410 operating systems course (a great course - build a preemptive multitasking operating system from scratch) was that the student projects could run on real - if old - hardware. There was a PC that could boot your stuff if you put it on a disk. This PC had to have PS/2 interface to the keyboard, though. A newer PC would be all USB. Apparently, the complexity of the codebase to talk to the USB device was around the similar level of complexity to the entire preemptive multitasking operating system the students were building (and, of course, considerably less educational). I mean, this wasn't a full-functioned OS, but it allowed multitasking and preemption of kernel calls, so seriously non-trivial. Multiply this story by all the devices a reasonable OS would be expected to talk to and it's a scary prospect for the OS researcher... and if you don't have those devices up and running, good luck supporting most workloads anyone cares about.

darksaints 5y ago
I'd argue that implementing a USB driver would probably be far more educational, in a practical sense, than implementing a kernel. The number of students who will go on to work on core kernels is vanishingly small, but driver work is never going to end. Plus, if we ever are to transition to the world of microkernels (something that OS researchers have been pushing for decades now), we're gonna need a massive amount of people writing drivers just to get us to the point where things are barely functional.
- toast0 5y ago
  It's not just a USB driver though. You've got to support PCI (or something) to find the USB controller, then you've got to fiddle with that, then you have to discover the devices, then you have to interface with them. Reading from a keyboard is much simpler. Probably makes more sense to run the computer headless with a serial terminal and make the students poke at a UART. PC standard UARTs are pretty easy, and I think it's easier to find a motherboard with a COM1 header than a PS/2 port (at least my latest hardware matches that description).
  - userbinator 5y ago
    You've got to support PCI (or something) to find the USB controller PCI is nothing in comparison to the complexity of USB, especially since the BIOS will usually have set up the I/O ranges for the devices already. It's literally a few dozen bytes of machine code to scan the PCI(e) bus for the device you want.
    - toast0 5y ago
      Oh sure, PCI isn't too bad, but it's way more work than inb(0x60) or inb(0x3f8). (although to be fair, you probably want to setup an interrupt, which is real work)
- glangdale 5y ago
  No. Driver work is painful but it mostly teaches you about the h/w in question; you don't learn deep principles from a lot of it. You just grind away, spec in hand, until stuff works. A lot of this winds up being "one damn thing after another". There's a good deal of craft to it, and I don't mean to deprecate it, but it's not broadly applicable. Conversely, the work they did in kernels has a huge amount of transfer to thinking about concurrency, which is truly valuable - and deep. It also meant that they acquired a far less magical idea of where things like processes and threads come from, having actually built the mechanisms to make them happen (you build a thread library in a warm-up project). As for microkernels - they often punt a lot of the hard concurrency stuff up to user space servers, which will wind up needing exactly the kind of concurrency that the students learned to build.
  - deleted 5y ago
    [deleted]
  - naasking 5y ago
    > No. Driver work is painful but it mostly teaches you about the h/w in question; you don't learn deep principles from a lot of it. You just grind away, spec in hand, until stuff works. A lot of this winds up being "one damn thing after another". There's a good deal of craft to it, and I don't mean to deprecate it, but it's not broadly applicable. I agree you don't learn deep principles from it, but I disagree that it's not broadly applicable based on your description. Grinding away with spec in hand, "one damn thing after another" sounds exactly like most programming people are likely to encounter in their career.
    - simiones 5y ago
      Sure, but you're not learning anything per se - perhaps some discipline. To some approximation, any two decently competent programmers will take similar amounts of time writing a device driver for a complex novel device regardless of previous experience - the vast majority of the job is in simply going through the spec.
      - 5y ago
- Ericson2314 5y ago
  I suppose it would be useful if it taught students how much this industry is hare rather than tortoise nonsense where the interfaces are sheer accidents, and the implementations are mad-dash to keep up. http://langsec.org/ http://langsec.org/ But what you propose is preprofesional drival. It's bad enough that education is subsidized for employers already, we don't need to stoop to them further.
0xcde4c3db 5y ago
It seems like the done thing these days (for people who care about running on real hardware) is to bring up educational/experimental kernels on Raspberry Pi, using the "mini UART" for console I/O and the SD interface for storage.
User23 5y ago
> I was TA'ing CMU's 15-410 operating systems course (a great course - build a preemptive multitasking operating system from scratch) That class bordered on legendary. Awesome that you got to TA it. I still have an irrational fondness for AFS.
- glangdale 5y ago
  It was a fantastic experience, although TA'ing it was daunting. Like the apocryphal Ledru-Rollin quote: "There go the people. I must follow them, for I am their leader", it was frankly hard to keep up with all the different approaches students conjured up. I believe that the course number changed at some point - or some shallow aspect of the course was set to change - and a bunch of industry people called the university to express concern. It is/was (I can't speak for the current iteration of the course, although I have no indication that quality has slipper or anything) truly transformative. I offered to run a 'franchise' of the course at the University of Sydney a while back and was informed that anything quite that transformative wasn't really an option; our job was at least in part to pass engineering students who didn't care that much about computing.
  - josephg 5y ago
    > I offered to run a 'franchise' of the course at the University of Sydney a while back and was informed that anything quite that transformative wasn't really an option Thats very disappointing. You might have more luck joining the University of New South Wales OS team. (For context, UNSW is a local rival of USYD). When I was a student there, the OS course was legendary. I only did the "basic" OS course. Our assignments led us to implement a bunch of syscalls for handling file IO, and write a page fault handler for a toy operating system. I'm not sure what they do in the advanced OS course - but it has a reputation for a reason. And it looks like[1] its still run by Gernot Heiser, who's a legend. He's the brain behind SeL4 - which is the world's first (and only?) formally verified OS kernel. I'm kicking myself for not doing his advanced OS course while I was a student. [1] http://www.cse.unsw.edu.au/~cs9242/current/ http://www.cse.unsw.edu.au/~cs9242/current/
userbinator 5y ago
PS/2 interfaces are actually still quite common for "enthusiast" motherboards, because of the low latency (the simplicity compared to USB certainly has a lot to do with it.) Laptops also tend to have PS/2 keyboards and a "mouse" (the trackpad, which can be switched to a proprietary protocol later for more features) because the EC they use provides that "for free" and that industry tends to have a "it works, why change it and break things" attitude.[1] https://news.ycombinator.com/item?id=17692499 https://news.ycombinator.com/item?id=17692499 That said, BIOSes can also emulate PS/2 with USB (no doubt via SMM). This is usually a setting called "legacy keyboard/mouse support". [1] The notable exception being Apple. Think Different. https://news.ycombinator.com/item?id=12924051 https://news.ycombinator.com/item?id=12924051
- Bigpet 5y ago
  I think there was a Ben Eater video debunking the "PS/2 is faster than USB" myth. That used to be true, but the PS/2 is so much slower (lower frequency) that by the time a package containing the mouse or keyboard info is finished transmitting the usb port started and finished a poll
  - bear8642 5y ago
    think might be at end of this one <https://www.youtube.com/watch?v=7aXbh9VUB3U https://www.youtube.com/watch?v=7aXbh9VUB3U> though there's a few he's done recently on USB spec
  - dmz73 5y ago
    USB is faster and it is better when there is nothing saturating the USB with traffic and CPU can keep up. Most of the time there will be no noticeable difference using the USB mouse or PS/2 mouse. But if CPU is busy and USB bus is saturated, USB mouse will start to be less responsive than PS/2 mouse. In the extreme cases even USB keyboard will become unresponsive where the old PS/2 keyboard still works fine. This can make a difference between being able to terminate a misbehaving process and resume working or having to power-cycle the computer.
    - 5y ago
hoseja 5y ago
I have a modern consumer motherboard that has intermittent problems connecting all it's USB ports, with the latest "stable" BIOS supposedly fixing this but also reportedly having worse performance.
throwawaylinux 5y ago
> A perennial complaint; Rob Pike had a trollier but similar version years back. I'm not nearly as optimistic that we have the resources to cope with the monstrous complexity of hardware now. "Hardware" (read: device drivers) is not notably complicated IMO. That's not to say they're simple, but driving devices from a software point of view is pretty similar to interacting with other software. You read the spec (APIs), write code to set up data structures or registers a certain way, parse responses, etc. Writing a modern full fledged USB stack is very complex, but would not be more complex than writing a modern full TCP/IP stack for example. A lot of device programming models have gotten simpler too. Device drivers used to be notorious deep voodoo magic e.g., in cases of the IDE disk driver, but that was not really "complexity" of the software logic so much as hundreds of special cases and deviations from standards or odd behavior in all manner of IDE controllers and devices. To the point where the real wizards were the ones who had access to internal data sheets, errata, or reverse engineered firmware from these devices, or otherwise spent countless hours poking at things and reconstructing quirks for themselves, probably bricking a lot of silicon and spinning rust in the process. But systems and devices are now getting to the point where things don't work that way anymore, silicon power is so cheap and firmware is on everything. The NVMe device for example sets up pretty simple packet-type command queues and sends requests and receives responses for operations - query device information, perform a read or write, etc). It's not quite that simple, there are some quirks and details, but it's quite a lot like writing a client for a server (HTTP, perhaps). I think other device interfaces will evolve toward cleaner simpler models like this too. One exception to this is GPUs of course. The other thing is with moore's law continuing to slow down there will be increased incentive to move more processing out to devices. It's long been happening with high end network devices, but with technologies like CXL.cache coming soon, I would expect that trend to keep ramping up and come to other accelerators (crypto, AI, even disk). So.. it's a mix. Things are definitely getting more complex, but AFAIKS the complexity of interacting with hardware is not increasing at a much greater rate than the complexity of interacting with other software. And, as always this complexity is made possible by abstractions and layers and interfaces. It's not clearly exceeding our ability to cope with it.
- bcantrill 5y ago
  This is entirely wrong, and I'm sorry that those who do work at the hardware/software interface have done their work apparently so well as to allow you to harbor this delusion. Hardware is complex -- exceedingly so -- but (just as the talk relays) more of that complexity is in hidden cores running operating systems you can't see. Indeed, the talk addresses you (yes, you!) more or less directly at ~31:47: "This is called 'ignorance.'" While you are not entirely to blame for your ignorance (if the talk errs, it's in pretending that these hidden systems are well documented -- or even documented at all), you are complicit it in. Please educate yourself as to the massive complexity that lies beneath you, perhaps by just watching the talk that you are commenting on.
choeger 5y ago
While I totally agree with your post (especially the point that a course where you implement your own OS for real hardware is great), I would like to add that USB is a massive foundation/abstraction layer. Once you got USB going for HID, you are much closer to USB for storage, USB for networking, etc. So it certainly is troublesome from an educational perspective but the complexity is where it belongs.
- xnormal 5y ago
  Implementing the minimal “bootstrap” HID interface can be done without most of the abstraction layers. It’s effectively hard-wired and barely parsing of any descriptors or anything.
DylanSp 5y ago
Unrelated to the article, but your description of the 15-410 course piqued my interest. Is there a way to access previous versions of the course materials? I took a look at https://www.cs.cmu.edu/~410/ https://www.cs.cmu.edu/~410/, but it seems like some parts of the site are still being worked on for fall 2021.
- rrss 5y ago
  I found that some of the documents from the 2006 version of the course are available here: https://www.cs.cmu.edu/~410-s06/projects.html https://www.cs.cmu.edu/~410-s06/projects.html
surajrmal 5y ago
Complexity increases over time but it's not for nought. USB, while complicated is a great deal simpler than dealing with a dozen ill documented buses. Additionally the amount of things you can do with it are much greater than what you could do with ps/2. The table stakes in general for operating systems is much higher which means making a new one from scratch is impractical for most folks but this is true for most things. Implementing a CPU that's competitive with the ones Intel it ARM is very challenging for a college student but I'm not sure I see that as a problem. Projects like RISC-V exist to give you a baseline to start from for CPU design just as Linux does got os design and they give you a fighting chance to compete. That's probably for the best, and starting from scratch, while sometimes useful, isn't really necessary for most.
alexgartrell 5y ago
Admittedly, I got a B in 410 and only managed to TA 213, but I do work in the space now. Yes, it’s hard to do research if you start at “let’s design a kernel from scratch,” but you’d never need to do that. You can just hack Linux or a bsd, or even use something like bpf to extend it. The thing that annoyed me about 410 is that when I switched to Linux I realized that pusha/popa didn’t matter at all. It was a good course for writing reentrant C and learning the very basics of hardware, but the really hard stuff is in the weird dynamics of memory management on NUMA systems and work conserving io, which you can’t get anywhere near if you are starting from scratch.
- naasking 5y ago
  > but you’d never need to do that. Why not? Linux has plenty of core design flaws that can't be solved without starting from scratch.
  - deleted 5y ago
    [deleted]
  - wallscratch 5y ago
    I’d be curious to hear some examples.
    - naasking 5y ago
      As but one example, how are you going to conduct security research using an microkernel or exokernel architecture on Linux?
  - bluGill 5y ago
    Then start with something else. There are dozens of operating systems out that there make an attempt to be better (for some definition of better), choose one and work with it. Starting from scratch means you spend years doing all the things they all get right before you can work on the interesting parts.
- seoaeu 5y ago
Const-me 5y ago
> the complexity of the codebase to talk to the USB device was around the similar level of complexity to the entire preemptive multitasking operating system the students were building For the complete USB stack, that's very likely. However, for minimal code needed to read keys from keyboard USB shouldn't be that bad. These students only need USB 1.1. They don't need to support bulk transfers (USB protocol has 2 distinct transports, one low latency another one high throughput). They don't need to support USB hubs between computer and the keyboard. The only device type they need is HID. USB consortium wouldn't allow to call that protocol USB due to the missing features, but it's good enough for education purposes.
- Cthulhu_ 5y ago
  It's still a LOT more to think about and that they have to filter out when developing it though.
  - Const-me 5y ago
    The professor could implement these boring lower-level pieces in a library which fires couple callbacks, one on connect/disconnect to provide two descriptors for HID device and their reports, another one for handling the HID report packets. BTW, USB specifies a small subset of HID interface called “boot interface”. It’s normally consumed by UEFI firmware who needs mouse and keyboard for the setup GUI which runs before OS launches. The subset only supports basic keyboards and mice.
    - bluGill 5y ago
      Then it isn't write your own OS from scratch. The constraints of the library will force your implementation around it, even if you think you have a better idea.
      - Const-me 5y ago
        This project https://github.com/Const-me/Vrmac https://github.com/Const-me/Vrmac implements unified keyboard+mouse input API over 3 distinct lower-level APIs: Win32 messages, XCB packets, and Linux raw input. Here’s for raw input: https://github.com/Const-me/Vrmac/tree/master/Vrmac/Input/Linux https://github.com/Const-me/Vrmac/tree/master/Vrmac/Input/Li... All 3 have unique quirks, yet I would not say they significantly affected the rest of the library. I think for the USB in that OS, a C API with intentionally very narrow scope would be OK for the job. The main reason why the real-life code is so complex is that scope being very wide. We have USB 2 and 3, mass storage, two-way audio, cameras and GPUs, hubs and composite devices, numerous wireless protocols on top, OTG, power delivery, power saving features, and more.

darksaints 5y ago

I'd argue that implementing a USB driver would probably be far more educational, in a practical sense, than implementing a kernel. The number of students who will go on to work on core kernels is vanishingly small, but driver work is never going to end. Plus, if we ever are to transition to the world of microkernels (something that OS researchers have been pushing for decades now), we're gonna need a massive amount of people writing drivers just to get us to the point where things are barely functional.

toast0 5y ago
It's not just a USB driver though. You've got to support PCI (or something) to find the USB controller, then you've got to fiddle with that, then you have to discover the devices, then you have to interface with them. Reading from a keyboard is much simpler. Probably makes more sense to run the computer headless with a serial terminal and make the students poke at a UART. PC standard UARTs are pretty easy, and I think it's easier to find a motherboard with a COM1 header than a PS/2 port (at least my latest hardware matches that description).
- userbinator 5y ago
  You've got to support PCI (or something) to find the USB controller PCI is nothing in comparison to the complexity of USB, especially since the BIOS will usually have set up the I/O ranges for the devices already. It's literally a few dozen bytes of machine code to scan the PCI(e) bus for the device you want.
  - toast0 5y ago
    Oh sure, PCI isn't too bad, but it's way more work than inb(0x60) or inb(0x3f8). (although to be fair, you probably want to setup an interrupt, which is real work)
glangdale 5y ago
No. Driver work is painful but it mostly teaches you about the h/w in question; you don't learn deep principles from a lot of it. You just grind away, spec in hand, until stuff works. A lot of this winds up being "one damn thing after another". There's a good deal of craft to it, and I don't mean to deprecate it, but it's not broadly applicable. Conversely, the work they did in kernels has a huge amount of transfer to thinking about concurrency, which is truly valuable - and deep. It also meant that they acquired a far less magical idea of where things like processes and threads come from, having actually built the mechanisms to make them happen (you build a thread library in a warm-up project). As for microkernels - they often punt a lot of the hard concurrency stuff up to user space servers, which will wind up needing exactly the kind of concurrency that the students learned to build.
- deleted 5y ago
  [deleted]
- naasking 5y ago
  > No. Driver work is painful but it mostly teaches you about the h/w in question; you don't learn deep principles from a lot of it. You just grind away, spec in hand, until stuff works. A lot of this winds up being "one damn thing after another". There's a good deal of craft to it, and I don't mean to deprecate it, but it's not broadly applicable. I agree you don't learn deep principles from it, but I disagree that it's not broadly applicable based on your description. Grinding away with spec in hand, "one damn thing after another" sounds exactly like most programming people are likely to encounter in their career.
  - simiones 5y ago
    Sure, but you're not learning anything per se - perhaps some discipline. To some approximation, any two decently competent programmers will take similar amounts of time writing a device driver for a complex novel device regardless of previous experience - the vast majority of the job is in simply going through the spec.
    - 5y ago
Ericson2314 5y ago
I suppose it would be useful if it taught students how much this industry is hare rather than tortoise nonsense where the interfaces are sheer accidents, and the implementations are mad-dash to keep up. http://langsec.org/ http://langsec.org/ But what you propose is preprofesional drival. It's bad enough that education is subsidized for employers already, we don't need to stoop to them further.

0xcde4c3db 5y ago

It seems like the done thing these days (for people who care about running on real hardware) is to bring up educational/experimental kernels on Raspberry Pi, using the "mini UART" for console I/O and the SD interface for storage.

User23 5y ago

> I was TA'ing CMU's 15-410 operating systems course (a great course - build a preemptive multitasking operating system from scratch) That class bordered on legendary. Awesome that you got to TA it. I still have an irrational fondness for AFS.

glangdale 5y ago
It was a fantastic experience, although TA'ing it was daunting. Like the apocryphal Ledru-Rollin quote: "There go the people. I must follow them, for I am their leader", it was frankly hard to keep up with all the different approaches students conjured up. I believe that the course number changed at some point - or some shallow aspect of the course was set to change - and a bunch of industry people called the university to express concern. It is/was (I can't speak for the current iteration of the course, although I have no indication that quality has slipper or anything) truly transformative. I offered to run a 'franchise' of the course at the University of Sydney a while back and was informed that anything quite that transformative wasn't really an option; our job was at least in part to pass engineering students who didn't care that much about computing.
- josephg 5y ago
  > I offered to run a 'franchise' of the course at the University of Sydney a while back and was informed that anything quite that transformative wasn't really an option Thats very disappointing. You might have more luck joining the University of New South Wales OS team. (For context, UNSW is a local rival of USYD). When I was a student there, the OS course was legendary. I only did the "basic" OS course. Our assignments led us to implement a bunch of syscalls for handling file IO, and write a page fault handler for a toy operating system. I'm not sure what they do in the advanced OS course - but it has a reputation for a reason. And it looks like[1] its still run by Gernot Heiser, who's a legend. He's the brain behind SeL4 - which is the world's first (and only?) formally verified OS kernel. I'm kicking myself for not doing his advanced OS course while I was a student. [1] http://www.cse.unsw.edu.au/~cs9242/current/ http://www.cse.unsw.edu.au/~cs9242/current/

userbinator 5y ago

PS/2 interfaces are actually still quite common for "enthusiast" motherboards, because of the low latency (the simplicity compared to USB certainly has a lot to do with it.) Laptops also tend to have PS/2 keyboards and a "mouse" (the trackpad, which can be switched to a proprietary protocol later for more features) because the EC they use provides that "for free" and that industry tends to have a "it works, why change it and break things" attitude.[1] https://news.ycombinator.com/item?id=17692499 https://news.ycombinator.com/item?id=17692499 That said, BIOSes can also emulate PS/2 with USB (no doubt via SMM). This is usually a setting called "legacy keyboard/mouse support". [1] The notable exception being Apple. Think Different. https://news.ycombinator.com/item?id=12924051 https://news.ycombinator.com/item?id=12924051

Bigpet 5y ago
I think there was a Ben Eater video debunking the "PS/2 is faster than USB" myth. That used to be true, but the PS/2 is so much slower (lower frequency) that by the time a package containing the mouse or keyboard info is finished transmitting the usb port started and finished a poll
- bear8642 5y ago
  think might be at end of this one <https://www.youtube.com/watch?v=7aXbh9VUB3U https://www.youtube.com/watch?v=7aXbh9VUB3U> though there's a few he's done recently on USB spec
- dmz73 5y ago
  USB is faster and it is better when there is nothing saturating the USB with traffic and CPU can keep up. Most of the time there will be no noticeable difference using the USB mouse or PS/2 mouse. But if CPU is busy and USB bus is saturated, USB mouse will start to be less responsive than PS/2 mouse. In the extreme cases even USB keyboard will become unresponsive where the old PS/2 keyboard still works fine. This can make a difference between being able to terminate a misbehaving process and resume working or having to power-cycle the computer.
  - 5y ago

hoseja 5y ago

I have a modern consumer motherboard that has intermittent problems connecting all it's USB ports, with the latest "stable" BIOS supposedly fixing this but also reportedly having worse performance.

throwawaylinux 5y ago

> A perennial complaint; Rob Pike had a trollier but similar version years back. I'm not nearly as optimistic that we have the resources to cope with the monstrous complexity of hardware now. "Hardware" (read: device drivers) is not notably complicated IMO. That's not to say they're simple, but driving devices from a software point of view is pretty similar to interacting with other software. You read the spec (APIs), write code to set up data structures or registers a certain way, parse responses, etc. Writing a modern full fledged USB stack is very complex, but would not be more complex than writing a modern full TCP/IP stack for example. A lot of device programming models have gotten simpler too. Device drivers used to be notorious deep voodoo magic e.g., in cases of the IDE disk driver, but that was not really "complexity" of the software logic so much as hundreds of special cases and deviations from standards or odd behavior in all manner of IDE controllers and devices. To the point where the real wizards were the ones who had access to internal data sheets, errata, or reverse engineered firmware from these devices, or otherwise spent countless hours poking at things and reconstructing quirks for themselves, probably bricking a lot of silicon and spinning rust in the process. But systems and devices are now getting to the point where things don't work that way anymore, silicon power is so cheap and firmware is on everything. The NVMe device for example sets up pretty simple packet-type command queues and sends requests and receives responses for operations - query device information, perform a read or write, etc). It's not quite that simple, there are some quirks and details, but it's quite a lot like writing a client for a server (HTTP, perhaps). I think other device interfaces will evolve toward cleaner simpler models like this too. One exception to this is GPUs of course. The other thing is with moore's law continuing to slow down there will be increased incentive to move more processing out to devices. It's long been happening with high end network devices, but with technologies like CXL.cache coming soon, I would expect that trend to keep ramping up and come to other accelerators (crypto, AI, even disk). So.. it's a mix. Things are definitely getting more complex, but AFAIKS the complexity of interacting with hardware is not increasing at a much greater rate than the complexity of interacting with other software. And, as always this complexity is made possible by abstractions and layers and interfaces. It's not clearly exceeding our ability to cope with it.

bcantrill 5y ago
This is entirely wrong, and I'm sorry that those who do work at the hardware/software interface have done their work apparently so well as to allow you to harbor this delusion. Hardware is complex -- exceedingly so -- but (just as the talk relays) more of that complexity is in hidden cores running operating systems you can't see. Indeed, the talk addresses you (yes, you!) more or less directly at ~31:47: "This is called 'ignorance.'" While you are not entirely to blame for your ignorance (if the talk errs, it's in pretending that these hidden systems are well documented -- or even documented at all), you are complicit it in. Please educate yourself as to the massive complexity that lies beneath you, perhaps by just watching the talk that you are commenting on.

choeger 5y ago

While I totally agree with your post (especially the point that a course where you implement your own OS for real hardware is great), I would like to add that USB is a massive foundation/abstraction layer. Once you got USB going for HID, you are much closer to USB for storage, USB for networking, etc. So it certainly is troublesome from an educational perspective but the complexity is where it belongs.

xnormal 5y ago
Implementing the minimal “bootstrap” HID interface can be done without most of the abstraction layers. It’s effectively hard-wired and barely parsing of any descriptors or anything.

DylanSp 5y ago

Unrelated to the article, but your description of the 15-410 course piqued my interest. Is there a way to access previous versions of the course materials? I took a look at https://www.cs.cmu.edu/~410/ https://www.cs.cmu.edu/~410/, but it seems like some parts of the site are still being worked on for fall 2021.

rrss 5y ago
I found that some of the documents from the 2006 version of the course are available here: https://www.cs.cmu.edu/~410-s06/projects.html https://www.cs.cmu.edu/~410-s06/projects.html

surajrmal 5y ago

Complexity increases over time but it's not for nought. USB, while complicated is a great deal simpler than dealing with a dozen ill documented buses. Additionally the amount of things you can do with it are much greater than what you could do with ps/2. The table stakes in general for operating systems is much higher which means making a new one from scratch is impractical for most folks but this is true for most things. Implementing a CPU that's competitive with the ones Intel it ARM is very challenging for a college student but I'm not sure I see that as a problem. Projects like RISC-V exist to give you a baseline to start from for CPU design just as Linux does got os design and they give you a fighting chance to compete. That's probably for the best, and starting from scratch, while sometimes useful, isn't really necessary for most.

alexgartrell 5y ago

Admittedly, I got a B in 410 and only managed to TA 213, but I do work in the space now. Yes, it’s hard to do research if you start at “let’s design a kernel from scratch,” but you’d never need to do that. You can just hack Linux or a bsd, or even use something like bpf to extend it. The thing that annoyed me about 410 is that when I switched to Linux I realized that pusha/popa didn’t matter at all. It was a good course for writing reentrant C and learning the very basics of hardware, but the really hard stuff is in the weird dynamics of memory management on NUMA systems and work conserving io, which you can’t get anywhere near if you are starting from scratch.

naasking 5y ago
> but you’d never need to do that. Why not? Linux has plenty of core design flaws that can't be solved without starting from scratch.
- deleted 5y ago
  [deleted]
- wallscratch 5y ago
  I’d be curious to hear some examples.
  - naasking 5y ago
    As but one example, how are you going to conduct security research using an microkernel or exokernel architecture on Linux?
- bluGill 5y ago
  Then start with something else. There are dozens of operating systems out that there make an attempt to be better (for some definition of better), choose one and work with it. Starting from scratch means you spend years doing all the things they all get right before you can work on the interesting parts.
seoaeu 5y ago

Const-me 5y ago

> the complexity of the codebase to talk to the USB device was around the similar level of complexity to the entire preemptive multitasking operating system the students were building For the complete USB stack, that's very likely. However, for minimal code needed to read keys from keyboard USB shouldn't be that bad. These students only need USB 1.1. They don't need to support bulk transfers (USB protocol has 2 distinct transports, one low latency another one high throughput). They don't need to support USB hubs between computer and the keyboard. The only device type they need is HID. USB consortium wouldn't allow to call that protocol USB due to the missing features, but it's good enough for education purposes.

Cthulhu_ 5y ago
It's still a LOT more to think about and that they have to filter out when developing it though.
- Const-me 5y ago
  The professor could implement these boring lower-level pieces in a library which fires couple callbacks, one on connect/disconnect to provide two descriptors for HID device and their reports, another one for handling the HID report packets. BTW, USB specifies a small subset of HID interface called “boot interface”. It’s normally consumed by UEFI firmware who needs mouse and keyboard for the setup GUI which runs before OS launches. The subset only supports basic keyboards and mice.
  - bluGill 5y ago
    Then it isn't write your own OS from scratch. The constraints of the library will force your implementation around it, even if you think you have a better idea.
    - Const-me 5y ago
      This project https://github.com/Const-me/Vrmac https://github.com/Const-me/Vrmac implements unified keyboard+mouse input API over 3 distinct lower-level APIs: Win32 messages, XCB packets, and Linux raw input. Here’s for raw input: https://github.com/Const-me/Vrmac/tree/master/Vrmac/Input/Linux https://github.com/Const-me/Vrmac/tree/master/Vrmac/Input/Li... All 3 have unique quirks, yet I would not say they significantly affected the rest of the library. I think for the USB in that OS, a C API with intentionally very narrow scope would be OK for the job. The main reason why the real-life code is so complex is that scope being very wide. We have USB 2 and 3, mass storage, two-way audio, cameras and GPUs, hubs and composite devices, numerous wireless protocols on top, OTG, power delivery, power saving features, and more.