10 ms·

It's Time for Operating Systems to Rediscover Hardware

188 points by matt_d 5y ago

SLWW 5y ago
TRUTH
klyrs 5y ago
[video] please
- IncRnd 5y ago
  The video is at the bottom of the page, once you turn on third party frames (or just youtube frames).
  - klyrs 5y ago
    The page consists of an abstract for the video, and not a transcript. Ergo, I'm asking that the story be tagged [video]
glangdale 5y ago
A perennial complaint; Rob Pike had a trollier but similar version years back. I'm not nearly as optimistic that we have the resources to cope with the monstrous complexity of hardware now. An anecdote I like to share is that while I was TA'ing CMU's 15-410 operating systems course (a great course - build a preemptive multitasking operating system from scratch) was that the student projects could run on real - if old - hardware. There was a PC that could boot your stuff if you put it on a disk. This PC had to have PS/2 interface to the keyboard, though. A newer PC would be all USB. Apparently, the complexity of the codebase to talk to the USB device was around the similar level of complexity to the entire preemptive multitasking operating system the students were building (and, of course, considerably less educational). I mean, this wasn't a full-functioned OS, but it allowed multitasking and preemption of kernel calls, so seriously non-trivial. Multiply this story by all the devices a reasonable OS would be expected to talk to and it's a scary prospect for the OS researcher... and if you don't have those devices up and running, good luck supporting most workloads anyone cares about.
- darksaints 5y ago
  I'd argue that implementing a USB driver would probably be far more educational, in a practical sense, than implementing a kernel. The number of students who will go on to work on core kernels is vanishingly small, but driver work is never going to end. Plus, if we ever are to transition to the world of microkernels (something that OS researchers have been pushing for decades now), we're gonna need a massive amount of people writing drivers just to get us to the point where things are barely functional.
  - toast0 5y ago
    It's not just a USB driver though. You've got to support PCI (or something) to find the USB controller, then you've got to fiddle with that, then you have to discover the devices, then you have to interface with them. Reading from a keyboard is much simpler. Probably makes more sense to run the computer headless with a serial terminal and make the students poke at a UART. PC standard UARTs are pretty easy, and I think it's easier to find a motherboard with a COM1 header than a PS/2 port (at least my latest hardware matches that description).
    - userbinator 5y ago
      You've got to support PCI (or something) to find the USB controller PCI is nothing in comparison to the complexity of USB, especially since the BIOS will usually have set up the I/O ranges for the devices already. It's literally a few dozen bytes of machine code to scan the PCI(e) bus for the device you want.
      - toast0 5y ago
        Oh sure, PCI isn't too bad, but it's way more work than inb(0x60) or inb(0x3f8). (although to be fair, you probably want to setup an interrupt, which is real work)
  - glangdale 5y ago
    No. Driver work is painful but it mostly teaches you about the h/w in question; you don't learn deep principles from a lot of it. You just grind away, spec in hand, until stuff works. A lot of this winds up being "one damn thing after another". There's a good deal of craft to it, and I don't mean to deprecate it, but it's not broadly applicable. Conversely, the work they did in kernels has a huge amount of transfer to thinking about concurrency, which is truly valuable - and deep. It also meant that they acquired a far less magical idea of where things like processes and threads come from, having actually built the mechanisms to make them happen (you build a thread library in a warm-up project). As for microkernels - they often punt a lot of the hard concurrency stuff up to user space servers, which will wind up needing exactly the kind of concurrency that the students learned to build.
ksec 5y ago
Off Topic. Why does it need a capital R in the Rediscover? ( Along with T in Time ) All I can see when I reading it was Redis something Hardware. Edit: And it turns out the title in video was actually without capital T and R.
- deleted 5y ago
  [deleted]
- Arnavion 5y ago
  https://en.m.wikipedia.org/wiki/Title_case https://en.m.wikipedia.org/wiki/Title_case
nathanaldensr 5y ago
As a regular-old application developer, I had no idea that operating systems were being so aggressively sequestered from the perspective of computer architecture. It seems to be quite a joke, actually, to call Linux an "operating system." Combine this awareness with how much closed-source code is running these chips without any knowledge by the operating system and thus the user... it seems pretty scary. I suspect much of the problem is "wall of confusion" differences: operating systems are primarily concerned with compatibility and reliability while computer architecture seems to be about rapid innovation and change.
- vlovich123 5y ago
  So this has actually always been the case. Hardware firmware blobs that initialize the device to talk on the bus yielded to EEPROM chips that had SW firmware burned into it to do that initialization as the EEPROM chips became cheaper and let the bringup process be more flexible. This yielded to microcontrollers that had programmable flash (still some EEPROM for some parts of HW, but now you can glue multiple pieces together). Now you have full-blown ARM cores doing that orchestration (but HW still has EEPROM for bug fixes on top of the base NOR-flashed code). It’s all about flexibility and cost. I don’t know though if I fully agree with the characterization that Linux has been shuttled off to the corner as chip vendors work around it. It’s more that the computer has actually always been a distributed system and OS/hardware developers have ignored that for a very long time. You wouldn’t say that Linux isn’t the OS for distributed clusters even though a lot of work goes into making those clusters run and that all typically runs in user space outside the OS boundary. It’s possible that there’s a better model out there that manages to unify things but I’m skeptical. All this code runs on different chips with different clock speeds. Code also resides in drastically different memory spaces and that’s unavoidable - an M3 might be able to load the firmware over PCIE, but it’s not executing out of main memory. Additionally these chips typically have drastically different cache coherency properties. Now maybe we need to revisit this all holistically, which I think is the actual pitch being made and one I can support. That needs to look like defining interfaces for how this stuff works (& yes, making it possible to integrate into Linux) and getting chip vendors to adopt. Sunk costs are real and trying to rework not just HW architecture but also the entire SW ecosystem can be a dead end. I’d certainly be keenly interested to hear the speaker’s ideas. He’s certainly far more knowledgeable about this space than I am.
new_realist 5y ago
Does modern hardware require OS-mediated isolation? If not, why should the OS get in the way of apps using the hardware directly?
- neltnerb 5y ago
  I certainly write plenty of code where I write to registers directly to configure hardware modules and don't think I'm using an OS in any way. I assume most of the security design and hardware permissions are handled at a higher level than I usually write code.
- mikewarot 5y ago
  The function of the operating system is to safely multiplex the hardware. If the OS doesn't control the hardware, it can't provide safety. If Linux is running inside a sandbox, it can't secure anything.
  - pjmlp 5y ago
    "an operating system is a collection of things that don't fit inside a language; there shouldn't be one" -- Dan Ingalls
- SilverRed 5y ago
  No one wants apps to directly interface with hardware. Imagine loading up a notes app and it directly trys to interface with your hard drive using its own built in FS driver and a bug in that causes your whole drive to be wiped. Or that you open two programs at once and they try to access the same device at once causing some kind of error and likely data loss or system instability. I don't even want apps to directly have access to the file level interface of my storage but rather an extra abstraction which lets me pick which files it can access.
  - SpaceL10n 5y ago
    > No one wants apps to directly interface with hardware. I build computer kiosks and I agree wholeheartedly. So many devices/peripherals require some sort of custom implementation. Most have some sort of SDK, but not all. Better hope that SDK comes in your tech stack's language(s) of choice. Why isn't it just data in / data out? Even printers and scanners are a pain to deal with. Sure, Windows (and TWAIN) abstracts some of the printer/scanner communication, but it all seems forgotten. Like, why is using new hardware in my software so difficult and how come I can't just implement new hardware the same way every time? To me, it's all just data in/data out from my language runtime to the device. I couldn't care less about the implementation details in between.
phendrenad2 5y ago
Unfortunately it isn't that simple. One of the comments on the youtube video gets to the heart of the matter: > I don't see how this might be changed. Most of those little red circles are not just "tiny computers" but also different implementers in different companies, all trying to protect their little fiefdom against their customers There are of course initiatives like libreboot/freeboot/whatever, and efforts to reverse-engineer the firmware of microcontrollers in peripherals, and there exists fully documented hardware such as the Raspberry Pi 4 (I think?), but even with full documentation, it's incredibly hard to write custom firmware, and the benefit is very slim (better security perhaps?)
- jillesvangurp 5y ago
  That actually goes to the heart of the problem, modern SOCs are basically proprietary components gobbled together on a chip with some internal networking/wiring that each come with their own software that does a lot of stuff before even getting around to booting Linux. The problem is the proprietary nature of this stuff and the fact that a lot of it is outside of the scope of scrutiny that goes into Linux. It's held together with what basically amounts to glorified duct tape. It's complicated. It has weird failure modes and occasionally this stuff has expensive issues. Like for example security issues. Very relevant if you want to build an open source phone platform or laptop. Not impossible; but it requires addressing a few things. Like connecting to a 5G network without relying on Qualcomm's proprietary software and hardware. The real solution is not merely integrating these things as is and trying to reverse engineer them but engineer them from the ground up to work together and make sense together. That larger, open operating system does not really exist. And many proprietary equivalents on the market right now have an emergent/accidental design rather than something that was designed from the ground up.
- opan 5y ago
  I always thought the Raspberry Pis were blob-ridden black boxes due to all the Broadcom bits.
- Const-me 5y ago
  > it's incredibly hard to write custom firmware I remember how it was with programmable GPUs. When programmable shaders were introduced in GeForce 3 (2001), nobody really understood what to do with them. The consensus was that it’s incredibly hard to write custom shaders. OpenGL consortium refused to support shaders for almost a decade (GLSL 1.0 released in 2008), that’s one of the reasons of the success of Direct3D https://softwareengineering.stackexchange.com/a/88055 https://softwareengineering.stackexchange.com/a/88055 Yet look at them after 20 years of evolution. People no longer coding shaders in assembly, we have quite a few higher-level languages for them. People have built better APIs to interface with programmable GPUs, both low level close to metal (D3D12, Metal, Vulkan) and high-level (CUDA, TensorFlow, game engines). People built debuggers and profilers for these shaders. Overall, people embraced programmable GPUs, leaned to use them, and the results are amazing. Visual quality of games and other real-time 3D skyrocketed, despite the increase of pixel count and density in the displays. I can see how the same path is possible for the rest of the programmable chips and pieces inside modern computers. > and the benefit is very slim Better products in very broad sense. For instance, when idle, a smartphone OS could power off not just the screen, but also CPU, RAM, and a few other components, and downscale itself to the slowest and the most power efficient core in the system.
ironman1478 5y ago
I've worked on firmware for a few SoC's that my company has designed or purchased and we use embedded linux on some. I think his observation about how Linux doesn't abstract the hardware accurately is totally right. The system isn't that complicated, but implementing everything within the world of Linux has been painful imo, so for future implementations we're evaluating RTOS's. The main reason people use Linux on these custom SoCs because of driver availability of key components (ethernet phys and persistent storage), large library of tools, and probably $ (who wants to pay for QNX). Those are good reasons, but if you have to write a large volume of code, it doesn't really feel like it's providing too much. It definitely felt like we were just using a hammer and seeing everything as a nail. Those are my 2 cents.
- elcritch 5y ago
  This issue with needing more flexibility on SoC's makes me wonder if its part of why the Linux Foundation has put a lot of work and resources into Zephyr RTOS [1]. Its gaining a lot of popularity and incorporates some of nice elements of linux like device tree syntax (DTS) files [2]. I could readily imagine using Linux & Zephyr on a SoC, and using shared DTS to describe the SoC architecture. Actually, come to think of it extending DTS syntax to describe the pointer swizzling the OP video mentions. Then both OO'es could derive correct physical address handling. 1: https://www.linuxfoundation.org/blog/the-power-of-zephyr-rtos/ https://www.linuxfoundation.org/blog/the-power-of-zephyr-rto... 2: https://docs.zephyrproject.org/latest/guides/dts/index.html https://docs.zephyrproject.org/latest/guides/dts/index.html
  - ironman1478 5y ago
    So, it's funny. It's not uncommon to run RTOSs and Linux on the same CPU, where the rtos coordinates tasks between the components that have real time requirements and the Linux side does stuff like stream data from a buffer that the rtos writes to. We had this architecture on some systems and the amount of cludging to get this all working was annoying to say the least. We specifically had issues are figuring out how to partition the memory regions for each OS and it wasn't trivial to validate. I wasn't around when the decisions at my employer when the decisions made around OS options, but this would've been a viable one if it was mature enough at that time. Thanks for the info!
fithisux 5y ago
Hardware should become more clever in order to take burden from drivers.
xfer 5y ago
It is a interesting debate to have but this is unlikely to happen for consumer hardware, since no OEM wants you to have that kind of access(talking of those phone SoCs), they will just give you a slice of ccNUMA to protect their "IP". That qualcomm bug that he showed is unlikely to go away for consumers. There is a message to build your own computers and that's probably what people will do and it could be interesting research for custom servers and future heterogeneous architectures. I suspect people in this industry are already doing it.
Ericson2314 5y ago
Conway's law at work! One one hand there is great vertical disintegration, on the other hand, even Apple is too lazy to actually do this right, still treating their software stack as a black box where breaking changes are fine but elegance is lost on the suits. I suppose the folks at https://oxide.computer/ https://oxide.computer/ are smirking if they see this video.
- bcantrill 5y ago
  It's very vindicating of our worldview, yes -- and also vindicates the de novo operating system (Rust kernel, Rust tasks) that we have developed for some of the use cases that Roscoe outlines. Looking forward to getting it all out there!
95014_refugee 5y ago
Having seen the coalface, the problem is not the OS, it's the (stupendously retarded) application model. There's very little (interesting) you can do when application developers are (for the most part) incapable and unwilling to embrace anything even remotely different from the DOS application model. The OS is, at its heart, a mapping from that application model onto the hardware. If a hardware feature or characteristic can't be expressed / surfaced within the application model, then there's little to no value in attempting to exploit it. Someone like  at least has internal customers, and a vertically-integrated embedded system can sometimes do better, but general-purpose systems are shackled to some minor variation on the POSIX application model for the forseeable future.
evilos 5y ago
Interesting talk but I'm not convinced of the value proposition of "rediscovering hardware". The speaker points out that there are lots of small processors inside a modern SoC that are often running their own software hidden from Linux. I agree that there are security implications here a la Qualcomm Hexagon vulnerabilities or Intel Management Engine backdoors but I'm not sure what the speaker's proposed alternative is. Is it that a "true" OS would directly execute the duties of these hidden processors? This is extra work that would make systems slower. Having dedicated hardware handle these duties is an optimization (hardware acceleration). Alternatively you could replace these small processors with fixed function hardware. This would make any system flaws permanent although it would be much harder to exploit any vulnerabilities. Does the lack of hidden processors make the system "better" somehow? (aside from potentially more secure) Are bus masters and hardware accelerators part of the OS? It would certainly be a more beautiful system that I would love to see, but I need help justifying the engineering cost. I hope to see some open hardware passion projects along these lines, but I doubt they will ever be mainstream.
- SilverRed 5y ago
  I'm increasingly seeing the value of Apple style products where almost every part of the device from the hardware, to firmware, to OS, to UI is all from the same company. The whole combined experience just makes things run so smoothly and allows security issues to be patched since Apple has the resources that maintain each part internally.
  - pas 5y ago
    Yeah, of course that works, for a while at least. Also developed in the open, patches are welcome, you can update your own NIC/GPU/Wifi-dongle components would work too. Unfortunately the whole industry (Apple as one of the main culprits) is quite hostile to anything like that. Though maybe we'll have a RedHat of hardware someday.
  - darkwater 5y ago
    Yeah, and as a side-effect you have to obey your one true lord.
OneTimePetes 5y ago
Oh, oh the ragnarok of systems programming, moores plateau stretches on, parallel is coming to a limit and now its goodbye abstractions & no-skill-lib-glue-code, hello 80s hardware hacks and efficient programming again? Will OSes be produced like games, were they start with a "reuseable" oo-engine and then that is broken down into data-orientated design. I thought we had more time.. Cant we global warm postpone this? Let the next generation sort it all out?
tomovo 5y ago
Perhaps related: The 30 million line problem (Casey Muratori): https://www.youtube.com/watch?v=kZRE7HIO3vk https://www.youtube.com/watch?v=kZRE7HIO3vk Among other things, advocating for more stable & open standard HW interfaces, rather than OS-level APIs.
ncmncm 5y ago
The real progress nowadays, since OSes have not stepped up, is in Kernel Bypass. (Show me a fancy new abstraction, and my reaction is, "Great—how do I bypass it?") We have para-kernels, demi-kernels, uni-kernels, all to get closer to the hardware. Probably future machines should have lots of FPGAs, and easy access to claim and program them. We will need good languages for the latter, although probably C++ will end up being that.
jdefr89 5y ago
I am confused. He is basically saying Linux isn't powering all the hardware simply because firmware is running on other more complex peripherals/SoCs on a device. That will always become the case as hardware gets more complicated and advanced. Linux simply need to provide the means to abstract the nitty gritty. Linux is still acting as the central "manager" of all the hardware. As far as security. Linux can act as the final security boundary for getting the machine to do anything useful.
xnormal 5y ago
Academics can sometimes learn a lot from industry. The ScyllaDB database and its underlying SeaStar library essentially implement a userspace OS that handles the hardware very differently than a typical modern OS with phenomenal improvements in performance. Historically, it started as a unikernel called OSv designed to run directly on hypervisors. But they eventually realized that it can run just as well on linux with kernel bypass working and io_uring and is easier to deploy.

188 points by matt_d 5y ago

SLWW 5y ago

TRUTH

klyrs 5y ago

[video] please

IncRnd 5y ago
The video is at the bottom of the page, once you turn on third party frames (or just youtube frames).
- klyrs 5y ago
  The page consists of an abstract for the video, and not a transcript. Ergo, I'm asking that the story be tagged [video]

glangdale 5y ago

A perennial complaint; Rob Pike had a trollier but similar version years back. I'm not nearly as optimistic that we have the resources to cope with the monstrous complexity of hardware now. An anecdote I like to share is that while I was TA'ing CMU's 15-410 operating systems course (a great course - build a preemptive multitasking operating system from scratch) was that the student projects could run on real - if old - hardware. There was a PC that could boot your stuff if you put it on a disk. This PC had to have PS/2 interface to the keyboard, though. A newer PC would be all USB. Apparently, the complexity of the codebase to talk to the USB device was around the similar level of complexity to the entire preemptive multitasking operating system the students were building (and, of course, considerably less educational). I mean, this wasn't a full-functioned OS, but it allowed multitasking and preemption of kernel calls, so seriously non-trivial. Multiply this story by all the devices a reasonable OS would be expected to talk to and it's a scary prospect for the OS researcher... and if you don't have those devices up and running, good luck supporting most workloads anyone cares about.

darksaints 5y ago
I'd argue that implementing a USB driver would probably be far more educational, in a practical sense, than implementing a kernel. The number of students who will go on to work on core kernels is vanishingly small, but driver work is never going to end. Plus, if we ever are to transition to the world of microkernels (something that OS researchers have been pushing for decades now), we're gonna need a massive amount of people writing drivers just to get us to the point where things are barely functional.
- toast0 5y ago
  It's not just a USB driver though. You've got to support PCI (or something) to find the USB controller, then you've got to fiddle with that, then you have to discover the devices, then you have to interface with them. Reading from a keyboard is much simpler. Probably makes more sense to run the computer headless with a serial terminal and make the students poke at a UART. PC standard UARTs are pretty easy, and I think it's easier to find a motherboard with a COM1 header than a PS/2 port (at least my latest hardware matches that description).
  - userbinator 5y ago
    You've got to support PCI (or something) to find the USB controller PCI is nothing in comparison to the complexity of USB, especially since the BIOS will usually have set up the I/O ranges for the devices already. It's literally a few dozen bytes of machine code to scan the PCI(e) bus for the device you want.
    - toast0 5y ago
      Oh sure, PCI isn't too bad, but it's way more work than inb(0x60) or inb(0x3f8). (although to be fair, you probably want to setup an interrupt, which is real work)
- glangdale 5y ago
  No. Driver work is painful but it mostly teaches you about the h/w in question; you don't learn deep principles from a lot of it. You just grind away, spec in hand, until stuff works. A lot of this winds up being "one damn thing after another". There's a good deal of craft to it, and I don't mean to deprecate it, but it's not broadly applicable. Conversely, the work they did in kernels has a huge amount of transfer to thinking about concurrency, which is truly valuable - and deep. It also meant that they acquired a far less magical idea of where things like processes and threads come from, having actually built the mechanisms to make them happen (you build a thread library in a warm-up project). As for microkernels - they often punt a lot of the hard concurrency stuff up to user space servers, which will wind up needing exactly the kind of concurrency that the students learned to build.

ksec 5y ago

Off Topic. Why does it need a capital R in the Rediscover? ( Along with T in Time ) All I can see when I reading it was Redis something Hardware. Edit: And it turns out the title in video was actually without capital T and R.

deleted 5y ago
[deleted]
Arnavion 5y ago
https://en.m.wikipedia.org/wiki/Title_case https://en.m.wikipedia.org/wiki/Title_case

nathanaldensr 5y ago

As a regular-old application developer, I had no idea that operating systems were being so aggressively sequestered from the perspective of computer architecture. It seems to be quite a joke, actually, to call Linux an "operating system." Combine this awareness with how much closed-source code is running these chips without any knowledge by the operating system and thus the user... it seems pretty scary. I suspect much of the problem is "wall of confusion" differences: operating systems are primarily concerned with compatibility and reliability while computer architecture seems to be about rapid innovation and change.

vlovich123 5y ago
So this has actually always been the case. Hardware firmware blobs that initialize the device to talk on the bus yielded to EEPROM chips that had SW firmware burned into it to do that initialization as the EEPROM chips became cheaper and let the bringup process be more flexible. This yielded to microcontrollers that had programmable flash (still some EEPROM for some parts of HW, but now you can glue multiple pieces together). Now you have full-blown ARM cores doing that orchestration (but HW still has EEPROM for bug fixes on top of the base NOR-flashed code). It’s all about flexibility and cost. I don’t know though if I fully agree with the characterization that Linux has been shuttled off to the corner as chip vendors work around it. It’s more that the computer has actually always been a distributed system and OS/hardware developers have ignored that for a very long time. You wouldn’t say that Linux isn’t the OS for distributed clusters even though a lot of work goes into making those clusters run and that all typically runs in user space outside the OS boundary. It’s possible that there’s a better model out there that manages to unify things but I’m skeptical. All this code runs on different chips with different clock speeds. Code also resides in drastically different memory spaces and that’s unavoidable - an M3 might be able to load the firmware over PCIE, but it’s not executing out of main memory. Additionally these chips typically have drastically different cache coherency properties. Now maybe we need to revisit this all holistically, which I think is the actual pitch being made and one I can support. That needs to look like defining interfaces for how this stuff works (& yes, making it possible to integrate into Linux) and getting chip vendors to adopt. Sunk costs are real and trying to rework not just HW architecture but also the entire SW ecosystem can be a dead end. I’d certainly be keenly interested to hear the speaker’s ideas. He’s certainly far more knowledgeable about this space than I am.

new_realist 5y ago

Does modern hardware require OS-mediated isolation? If not, why should the OS get in the way of apps using the hardware directly?

neltnerb 5y ago
I certainly write plenty of code where I write to registers directly to configure hardware modules and don't think I'm using an OS in any way. I assume most of the security design and hardware permissions are handled at a higher level than I usually write code.
mikewarot 5y ago
The function of the operating system is to safely multiplex the hardware. If the OS doesn't control the hardware, it can't provide safety. If Linux is running inside a sandbox, it can't secure anything.
- pjmlp 5y ago
  "an operating system is a collection of things that don't fit inside a language; there shouldn't be one" -- Dan Ingalls
SilverRed 5y ago
No one wants apps to directly interface with hardware. Imagine loading up a notes app and it directly trys to interface with your hard drive using its own built in FS driver and a bug in that causes your whole drive to be wiped. Or that you open two programs at once and they try to access the same device at once causing some kind of error and likely data loss or system instability. I don't even want apps to directly have access to the file level interface of my storage but rather an extra abstraction which lets me pick which files it can access.
- SpaceL10n 5y ago
  > No one wants apps to directly interface with hardware. I build computer kiosks and I agree wholeheartedly. So many devices/peripherals require some sort of custom implementation. Most have some sort of SDK, but not all. Better hope that SDK comes in your tech stack's language(s) of choice. Why isn't it just data in / data out? Even printers and scanners are a pain to deal with. Sure, Windows (and TWAIN) abstracts some of the printer/scanner communication, but it all seems forgotten. Like, why is using new hardware in my software so difficult and how come I can't just implement new hardware the same way every time? To me, it's all just data in/data out from my language runtime to the device. I couldn't care less about the implementation details in between.

phendrenad2 5y ago

Unfortunately it isn't that simple. One of the comments on the youtube video gets to the heart of the matter: > I don't see how this might be changed. Most of those little red circles are not just "tiny computers" but also different implementers in different companies, all trying to protect their little fiefdom against their customers There are of course initiatives like libreboot/freeboot/whatever, and efforts to reverse-engineer the firmware of microcontrollers in peripherals, and there exists fully documented hardware such as the Raspberry Pi 4 (I think?), but even with full documentation, it's incredibly hard to write custom firmware, and the benefit is very slim (better security perhaps?)

jillesvangurp 5y ago
That actually goes to the heart of the problem, modern SOCs are basically proprietary components gobbled together on a chip with some internal networking/wiring that each come with their own software that does a lot of stuff before even getting around to booting Linux. The problem is the proprietary nature of this stuff and the fact that a lot of it is outside of the scope of scrutiny that goes into Linux. It's held together with what basically amounts to glorified duct tape. It's complicated. It has weird failure modes and occasionally this stuff has expensive issues. Like for example security issues. Very relevant if you want to build an open source phone platform or laptop. Not impossible; but it requires addressing a few things. Like connecting to a 5G network without relying on Qualcomm's proprietary software and hardware. The real solution is not merely integrating these things as is and trying to reverse engineer them but engineer them from the ground up to work together and make sense together. That larger, open operating system does not really exist. And many proprietary equivalents on the market right now have an emergent/accidental design rather than something that was designed from the ground up.
opan 5y ago
I always thought the Raspberry Pis were blob-ridden black boxes due to all the Broadcom bits.
Const-me 5y ago
> it's incredibly hard to write custom firmware I remember how it was with programmable GPUs. When programmable shaders were introduced in GeForce 3 (2001), nobody really understood what to do with them. The consensus was that it’s incredibly hard to write custom shaders. OpenGL consortium refused to support shaders for almost a decade (GLSL 1.0 released in 2008), that’s one of the reasons of the success of Direct3D https://softwareengineering.stackexchange.com/a/88055 https://softwareengineering.stackexchange.com/a/88055 Yet look at them after 20 years of evolution. People no longer coding shaders in assembly, we have quite a few higher-level languages for them. People have built better APIs to interface with programmable GPUs, both low level close to metal (D3D12, Metal, Vulkan) and high-level (CUDA, TensorFlow, game engines). People built debuggers and profilers for these shaders. Overall, people embraced programmable GPUs, leaned to use them, and the results are amazing. Visual quality of games and other real-time 3D skyrocketed, despite the increase of pixel count and density in the displays. I can see how the same path is possible for the rest of the programmable chips and pieces inside modern computers. > and the benefit is very slim Better products in very broad sense. For instance, when idle, a smartphone OS could power off not just the screen, but also CPU, RAM, and a few other components, and downscale itself to the slowest and the most power efficient core in the system.

ironman1478 5y ago

I've worked on firmware for a few SoC's that my company has designed or purchased and we use embedded linux on some. I think his observation about how Linux doesn't abstract the hardware accurately is totally right. The system isn't that complicated, but implementing everything within the world of Linux has been painful imo, so for future implementations we're evaluating RTOS's. The main reason people use Linux on these custom SoCs because of driver availability of key components (ethernet phys and persistent storage), large library of tools, and probably $ (who wants to pay for QNX). Those are good reasons, but if you have to write a large volume of code, it doesn't really feel like it's providing too much. It definitely felt like we were just using a hammer and seeing everything as a nail. Those are my 2 cents.

elcritch 5y ago
This issue with needing more flexibility on SoC's makes me wonder if its part of why the Linux Foundation has put a lot of work and resources into Zephyr RTOS [1]. Its gaining a lot of popularity and incorporates some of nice elements of linux like device tree syntax (DTS) files [2]. I could readily imagine using Linux & Zephyr on a SoC, and using shared DTS to describe the SoC architecture. Actually, come to think of it extending DTS syntax to describe the pointer swizzling the OP video mentions. Then both OO'es could derive correct physical address handling. 1: https://www.linuxfoundation.org/blog/the-power-of-zephyr-rtos/ https://www.linuxfoundation.org/blog/the-power-of-zephyr-rto... 2: https://docs.zephyrproject.org/latest/guides/dts/index.html https://docs.zephyrproject.org/latest/guides/dts/index.html
- ironman1478 5y ago
  So, it's funny. It's not uncommon to run RTOSs and Linux on the same CPU, where the rtos coordinates tasks between the components that have real time requirements and the Linux side does stuff like stream data from a buffer that the rtos writes to. We had this architecture on some systems and the amount of cludging to get this all working was annoying to say the least. We specifically had issues are figuring out how to partition the memory regions for each OS and it wasn't trivial to validate. I wasn't around when the decisions at my employer when the decisions made around OS options, but this would've been a viable one if it was mature enough at that time. Thanks for the info!

fithisux 5y ago

Hardware should become more clever in order to take burden from drivers.

xfer 5y ago

It is a interesting debate to have but this is unlikely to happen for consumer hardware, since no OEM wants you to have that kind of access(talking of those phone SoCs), they will just give you a slice of ccNUMA to protect their "IP". That qualcomm bug that he showed is unlikely to go away for consumers. There is a message to build your own computers and that's probably what people will do and it could be interesting research for custom servers and future heterogeneous architectures. I suspect people in this industry are already doing it.

Ericson2314 5y ago

Conway's law at work! One one hand there is great vertical disintegration, on the other hand, even Apple is too lazy to actually do this right, still treating their software stack as a black box where breaking changes are fine but elegance is lost on the suits. I suppose the folks at https://oxide.computer/ https://oxide.computer/ are smirking if they see this video.

bcantrill 5y ago
It's very vindicating of our worldview, yes -- and also vindicates the de novo operating system (Rust kernel, Rust tasks) that we have developed for some of the use cases that Roscoe outlines. Looking forward to getting it all out there!

95014_refugee 5y ago

Having seen the coalface, the problem is not the OS, it's the (stupendously retarded) application model. There's very little (interesting) you can do when application developers are (for the most part) incapable and unwilling to embrace anything even remotely different from the DOS application model. The OS is, at its heart, a mapping from that application model onto the hardware. If a hardware feature or characteristic can't be expressed / surfaced within the application model, then there's little to no value in attempting to exploit it. Someone like  at least has internal customers, and a vertically-integrated embedded system can sometimes do better, but general-purpose systems are shackled to some minor variation on the POSIX application model for the forseeable future.

evilos 5y ago

Interesting talk but I'm not convinced of the value proposition of "rediscovering hardware". The speaker points out that there are lots of small processors inside a modern SoC that are often running their own software hidden from Linux. I agree that there are security implications here a la Qualcomm Hexagon vulnerabilities or Intel Management Engine backdoors but I'm not sure what the speaker's proposed alternative is. Is it that a "true" OS would directly execute the duties of these hidden processors? This is extra work that would make systems slower. Having dedicated hardware handle these duties is an optimization (hardware acceleration). Alternatively you could replace these small processors with fixed function hardware. This would make any system flaws permanent although it would be much harder to exploit any vulnerabilities. Does the lack of hidden processors make the system "better" somehow? (aside from potentially more secure) Are bus masters and hardware accelerators part of the OS? It would certainly be a more beautiful system that I would love to see, but I need help justifying the engineering cost. I hope to see some open hardware passion projects along these lines, but I doubt they will ever be mainstream.

SilverRed 5y ago
I'm increasingly seeing the value of Apple style products where almost every part of the device from the hardware, to firmware, to OS, to UI is all from the same company. The whole combined experience just makes things run so smoothly and allows security issues to be patched since Apple has the resources that maintain each part internally.
- pas 5y ago
  Yeah, of course that works, for a while at least. Also developed in the open, patches are welcome, you can update your own NIC/GPU/Wifi-dongle components would work too. Unfortunately the whole industry (Apple as one of the main culprits) is quite hostile to anything like that. Though maybe we'll have a RedHat of hardware someday.
- darkwater 5y ago
  Yeah, and as a side-effect you have to obey your one true lord.

OneTimePetes 5y ago

Oh, oh the ragnarok of systems programming, moores plateau stretches on, parallel is coming to a limit and now its goodbye abstractions & no-skill-lib-glue-code, hello 80s hardware hacks and efficient programming again? Will OSes be produced like games, were they start with a "reuseable" oo-engine and then that is broken down into data-orientated design. I thought we had more time.. Cant we global warm postpone this? Let the next generation sort it all out?

tomovo 5y ago

Perhaps related: The 30 million line problem (Casey Muratori): https://www.youtube.com/watch?v=kZRE7HIO3vk https://www.youtube.com/watch?v=kZRE7HIO3vk Among other things, advocating for more stable & open standard HW interfaces, rather than OS-level APIs.

ncmncm 5y ago

The real progress nowadays, since OSes have not stepped up, is in Kernel Bypass. (Show me a fancy new abstraction, and my reaction is, "Great—how do I bypass it?") We have para-kernels, demi-kernels, uni-kernels, all to get closer to the hardware. Probably future machines should have lots of FPGAs, and easy access to claim and program them. We will need good languages for the latter, although probably C++ will end up being that.

jdefr89 5y ago

I am confused. He is basically saying Linux isn't powering all the hardware simply because firmware is running on other more complex peripherals/SoCs on a device. That will always become the case as hardware gets more complicated and advanced. Linux simply need to provide the means to abstract the nitty gritty. Linux is still acting as the central "manager" of all the hardware. As far as security. Linux can act as the final security boundary for getting the machine to do anything useful.

xnormal 5y ago

Academics can sometimes learn a lot from industry. The ScyllaDB database and its underlying SeaStar library essentially implement a userspace OS that handles the hardware very differently than a typical modern OS with phenomenal improvements in performance. Historically, it started as a unikernel called OSv designed to run directly on hypervisors. But they eventually realized that it can run just as well on linux with kernel bypass working and io_uring and is easier to deploy.