5 ms·

So, what's your suggestion for... umm... practical/real world OS dev class? xv6 code is really compact, but toyish?

by anta40 4y ago

mhh__ 4y ago
Linux? (Obviously depends on what you want to teach but in terms of practicality it may be daunting to comprehend the whole kernel – of course it is – but equally the ground is very well trodden)
- anta40 4y ago
  Err... well I get it. What I mean is something less complex than Linux, and still usable enough for real world tasks? I can understand the xv6 codebase in a few days (obviously couldn't say the same thing about Linux), and it's very easy to rebuild, but for me it only stays inside QEMU. Or perhaps take a more realistic approach: production-grade kernel is complex, thus you are not be expected to understand the whole thing in short time?
  - hakfoo 4y ago
    I suppose that reveals something: cheap virtualization completely changes the OS development experience. Even beyond teaching toys, there are huge projects now (SerenityOS for example) where "runs on bare metal" is barely, or not a consideration. I wonder if this causes stagnation in the niche of device driver development-- if all you have to worry about is a handful of "friendly" VM devices, who is learning to deal with the hassles of persnickety real hardware?
    - throwawaylinux 4y ago
      No it doesn't because the cutting edge of device driver development is on high performance devices in the hypervisor or IO server layer, passthrough and virtualized functions, etc. So it's all the same challenges as before with additional complexity that you also have to virtualize them either by passing function to the guest or by introducing a virtual server layer above them that has to perform well. I don't know if I've ever seen much innovative device driver development work going on in toy or learning operating systems before virtio type interfaces. Before that most of them used a couple of the simplest hardware available anyway. There has long been (and still is) a bunch of different emulated hardware that something like qemu offers (and not too hard to add your own emulated hardware) that you can trivially add to your VM if you want to develop device drivers. A number of real hardware companies nowadays actually contribute their hardware models to qemu and even develop to them internally for pre-release development and CI and regression testing. So I would argue running in VMs with capable device model support like QEMU has actually made learning and developing hardware device drivers more accessible rather than less.
      - hakfoo 4y ago
        I always figured that a major source of complexity in drivers was papering over imperfections in the underlying hardware. I imagine things like video and chipset drivers are full of "if we're on stepping A0 or A1 hardware, we have to do this magic wiggle dance to get feature XYZ to actually work" hacks, most of which are only documented within the manufacturer's own propriatery-driver team. There's probably also a lot of compromise for external confounding variables-- "Our part works to spec 98% of the time, but if you install it in specific systems, they're at a specific corner of the compatibility envelope, and you have to do ABC to get it actually stable." I'd suspect as a result, something like "Driver for VirtualBox SVGA" is going to be a lot simpler than a comparable level of functionality targeting real hardware. OTOH, we do have stuff like 86box which seems to be trying to be a closer simulation of real hardware.
boricj 4y ago
xv6 is pretty much the best osdev beginner's ressource out there. Once you've mastered it, "Operating Systems: Design and Implementation" is still a worthwhile read even though it's getting a bit old. After that, it depends on what you find interesting. Personally, I think Unix is a fossilized relic whose 50 year old design has been pushed far beyond its use-by date. I'm a fan of Fuchsia OS and especially its Zircon kernel, its design documentation is well worth a read just to shake off the idea that POSIX makes for a great syscall layer in the 21st century. In the end, it doesn't matter too much, you'll always learn something no matter what the operating system you look into. Try things out and discover what you like instead of relying on what others find interesting. I just wish that people stop thinking that the traditional, monolithic Unix system design is some sort of holy scripture that shall not be questioned.
- eestrada 4y ago
  I'm guessing the stickiness of Unix is that a lot of useful software, espescially FOSS, just compiles and works on Unix-y operating systems, assuming the right syscalls are implemented. There is far less reinventing of the wheel by going that route.
  - boricj 4y ago
    Implementing POSIX as a compatibility layer on systems not designed to be Unix-like is possible. You can even skip a fair number of the most problematic parts and still run useful programs. OpenVMS doesn't have fork(), just vfork() and it can run LLVM and modified versions of bash and GCC. Windows NT used to have a POSIX subsystem. Heck, IBM even managed to certify z/OS to be POSIX compliant. Just because you happen to need some level of POSIX compatibility for interoperability doesn't mean that POSIX has to be the basis for your syscall layer.
- snvzz 4y ago
  Did you look into Genode?
- pjmlp 4y ago
  That is why I rather be bothered to look into other stacks, specially how mobile and cloud OS are being designed. UNIX had a couple of interesting design options and that is about it, time to move on.
veqq 4y ago
The best I've found by far: Introduction to Operating System Abstractions using Plan 9 from Bell Labs https://archive.org/details/plan9designintro https://archive.org/details/plan9designintro
lproven 4y ago
The key problem is: > practical/real world Definitions of that vary wildly. Oberon is a personal favourite of mine: tiny, a few hundred KLOC for the entire OS, compiler IDE, UI, in a native-code-compiled, type-safe language.
- anta40 4y ago
  Ah, another Niklaus Wirth's work. Any guide on how to build and run it?
  - lproven 4y ago
    It depends on what platform you wish to work. There is quite comprehensive documentation on wikibooks: https://en.wikibooks.org/wiki/Oberon https://en.wikibooks.org/wiki/Oberon You can run it in a browser: https://schierlm.github.io/OberonEmulator/ https://schierlm.github.io/OberonEmulator/ There is an app in the Mac App Store: https://apps.apple.com/us/app/oberon-workstation/id1057155516 https://apps.apple.com/us/app/oberon-workstation/id105715551... You can run it on Linux: http://oberon.wikidot.com/ http://oberon.wikidot.com/ It also runs atop Windows and macOS. Lots of choices, really. :-)
  - carapace 4y ago
    You can use it live in the browser from here: http://schierlm.github.io/OberonEmulator/ http://schierlm.github.io/OberonEmulator/ Book and code are available from Wirth's site: https://people.inf.ethz.ch/wirth/ProjectOberon/index.html https://people.inf.ethz.ch/wirth/ProjectOberon/index.html But the mirror is a little nicer looking (although older? It says, "The second (2013) edition of the book and source code are published on Prof. Wirth's website. We provide links to the original material here, and local zipped copies, with kind permission from the authors."): http://www.projectoberon.com/ http://www.projectoberon.com/ Several emulators for the Oberon RISC CPU are available: In C with SDL graphics: https://github.com/pdewacht/oberon-risc-emu https://github.com/pdewacht/oberon-risc-emu In JS and Java, from the live version above: http://schierlm.github.io/OberonEmulator/ http://schierlm.github.io/OberonEmulator/ In Go: https://github.com/fzipp/oberon https://github.com/fzipp/oberon People have used Wirth's Verilog with FPGAs to make actual Oberon workstations. (My own somewhat embarrassing Python emu: https://git.sr.ht/~sforman/PythonOberon https://git.sr.ht/~sforman/PythonOberon )

So, what's your suggestion for... umm... practical/real world OS dev class? xv6 code is really compact, but toyish?

by anta40 4y ago

So, what's your suggestion for... umm... practical/real world OS dev class? xv6 code is really compact, but toyish?

Linux? (Obviously depends on what you want to teach but in terms of practicality it may be daunting to comprehend the whole kernel – of course it is – but equally the ground is very well trodden)

anta40 4y ago
Err... well I get it. What I mean is something less complex than Linux, and still usable enough for real world tasks? I can understand the xv6 codebase in a few days (obviously couldn't say the same thing about Linux), and it's very easy to rebuild, but for me it only stays inside QEMU. Or perhaps take a more realistic approach: production-grade kernel is complex, thus you are not be expected to understand the whole thing in short time?
- hakfoo 4y ago
  I suppose that reveals something: cheap virtualization completely changes the OS development experience. Even beyond teaching toys, there are huge projects now (SerenityOS for example) where "runs on bare metal" is barely, or not a consideration. I wonder if this causes stagnation in the niche of device driver development-- if all you have to worry about is a handful of "friendly" VM devices, who is learning to deal with the hassles of persnickety real hardware?
  - throwawaylinux 4y ago
    No it doesn't because the cutting edge of device driver development is on high performance devices in the hypervisor or IO server layer, passthrough and virtualized functions, etc. So it's all the same challenges as before with additional complexity that you also have to virtualize them either by passing function to the guest or by introducing a virtual server layer above them that has to perform well. I don't know if I've ever seen much innovative device driver development work going on in toy or learning operating systems before virtio type interfaces. Before that most of them used a couple of the simplest hardware available anyway. There has long been (and still is) a bunch of different emulated hardware that something like qemu offers (and not too hard to add your own emulated hardware) that you can trivially add to your VM if you want to develop device drivers. A number of real hardware companies nowadays actually contribute their hardware models to qemu and even develop to them internally for pre-release development and CI and regression testing. So I would argue running in VMs with capable device model support like QEMU has actually made learning and developing hardware device drivers more accessible rather than less.
    - hakfoo 4y ago
      I always figured that a major source of complexity in drivers was papering over imperfections in the underlying hardware. I imagine things like video and chipset drivers are full of "if we're on stepping A0 or A1 hardware, we have to do this magic wiggle dance to get feature XYZ to actually work" hacks, most of which are only documented within the manufacturer's own propriatery-driver team. There's probably also a lot of compromise for external confounding variables-- "Our part works to spec 98% of the time, but if you install it in specific systems, they're at a specific corner of the compatibility envelope, and you have to do ABC to get it actually stable." I'd suspect as a result, something like "Driver for VirtualBox SVGA" is going to be a lot simpler than a comparable level of functionality targeting real hardware. OTOH, we do have stuff like 86box which seems to be trying to be a closer simulation of real hardware.

boricj 4y ago

xv6 is pretty much the best osdev beginner's ressource out there. Once you've mastered it, "Operating Systems: Design and Implementation" is still a worthwhile read even though it's getting a bit old. After that, it depends on what you find interesting. Personally, I think Unix is a fossilized relic whose 50 year old design has been pushed far beyond its use-by date. I'm a fan of Fuchsia OS and especially its Zircon kernel, its design documentation is well worth a read just to shake off the idea that POSIX makes for a great syscall layer in the 21st century. In the end, it doesn't matter too much, you'll always learn something no matter what the operating system you look into. Try things out and discover what you like instead of relying on what others find interesting. I just wish that people stop thinking that the traditional, monolithic Unix system design is some sort of holy scripture that shall not be questioned.

eestrada 4y ago
I'm guessing the stickiness of Unix is that a lot of useful software, espescially FOSS, just compiles and works on Unix-y operating systems, assuming the right syscalls are implemented. There is far less reinventing of the wheel by going that route.
- boricj 4y ago
  Implementing POSIX as a compatibility layer on systems not designed to be Unix-like is possible. You can even skip a fair number of the most problematic parts and still run useful programs. OpenVMS doesn't have fork(), just vfork() and it can run LLVM and modified versions of bash and GCC. Windows NT used to have a POSIX subsystem. Heck, IBM even managed to certify z/OS to be POSIX compliant. Just because you happen to need some level of POSIX compatibility for interoperability doesn't mean that POSIX has to be the basis for your syscall layer.
snvzz 4y ago
Did you look into Genode?
pjmlp 4y ago
That is why I rather be bothered to look into other stacks, specially how mobile and cloud OS are being designed. UNIX had a couple of interesting design options and that is about it, time to move on.

veqq 4y ago

The best I've found by far: Introduction to Operating System Abstractions using Plan 9 from Bell Labs https://archive.org/details/plan9designintro https://archive.org/details/plan9designintro

lproven 4y ago

The key problem is: > practical/real world Definitions of that vary wildly. Oberon is a personal favourite of mine: tiny, a few hundred KLOC for the entire OS, compiler IDE, UI, in a native-code-compiled, type-safe language.

anta40 4y ago
Ah, another Niklaus Wirth's work. Any guide on how to build and run it?
- lproven 4y ago
  It depends on what platform you wish to work. There is quite comprehensive documentation on wikibooks: https://en.wikibooks.org/wiki/Oberon https://en.wikibooks.org/wiki/Oberon You can run it in a browser: https://schierlm.github.io/OberonEmulator/ https://schierlm.github.io/OberonEmulator/ There is an app in the Mac App Store: https://apps.apple.com/us/app/oberon-workstation/id1057155516 https://apps.apple.com/us/app/oberon-workstation/id105715551... You can run it on Linux: http://oberon.wikidot.com/ http://oberon.wikidot.com/ It also runs atop Windows and macOS. Lots of choices, really. :-)
- carapace 4y ago
  You can use it live in the browser from here: http://schierlm.github.io/OberonEmulator/ http://schierlm.github.io/OberonEmulator/ Book and code are available from Wirth's site: https://people.inf.ethz.ch/wirth/ProjectOberon/index.html https://people.inf.ethz.ch/wirth/ProjectOberon/index.html But the mirror is a little nicer looking (although older? It says, "The second (2013) edition of the book and source code are published on Prof. Wirth's website. We provide links to the original material here, and local zipped copies, with kind permission from the authors."): http://www.projectoberon.com/ http://www.projectoberon.com/ Several emulators for the Oberon RISC CPU are available: In C with SDL graphics: https://github.com/pdewacht/oberon-risc-emu https://github.com/pdewacht/oberon-risc-emu In JS and Java, from the live version above: http://schierlm.github.io/OberonEmulator/ http://schierlm.github.io/OberonEmulator/ In Go: https://github.com/fzipp/oberon https://github.com/fzipp/oberon People have used Wirth's Verilog with FPGAs to make actual Oberon workstations. (My own somewhat embarrassing Python emu: https://git.sr.ht/~sforman/PythonOberon https://git.sr.ht/~sforman/PythonOberon )