9 ms·
Writing an OS in Rust: Introduction to Paging
- JesseWright 8y agoI've followed Phil's site on making an operating system. Ever since having worked on PintOS in college, operating systems and their internals have fascinated me. I've tried learning more about writing operating systems via resources like the OSdev wiki, but much of its material almost seems to discourage working on an OS of any kind unless you're already sure of what you're doing. Phil's articles seems so much more accessible for someone just wanting to dip their toes in the water.
- fermienrico 8y agoI agree. Wouldn't it be amazing to have a "Handmade Hero" series but for OS development in Rust? That would be incredible. A lot of the OS development, especially esoteric subjects are behind walls of IP control barring Linux kernel. Even then, just reading the source code is different than having access to the knowledge base that enables you to write such a code. I am new to all of this but like you, I am fascinated.
- jb3689 8y agoJust found "Handmade Hero" last weekend and now this. Programmer's dream
- faitswulff 8y agoHow does one approach watching Handmade Hero? There's just so much content...
- fermienrico 8y agoThink it is a problem but every other resource in game development is summarized, organized and bookish. Handmade Hero is unique in that it is: - Real time, watching a dude program a game - He speaks his though process, stumbles, gets up and resolves issues - Community interaction as they progress through the game All, at the expense of content bloat. I think its popularity has spoken for itself in that Handmade Hero is a totally unique form of education.
- johnc1231 8y agoIs there any video somewhere of the actual game the handmade hero series builds up to? I would want to know the goal before jumping in
- AnIdiotOnTheNet 8y agoThe game is still actively in development. It isn't just a clone of some other game, it's a new game and the actual design of the game is another part of the project that's evolving as the engine is built.
- smrq 8y agoTo be honest, I think this is the primary weakness of Handmade Hero. Casey is plainly a more talented programmer than designer, and his missteps in the latter domain felt to me more like wasted time than valuable experience. This is why I stopped watching somewhere between episode 200 and 300.
- db48x 8y agoBacktracking, experimenting, and changing your mind is part of the game development process. No game ever ends up being exactly what the designers imagined at the start.
- tormeh 8y ago>A lot of the OS development, especially esoteric subjects are behind walls of IP control barring Linux kernel. Join the Redox community! Development is on gitlab[0] and chat is on Mattermost. 0: https://gitlab.redox-os.org/redox-os/ https://gitlab.redox-os.org/redox-os/
- rixrax 8y ago+1! Redox Os (micro kernel Os written in Rust) feels like a gust of fresh air! Godspeed! I am rooting for you! I don’t know what this says about me but I haven’t been this excited about new OS since I first installed OpenBSD 2.5. :)
- simias 8y agoI think it's one of these things that's more approachable if you start by working on specific subsystems of an existing OS. This way you don't have to figure everything out at once. At least that's how I did it. I think writing an OS from scratch is discouraging and not very accessible because... writing an OS is discouraging and not very accessible. It's a bit as if a painter was saying "It tried looking up guides on how to paint The Wedding Feast at Cana but they all make it look super difficult, I wish I could get an easy step-by-step tutorial". Even if you break it down in small, digestible parts I'd wager that you'll end up with a few hundred episodes before you even get a basic microkernel up and running on a modern system. It's truly a daunting task. I learned that the hard way: I tried to write a guide on how to emulate a PlayStation from scratch. At first you focus on emulating the instructions in the CPU, that's relatively focused and straightforward. But then once you're done with that you need to explain the various peripherals, and the main bus, and the interrupts, and the cache, and the timers, and the pipelines, and the intricacies of the various CD formats, and the memorycard filesystem, and video modes, and paging, and high impedance, and metastability and everything kind of interacts with everything else so it's difficult to maintain a coherent narrative. On top of that a PlayStation is a ridiculously simple system compared to a modern desktop computer. The task of writing a very accessible guide on how to write an OS that would take you from zero to a working userland is absolutely tremendous. I think you could probably write a thousand-page book on the virtual memory subsystem alone.
- Analemma_ 8y agoFor what it's worth, this is exactly how most Operating Systems courses in undergraduate CS work, so it definitely seems like many different smart people have converged on this as the right way to go. At least in my course, we did build a toy OS by the end of the semester, but it was rigorously divided projects for the individual subsystems in isolation: write the pre-emptive scheduler, write the networking layer, write the filesystem, etc. A modern OS is an incredibly complex beast and trying to grasp it all at once is just going to lead to madness.
- JesseWright 8y agoI fully agree, but I think it's good to have options on how to approach the issue. Generally, I've referred to OSDev when I need more real-world information on how things tend to work in practice and resources like Phil's when writing a toy operating system. I wouldn't consider Phil's blog a good resource for, e.g., gaining particular insight into how something like the Linux kernel works, but I also don't think Phil intended it for that use.
- vectorEQ 8y agodefinitely not an easy journey :D to learn to write x64 OS. After getting paging and interrupts it plunged me into the depths of compilers / assemblers / file formats and linking & loading all to be tackled at the same time :'D would use ELF to prevent 2 of the topics, but where's the fun in that :D like these articles too as they are fairly complete where a lot of other tutorials omit a long of things or are just some scattered collection of information which is hard to put into a project.
- renholder 8y agoSame here. Especially MenuetOS[1], because it was written in Assembly and originally fit on a floppy disk. That's a level of patience and programming (in Assembly) that I can only admire from afar... [1] - https://en.wikipedia.org/wiki/MenuetOS https://en.wikipedia.org/wiki/MenuetOS
- gshack 8y agoHoly heck, it's still in active development
- jstimpfle 8y agoI've never programmed in Rust because I still have trouble seeing the point. It seems overly restrictive. A copy of any data has nearly zero marginal cost (and you can pay the cost by doing the work yourself), so in a free machine, it would have nearly zero price. If widely useful data is protected by the borrow checker, far fewer people will use it. It is easy to show that the total utility of a piece of data to the rest of the program is reduced by assigning an owner to it. Each potential user of the data, faced with the need to mess with the borrow checker to use it, may choose to mess, or may forego use of the data and write a complicated workaround instead. To counter the borrow checker, I'm currently designing a programming language featuring a mechanism that grants free access to data structures, with the restriction that copiers must grant the same rights to anyone else: 0. the freedom to use the data for any purpose, 1. the freedom to change the data to suit your needs, 2. the freedom to share the data with your friends and neighbors, and 3. the freedom to share the changes you make.
- nicoburns 8y ago> A copy of any data has nearly zero marginal cost I don't think this is true at all. Reducing copies and allocations in my programs is one of the things which has speeded them up the most. > If widely useful data is protected by the borrow checker, far fewer people will use it. Are you also against private struct/class fields? Because they also restrict access to useful data. For me, the need for Rust is as a language that gives me maximum possible performance, while giving me safety guards that C and C++ do not. This makes Rust much easier to use than those languages for me.
- fermienrico 8y agoAs a side note: I wish Rust would come to embedded development ASAP running a real time OS written in Rust. You can use massive Boost libraries and Qt frameworks on non-embedded systems with C/C++, but in embedded, there is no such luxury. I am new to it but it is painful and I wish to see a day where we have a complete LLVM based ARM compiler toolchain with Rust sitting on top. I think there were some blog posts about running Rust on STM32 but it is still very early.
- nicoburns 8y agoIt is still pretty early, but if there's been quite a bit of progress this year. This is probably the closest to what you are looking for that exists at the moment: https://github.com/japaric/cortex-m-rtfm https://github.com/japaric/cortex-m-rtfm You can also follow the Rust Embedded Working Group blog if you wish: https://rust-embedded.github.io/blog/ https://rust-embedded.github.io/blog/
- fermienrico 8y agoThank you for sharing, subscribed. As I read through the Rust Embedded Work Group manifesto, it is comforting to see the level of organizational skills, planning and vision of the team involved. Bravo. https://github.com/rust-embedded/wg https://github.com/rust-embedded/wg
- bluejekyll 8y agoHave you looked at TockOS? https://www.tockos.org https://www.tockos.org I took a class in it about 2 years ago now, it was really cool. I’m not an embedded developer though, so not sure of your needs here.
- vardump 8y agoOoh that looks pretty interesting! Tock appears to be a Rust based embedded OS, so relevant to this discussion. https://github.com/tock/tock https://github.com/tock/tock
- wyldfire 8y ago
- abalone 8y agoAny idea what tool might have been used to create the diagrams?
- phil-opp 8y agoThey were created with draw.io (https://about.draw.io/ https://about.draw.io/).
- amelius 8y agoOff topic: can we please get support for storing arbitrary data structures inside mmapped files (with nonfixed base address)?
- Zoh0eich 8y agoThat's not possible to do safely for arbitrary data structures because they can contain pointers or references. If you reload to a different address then the pointers, even within the allocated chunk, would be invalid and require fixup. And if you require fixup then you might as well do (de)serialization for your data structures. Just allocating into such a memory region (essentially custom page swapping) will eventually be possible with data structures that can be constructed with custom allocators, but it still wouldn't be ok to reload those.
- amelius 8y agoPerhaps this problem can be overcome by modifying the concept of a pointer slightly, to make it a "based pointer" (?) I found an old discussion on this topic here: https://news.ycombinator.com/item?id=13890011 https://news.ycombinator.com/item?id=13890011 Note that the Microsoft C++ compiler implements based pointers, using the __based keyword.
- Zoh0eich 8y agoSpecialized data structures can be built that contain primitives and other types supporting this kind of thing. But you still can't store arbitrary data because they could contain FFI pointers, handles to native resources, things that need to run destructors etc. So to do this safely all the types must be whitelisted through a custom trait. Copy is close, but not quite, since references are still Copy.
- mapcars 8y agoWhy not join Redox?
- deleted 8y ago[deleted]
- justizin 8y agoRedox looks awesome, my impression is that this is more about learning by implementing from scratch. I could imagine feeling more comfortable contributing to something like Redox after working with this course.
- phkahler 8y agoOn page tables... Why would they go to 5 levels rather than just increasing the page size and the number of bits in the indexes from 9 to 12 and make each page 32K or 64K?
- phil-opp 8y agoI assume backwards compatibility. It's a much smaller change to existing operating systems than to create a completely new page table format.
- jstewartmobile 8y agohttps://os.phil-opp.com/paging-introduction/#example-translation https://os.phil-opp.com/paging-introduction/#example-transla... Frame entry 511 for "Level 2 Page Table" should be 32KiB instead of 24KiB.
- phil-opp 8y agoThanks! Fixed in https://github.com/phil-opp/blog_os/commit/f423c068fec80372564f5dc70ee3ad36cbfceeb9?short_path=06711e9#diff-06711e99ec881661414cc904d2d1eb56 https://github.com/phil-opp/blog_os/commit/f423c068fec803725...
- crb002 8y agoCurious to see a re-write of MINIX in Rust.
- sitzkrieg 8y agoPhil writes this topic very clearly unlike many others, good stuff
- miohtama 8y agoNot only this is interesting well understandable Rust code for non-Rust developer, but it is also one of the best tutorial how page tables work on modern PC CPUs.
- billsix 8y agoI had trouble understanding how paging worked it until it was explained in Rust
- zbentley 8y agoRegardless of the platform, this is a truly excellent introduction to memory segmentation, which in my experience tends to be relevant far outside of the realm of operating system (or even single process application) development. Just the other day I was struggling to explain several concepts illustrated eloquently here to a coworker trying to design a scheduling/persistence system for variably-sized datasets that needs to be stored in hundreds of Redis instances across a compute cluster.
- lxe 8y agoWith memory growing arguably beyond what most application workloads really require anyways, and possibly hardware such as Intel Optane, do we still need things like VM, paging, and segmentation?
- cowboysauce 8y ago>do we still need things like VM, paging, and segmentation? Yes. To begin with, on modern processors, paging is tied with the memory protection system. Memory is protected on a per-page basis. Memory mapped files are implemented through paging. There's also backwards compatibility. Virtual memory allows 32-bit applications to have their physical memory located anywhere. Without that, we'd have to ensure that they reside below 4GB. Programs would start up slower as the executable would have to be relocated every time. Additionally, it's hard to imagine a hypervisor that doesn't use virtual memory. Virtual memory allows hypervisors to present a unified address space to their guests; no matter how fragmented physical memory gets.
- lxe 8y agoI was thinking in terms detached from typical system architecture. What if we don't tie paging with memory protection when designing CPUs? What if we don't assume, build, or run 32 bit applications? What if we do actually put a hard physical 4GB (or whatever) limit on processes? What if we don't need hypervisors? What if we trade off hardware and software complexity we've built decades ago for limitations we had decades ago, and build something simpler from scratch?
- hyperthreading 8y agoThe article seems written much easier to follow than PintOS manual. Could anyone tell me how much writing OS became enjoyable by adopting Rust? Dealing with a excessive amount of unsafe pointers was really a pain in my memory.
- yesimahuman 8y agoNothing like finally grokking a major part of the CS operating systems curriculum. Wonderful write up
- nohope 8y agoI'm a few surprised nobody have mentioned UlixOS [1]. I'm reading their book, it is just great on many aspects, although it rushes to explain other things that should be detailed a bit more. The chapter about paging is very explicative too. [1] http://ulixos.org/ http://ulixos.org/
- hi41 8y agoWhile in college, I did not understand paging well. Thank you for the wonderful post.