16 ms·
Theseus: A modern experimental OS written from scratch in Rust
- monocasa 6y agoWas looking for the "why" behind this OS, and one of the authors gave a great OSDI talk on that very subject. https://www.youtube.com/watch?v=j4ZPZoPNjkw https://www.youtube.com/watch?v=j4ZPZoPNjkw
- ipodopt 6y agoThank you for the link! EDIT: End to end we do not give enough information from the application to OS to the hardware for perfect optimizations. We do not start with enough information and we lose too much down the stack. Especially around concurrency. I think that at the end of the day there is some atomic information needed by hardware and OS to make optimizes about all code running on the system. I see the approach taken here in an effort to retain needed information by integrating compilation and OS level functionality. I think this is correct but we will see a shift to OSes having there own byte code as a translation layer between higher level languages. I also think high level languages lack some needed information in the first place...
- User23 6y agoI can't recall the details or find them searching online, but I recall reading back in around 2000 about an emulator that was actually faster than bare metal when emulating its own hardware. I think maybe it was a Sun Microsystems project, but I'm not sure. Does anyone else recall this?
- sweetsocks21 6y agoYou might be thinking of the Taos Operating System? https://news.ycombinator.com/item?id=9806607 https://news.ycombinator.com/item?id=9806607 http://www.uruk.org/emu/Taos.html http://www.uruk.org/emu/Taos.html It had an object oriented assembly language that everything compiled to and it JIT'd/on-fly-compiled code IIRC. Sounded very cool.
- CyberRabbi 6y agoThese things always sound cool.
- pjmlp 6y agoUnfortunely when these things actually suceed on the market, we only get IBM and Unysis mainframes, Android, watchOS and .NET Native/WinRT, which are always a kind of compromise of the whole idea.
- mcshaner1 6y agoAre you thinking of HP’s Dynamo? http://cseweb.ucsd.edu/classes/sp00/cse231/dynamopldi.pdf http://cseweb.ucsd.edu/classes/sp00/cse231/dynamopldi.pdf
- KMag 6y agoIt was almost definitely HP Dynamo. (Edit: if you combine ideas from HP Dynamo, SafeTSA JIT-optimized bytecode, and IBM's AS/400's TIMI/Technology Independent Machine Interface, you get a better version of the current Android Run Time for bytecode-distributed apps that compile ahead of time to native code and self-optimize at runtime based on low-overhead profiling.) The really nice thing about Dynamo was that it was a relatively simple trace-based JIT compiler from native code to native code (plus a native code interpreter for non-hotspots). This meant that it would automatically inline hotspots across DLLs and through C++ virtual method dispatches (with appropriate guards to jump back to interpreter mode if the virtual method implementation didn't match or the PLT entry got modified). They didn't have to do any special-casing of the interpreter to handle virtual method calls or cross-DLL calls, it's just a natural consequence of a trace-based JIT from native code to native code. The only downsides of something like Dynamo are (1) a bit of complexity and space usage (2) some startup overhead due to starting in interpretive mode and (3) if your program is abnormal in not having a roughly Zipf distribution of CPU usage, the overhead is going to be higher. Ever since I read about Michael Franz et al.'s SafeTSA SSA-based JVM bytecode that more quickly generated higher-performing native code, I've had a long-term back-burner idea to write a C compiler that generates native code in a particular way (functions are all compiled to arrays of pointers to strait-line extended basic blocks) that makes tracing easier, and also storing a SafeTSA-like SSA bytecode along with the native code. That way, a Dynamo-like runtime wouldn't use an interpreter, and when it came to generate an optimized trace, it could skip the first step of decompiling native code to an SSA form. (Also, the SSA would be a bit cleaner as input for an optimizer, as the compilation-decompilation round-trip tends to make the SSA a bit harder to optimize, as shown by Franz's modification of Pizza/JikesRVM to run both SafeTSA and JVM bytecode.) Once you have your trace, you don't need on-stack replacement to get code in a tight loop to go into the optimized trace, you just swap one pointer to native code in the function's array of basic blocks. (All basic blocks are strait-line code, so the only way to loop is to jump back to the start of the same basic block via the array of basic block pointers.) The background for HP Dynamo is that during the Unix wars, there were a bunch of RISC system vendors vying for both the high-end workstation and server markets. Sun had SPARC, SGI had MIPS, DEC had Alpha AXP (and earlier, some MIPS DECStations) and HP had PA-RISC. The HP Dynamo research project wanted to show that emulation via dynamic recompilation could be fast, so to get an apples-to-apples comparison for emulation overhead, they wrote a PA-RISC emulator for PA-RISC.
- soamv 6y agoMaybe VMware? There was a paper about how, in some cases, emulation was faster than hardware virtualization support. This was way back when intel's hardware virtualization support was new and VMware had already spent years optimizing software virtualization.
- brodie 6y agoI’m being somewhat facetious or jocular when I say this—and somewhat serious—but…I wonder: Is that a reflection of the quality of the software implementation? Or is it a reflection of the hardware it’s trying to implement? Or perhaps it’s related to the hardware the emulator is running on? Or did the emulator emulate the hardware while running on /that/ hardware? Did it pull efficiency gains out of seemingly thin air?
- cosmotic 6y agoWhat does modern mean in this context?
- rubiquity 6y agoIt’s being written in Rust in the year 2021.
- worik 6y agoThat is a very weak definition of "modern"
- fiddlerwoaroof 6y agoI find that the word “modern” is usually meaningless marketing filler.
- Rexxar 6y agoThat's the original and etymological definition of the word. We are used to progress so we see modern thing as better but modern things are just new or trendy not necessary better. Programming everything in javascript is quite modern but is it a progress ? We will see in the long run.
- 3836293648 6y ago2017*
- gameswithgo 6y agowhat does it ever mean? its devoid of meaning
- O_H_E 6y agofrom this https://news.ycombinator.com/item?id=25742187 https://news.ycombinator.com/item?id=25742187 it might mean that it is not a unix/posix style OS, but exploring new ideas.
- brod 6y ago
- qwantim1 6y agoIf it were to have a Linux emulator, it could be named Liquor. (“Liquor is quite useful on removal.”)
- raggi 6y agoI do wonder about single address space, I've seen a few experimental systems follow this strategy lately and it feels reliant on hardware less defective than we presently have in commodity user gui performance systems.
- kev009 6y agoIBM i (OS/400) has had single address space, tagged pointers, and an number of other "novel" things in the loop for eons. Many people interact with it daily, indirectly, in most of your big box retail stores (think stores like Costco, Lowes, etc).
- coldtea 6y agoParent did say "it [single address space] feels reliant on hardware less defective than we presently have in commodity user gui performance systems". OS/400 and successors don't run on this kind of hardware.
- kevin_thibedeau 6y agoIs that kind of hardware going to be available in 50 years?
- saghm 6y agoI did a ctrl+F for the word "redox" here, since that's already modern experimental OS being written in Rust that's existed for some time, and I was surprised not to see it mentioned at all; I figured there would be a section for modern OS neophytes like me comparing and constrasting the two. Anyone have any insight into how this project differs from Redox?
- shmerl 6y agoRedox might have different goals which leads to different design approach.
- raggi 6y agoRedox is a unix style operating system with a unix style design and ABI. This operating system shares none of those goals, it seems to focus on a particular challenge related to the abstraction between subsystems of the operating system, following a term they call "state spilling". The goal appears to be to design an architecture that supports in-place updates of most components due to them having tightly constrained state management and inter-component dependencies.
- pjmlp 6y agoI wouldn't call micro-kernels a " unix style operating system with a unix style design", quite the opposite. The only thing in common with UNIX from current Redox is the POSIX support.
- monocasa 6y agoA microkernel is a valid implementation for a unix like. The way that posix filesystem semantics are a kernel provided abstraction is very "unix style".
- harry8 6y agoYeah totally valid implementation of which there have been a great many attempts, far too many to list here. Precisely none of them in 3-5 decades has managed to deliver on the promise of microkernels prompting the question: "Were huge microkernel advantages actually greatly oversold, is posix a massively debilitating factor that prevents microkernel based systems from delivering on their raison d'etre or was every single one of those implementations a poor one? [1]" [1] including QNX
- Ericson2314 6y ago:) I've always have had a soft-spot for language-based designs. It's the only way we can get fearless refactoring throughout the stack. If I may plug, check out https://internals.rust-lang.org/t/dogfooding-z-build-std-in-rustbuild/13775/19 https://internals.rust-lang.org/t/dogfooding-z-build-std-in-.... I'm it can heave us over the fence for the types of things that will make having a no_std cargo ecosystem easier.
- spicybright 6y ago2021 is the year of the lisp machine, I just know it!!! :(
- im3w1l 6y agoIf you explained why you want it to be, someone might hear.
- antonzabirko 6y agoThe memes have foretold the coming of rustOs.
- ids17 6y agoWhich is better for writing OS C,C++ or Rust?
- lbj 6y agoOne of the most interesting projects I've seen on here in a while. Thinking of the massive efforts of thousands of developers who get Linux to where it is today, its a daunting undertaking, but definitely one that could be worthwhile, though I think it will depend on rallying lots and lots of skilled engineers. In the early days of Linux, getting something like your wifi to work could be a hassle, I wonder if enough have changed in the driver space that we can quickly power through some of these challenges today.
- cwyers 6y agoThe name alludes to the Ship of Theseus, a thought experiment where, over the course of many repairs, every piece of a ship is eventually replaced. Given that, I found it interesting that this is applied to a novel greenfield OS effort, instead of a project to rewrite an existing OS in Rust piece-by-piece.
- samvher 6y agoFrom a comment by raggi below: The goal appears to be to design an architecture that supports in-place updates of most components due to them having tightly constrained state management and inter-component dependencies. Maybe that explains the reference.
- cgh 6y agoYes, after watching the presentation linked in another comment, it appears the building blocks of Theseus are small code units called "cells", which correspond 1:1 with Rust crates. These can be swapped out because the dynamic loader ensures there are no memory boundary overlaps.
- Blikkentrekker 6y agoIt's such a silly “thought experiment”. The simple answer is that “identity” is pseudoscience and does not exist; it's a man-made delusion created by the human mind to simplify reality and deal with it more easily. “identity” does not even apply to truly atomic particles, because there is no way to say that they aren't “different” in a next instance of time. There are far too many “thought experiments” and “quæstions” that simply arise by trying to treat as rigorous man-made delusions and distinctions such as this one. “Is the glass half full or half empty?” — that is a non-distinction created by men, not by nature.
- milgrim 6y agoWhy is it a silly thought experiment? It makes us aware of exactly what you are describing.
- _underfl0w_ 6y agoI was wondering how the project relates to the name - i.e. if there's some single holdover remnant or concept about it or if it's a hodgepodge of parts or something. Seems like it might be due to the modular nature of the kernel being spread across several Rust Crates. Clever reference! From the "High Level Overview" doc (not the readme) : > The Theseus kernel is composed of many small entities, each contained within a single Rust crate, and built all together as a cargo virtual workspace.
- rocqua 6y agoI think its because they want to enable hot-swapping elements of the kernel, so you could have replaced every part of the kernel without ever having to have rebooted.
- pjmlp 6y agoThankfully it isn't yet another UNIX clone. Good to explore other paths.
- deleted 6y ago[deleted]
- xiaodai 6y agoHow is different to reduxos?
- dioramayuanito 6y agoRedox OS ??? https://www.redox-os.org/ https://www.redox-os.org/
- agumonkey 6y agolinks for the lazy: - http://kevinaboos.web.rice.edu/publications.html http://kevinaboos.web.rice.edu/publications.html - https://www.youtube.com/watch?v=JWGPLVYXZlU https://www.youtube.com/watch?v=JWGPLVYXZlU
- mdtrooper 6y agoAnother "open source" OS (instead free software) warm from oven ready for steal for "big companies". When I knew about Redox OS, I thought great but has it MIT license? really? And this new OS is under MIT license too....really?
- mdtrooper 6y agohttps://discourse.redox-os.org/t/mit-licence-really-suitable/92 https://discourse.redox-os.org/t/mit-licence-really-suitable...
- WhyNotHugo 6y agoI love the idea of using Rust for this kind of thing. Is there any chance that specific drivers for modules for the Linux kernel be written in Rust? Has there been any interesting research or discussion on the topic?
- steveklabnik 6y agohttps://lwn.net/Articles/829858/ https://lwn.net/Articles/829858/
- WhyNotHugo 6y agoThanks, very interesting read!
- flaburgan 6y agoHm, why didn't they join https://www.redox-os.org/ https://www.redox-os.org/ instead?
- lvass 6y agoI wonder if writing things in Rust just to put that in the headline is becoming a viable strategy for software adoption. There is often very little reason why programming languages matters to end users, but it strongly correlates with visibility in some communities.
- lewisinc 6y agoThat, and I think it's also fair to acknowledge that language choice can be a serious barrier of entry to contributing to a project or using it in tandem with other technologies. Also if I'm looking to learn from real-world projects I would, personally, prefer to know ahead of time. "Hi, my project does X, I wrote it in Y," seems a reasonable way to introduce something.
- oregontechninja 6y agoThere is also Redox OS as another rust project. I'll edit with a link later but y'all know how to google.
- b34r 6y agoLooking at the maintainer’s comments on the open pull requests, I’m not surprised there hasn’t been a lot of progress on this project. Major attitude detected.
- rkangel 6y agoThere's an analogy here with the fact that it is so routine that we now don't remark on SpaceX recovering their first stage on every launch. We will know that Rust has succeeded when people no longer feel the need to remark on it in the title.
- smaddox 6y agoI finally finished reading the three relevant papers. This is absolutely brilliant. This isn't my area of expertise, but to me this looks like the future of OS design. I would love to use this as my base OS, if/when it can be brought to the point of being able to run virtualized linux applications. The concept of state spill also explains why OOP code bases are often a nightmare of complexity, despite pervasive modularization and encapsulation.