7 ms·
Interview with Andreas Kling of Serenity OS (2022)
- Ezhik 3y ago> creating something strongly influenced by the past, but in a modern way, and in a sort of a high fidelity way with 2020 hindsight This is a great take on retrocomputing and tech nostalgia. Thank you Andreas for always being a shining reminder of that innocent love for personal computing I almost lost.
- josephg 3y agoI'm increasingly tempted to try my hand at making a toy OS - or at least a kernel. There's a few things I desperately want an operating system to do: 1. Have either database-like transactions or (at a minimum) write barriers for file operations. The current filesystem semantics are terrible for databases and other applications which need to not corrupt their data after a crash. fsync() is too heavy, because you can't issue subsequent writes while waiting for fsync. Write barriers have better ergonomics (so, less complex, less buggy database implementations). And they're finer grained - so databases can also run much faster. Seriously, I could rant for hours about this. It drives me nuts. 2. Use event queue like semantics for subscriptions. There's dozens of OS APIs in which the OS exposes some data to applications where the data changes over time. Eg, filesystem watching, USB device insertion/removal, Bluetooth, watching a network device, or a socket, or APIs for htop to see the set of active processes. Each of these APIs has an entirely bespoke - and oftentimes idiosyncratic - API to get changes to that data over time. Sometimes you need to poll and parse a file in procfs. Sometimes there's syscall based APIs. And some of the subscription APIs are just broken sometimes, and they'll fail to notify you about changes sometimes. I want to try replacing all of that with a simple, unified API which lets you say "tell me the state of <kernel resource X> and tell me when it changes". The kernel then replies with the object's current state and I get messages notifying my application about changes. We should be able to use the same API for every type of kernel object. It would also be interesting to try making all userspace programs be wasm bundles and run them in ring0. But I suspect that will run slower on modern hardware compared to using native binaries and context switching. ... Maybe. It might depend on the application. I think this stuff could be done in linux. Event queue style subscriptions would work especially well with io_uring. But I'm kind of excited by the idea of just noodling myself and seeing where I end up. Making stuff is fun.
- actionfromafar 3y agoSome of the issues can be improved "hackily" by for instance giving a database software exclusive access to two discrete disks - one for the db file, one for the journal.
- shiroiuma 3y agoWhy is this "hacky"? I've long wondered why databases need to be on filesystems, other than simply convenience. If you think about it, a filesystem is, itself, basically a database (not a relational one of course). So it seems like you could improve performance a lot by eliminating the filesystem layer and going straight to direct disk access. Also, how this would be done would probably change depending on whether you're using a spinning-rust HD or an SSD, or a RAID array.
- josephg 3y agoLots - maybe most - applications need to essentially run their own mini database to store user data. I obviously don't want a partition per application, or per word document. Abstracting away your hard disk is literally the job of your filesystem.
- shiroiuma 3y agoI don't mean that some little SQLite DB for your web browser's storage and settings needs its own partition; that's obviously going too far. I mean for really huge databases, where an entire server is dedicated to running that DB and it has high performance needs.
- josephg 3y agoFair enough. But I am including all of those little databases. I agree with you - you could hack around this for a big database by putting it on a dedicated partition (even though you shouldn't need to). But I'm also thinking about applications with little databases that need to live inside the filesystem. They shouldn't need to embed sqlite in order to survive a crash. A different syscall API would allow you to write a small, fast, efficient database engines (like redis or mysql) without needing all of the complex, performance gobbling tricks that these databases currently have. A better filesystem API would help databases everywhere.
- 0xEF 3y agoFor those looking to try their hand at building an OS, there's Linux From Scratch. https://www.linuxfromscratch.org/ https://www.linuxfromscratch.org/ I failed tremendously, but I had a lot of incorrect ideas about what makes an operating system go. The learning experience was worth it, though.
- quintussss 3y agoBut isn't this just about compiling the system? Not really 'building', ie programming a operating system.
- sph 3y agoThat teaches how to compile Linux kernel and software, not to create an OS out of thin air. The go-to guide for anyone that wants to build their own operating system has always been the OSDev community wiki and forum: https://wiki.osdev.org/Main_Page https://wiki.osdev.org/Main_Page
- PhilipRoman 3y agoTo be fair there is a lot you can do in userland. Graphics protocols, package management, isolation, security etc. And even more can be done by incrementally patching the kernel where needed. It's just that Linux is kind of usable out of the box with just the kernel and /bin/sh. But for other OSes this isn't true, the kernel can expose a vastly different interface than the actual os which users interact with.
- swozey 3y agoI've done every bit of the kernely/cgroupy/bpf/vfio/syscall stuff for a long time and I honestly don't think I could tell you pretty much anything about building an OS. I actually came into this thread thinking, "damn, I have no idea specifically WHAT I would do/not do if I made an OS, or where to even start" not that I could at all. But I mean feature-wise, etc. Like, it's obviously a ton of apis that you call to invoke things at the system level but I have no idea what to do at the system level. Maybe you just communicate with APIs over there for everything on the motherboard. I know Nvidia has NVAPI I've used that a bunch through their SDK, not more low level. And you start with probably a super baby kernel.. There are also different styles of OS, like windows with the registry which I have broken and fixed thousands of times, and then nix which is pretty much file based. I don't know what else would be better than either of these, I hate the registry thing but maybe it has some huge benefit I haven't figured out. When I look at C I see heiroglypics. Maybe me learning rust is helping with that. I do really need to learn it for security sake and (live)patching, etc. I write go and ruby so I don't touch memory much. I'm the same about filesystems. I know a bunch of them and hve done tons of performance testing for SANs etc but couldn't tell you a thing about the lower level, making one, stuff. edit: lol I guess a lot of this is installing drivers too
- torginus 3y agoI just wonder how much of the truly exceptional engineering in today's software was done by people on drugs (yes, 'ADHD' medication counts). I fully expect it to be a significant chunk, but I hope it's not the majority.
- nielsole 3y agoCore to Andreas' story is that he did his engineering as an escape from drugs, not the other way around as you allude to.
- dizhn 3y agoHe was a pretty high level professional programmer before he started coding serenity as therapy. I don't know the exact timeline of course.
- mysterydip 3y ago> 'ADHD' I'm confused by the use of quotes. Do you think ADHD is a made-up thing?
- crimsontech 3y agoI took it to mean people using it when they don’t have ADHD.
- mysterydip 3y agoAh that makes sense, thanks!
- andoando 3y agoI am diagnosed and I think it is
- dmurray 3y agoI suspect whether it's a majority or not depends on how you count caffeine.
- SSNLF 3y agoI love this but just looking at the comments - while admittedly cool, implementing your own OS isn't black magic or particularly arcane knowledge; sure it's a whole lot of work where you're often reimplementing the wheel, but not outside the realm of any reasonably skilled graduate I would think.
- sph 3y agoWriting an OS is not any harder than writing a database or a programming language. The complexity lies in how far do you want to go with it. While not black-magic, dealing with hardware and its quirks is a different skillset than writing React logic. A bug might need a reboot, or in some rare cases with terrible hardware, turning it into a paperweight. In fact, I believe that learning to write low-level code for simple hardware (say, an Arduino or a RP2040), is a smoother path for a complete newbie than learning the massively complex frontend world, and provides much better foundational knowledge.
- Dessesaf 3y agoThe really challenging part when you're building an OS comes from not building an incrementally bigger and bigger mess until you're stuck and can't make any progress any more. You're right, there's no magic here. It's just lots and lots of work. And there are important decisions to be made about how you design and abstract everything. None of them are hard, but making so many decisions is the very thing that's hard about most programming. The answer of course is to have lots of programming experience. Which is why building an actual OS to the level of SerenityOS is not something a reasonably skilled graduate would be able to do.
- adamgordonbell 3y agoPreviously https://news.ycombinator.com/item?id=30214371 https://news.ycombinator.com/item?id=30214371
- symmetricsaurus 3y agoShould have a (2022) perhaps this post?
- alberth 3y agohttps://www.redox-os.org/ https://www.redox-os.org/ Redox is a super interesting project. Not because it’s Rust based. But because rarely do people start from scratch to build a new OS. (eg, OpenBSD forked NetBSD, Dragonly forked FreeBSD, Ubuntu from Debian, etc) —- https://fuchsia.dev/ https://fuchsia.dev/ Would be another of interest.
- nwah1 3y agoGood recommendations. OSes today are more than kernels. Filesystems, init systems, shells, and the configuration conventions would all be part of it. And probably also the windowing system and desktop environment. The most innovative concepts historically would be Plan9, BeOS, and seL4. Plan9's approach of "everything is a file" is something that was unique. BeOS's extensive use of journalling and metadata in the filesystem gave it big advantages. seL4 being the most rigorous application of a micro-kernel architecture, and making use of formal verification make this one stand out. Capability-based operating systems also offer immense benefits. Fuchsia and Redox are clearly drawing some inspiration from these. Along with GNU Hurd, which you didn't mention. As for shells, nushell is clearly revolutionary. Something that points to what a better world might have been. I would also recommend looking at Lennart Poettering's vision that extends on innovations like cgroups, journalling filesystems, ostree, etc. https://0pointer.net/blog/projects/stateless.html https://0pointer.net/blog/projects/stateless.html And of course, Nix and Guix both align with this vision of reproducibility, statelessness, and transactionality... along with Fedora Silverblue. I kinda think of this as making the filesystem work more like git. A distro is a branch. A rollback is a branch from a previous commit. And then the build system would be completely idempotent... every compilation from any system would be byte-for-byte identical, and thus verifiable in a security sense. And every upgrade on any system could be undone.
- openquery 3y ago> But at least in my case, whenever I tell people about some idea, I kind of tend to lose interest in the idea. So I just went to work on it instead. I think that's great advice.
- JohnFen 3y agoI agree. I've learned that it's best if I don't tell people my ideas until I've at least started to work on them unless I need to pick someone's brain.
- beardedetim 3y agoI read somewhere long ago that there's some chemical released when you talk about your plans, similar to the chemical that gets released when you complete your plan. So when you talk about your ideas, you feel like you've already accomplished them. Ever since I heard this, I've been protective of what I talk about and it seems to have helped me ship more things than before.
- mewpmewp2 3y agoOn the other hand sometimes it can make you commit something to someone. It really depends.
- 01HNNWZ0MV43FF 3y agoI also noticed people don't get impressed when I talk about my ideas, but they are impressed when I show screenshots
- andoando 3y agoFor me I feel like a large part of it is that no one else shares the same excitement that I do. If I share something I think is really cool and everyone is like “I don’t get it”, those thoughts inevitably just murk my brain.
- drannex 3y agoFigured I would write this here, but I would scratch the concept of a file system and replace it with a database.
- jerrygenser 3y agoIsn't that how z/os from IBM works? The API is a SQL api into files I think
- drannex 3y agoAFAIK it still uses a classical file system, just with a DQL system layered on top. I am talking about completely scrapping the idea of file systems, folders, the lot of it. A central database that stores all blobs, configurations, info, and other metadata (modified, created, revision history, diffs, keys, tags, and authority). This would enable real time distributed scalable systems, potentially immutable backups, and a lot more (such as fully auditable data transfers, messaging, access, and more for rapid debugging). If an operationg system was built today for tomorrow it would certainly be built on/as a database.
- p_l 3y agoYou're mixing up IBM z (mainframes, z/OS) with IBM i (AS/400). The latter has a single level storage system where disks and memory are part of single address space, and a variant of DB2 database is part of core OS thus making it common to use SQL for many things people wouldn't expect.
- thequux 3y agoZ doesn't really have files or databases, but rather an emulation of variable-length records contained within a relatively fixed allocation on disk. Mainframe DASDs also have a much richer interface than just read/write a given block; for example, you can scan for records within a track that have a given prefix
- Pet_Ant 3y agoI would like to scrap the idea of a file and replace it with an object. If it's a "file" it just regurgitates it's contents, but it could also return aspects of itself, decrypt itself, stream from an offscreen resource. Just let them be smart, and what we know as sending "files" would just become object serialisation.
- JohnFen 3y agoI've built three over the decades, but each for very specific use cases and with very specific requirements. It's usually better to use an existing, established OS -- or none at all, depending. But sometimes, a special-built solution makes the most sense.
- eternityforest 3y agoI'd focus on high level APIs, managed code and JIT, and stuff like that, and everything would be written in Rust plus a scripting language for everything that isn't performance critical. More realistically, I just wouldn't build an OS either. That would be a nightmarishly large project!
- wglb 3y agoIf I were so motivated, I would start with the principles outlined in https://en.wikipedia.org/wiki/THE_multiprogramming_system https://en.wikipedia.org/wiki/THE_multiprogramming_system. Using Windows NT or Window 95 as an inspiration would make the task much harder. NT is an absolutely massive, full features, in some sense, and extension of Vax VMS, which itself is extremely feature rich. I would take the challenge to build something that is the simplest thing that could possibly work.
- Joel_Mckay 3y agoConsidering the use-cases, multi-tasking operating systems are not necessarily going to escape the trend toward information appliances. In my opinion, if the workstation does survive, than a posix style OS similar to the abandoned HeliOS will become necessary again at some point... as Moore's law is effectively dead, and a 15% context-switching overhead on existing threading and mailbox models is silly. Would I want to write one from scratch? https://www.youtube.com/watch?v=lITBGjNEp08 https://www.youtube.com/watch?v=lITBGjNEp08 Have a great day, and remember to have fun =)
- cyberax 3y agoAnother suggestion: replace syscalls with io_uring-like command queues.
- cedws 3y agoApparently io_uring has been a security nightmare to the point that Google disabled it. I don't know the details but you'd definitely want to ensure you aren't repeating those mistakes.
- cyberax 3y agoIt's a new way to submit commands, and there are tons of teething issues. Especially considering that Linux had not been designed to work with async APIs from the beginning. These are not fundamental issues, but definitely something that should be kept in mind.
- BMSR 3y agoI would not. I would use Linux and its ecosystem as a base. Use a very hackable window manager that allows me to customize the window management and overall experience to my liking. Write a collection of scripts and programs that allow me to do things faster. Maybe release a distro with all of this together ready to use.
- morphle 3y ago“An operating system is a collection of things that don’t fit inside a programming language; there shouldn’t be one” - Dan Ingalls 1981
- spankalee 3y agoThat's a really terrible definition that dates from before multitasking, GUIs, and window systems and desktops.
- aidenn0 3y agoHuh? Multitasking dates back to the time-sharing systems of the '60s. Granted, while windowing systems existed in '81, they weren't widespread yet (Wikipedia lists PERQ as the first commercially released GUI workstation in '79).
- morphle 3y agoDan Ingalls wrote this in the Byte Magazine [1] dedicated to Smalltalk-80 in 1981. Smalltalk is its own operating system, it ran on the 1973 Alto that had 16 core processor and was multitasking. It was the first Graphical User Interface (GUI) that introduced the concept of desktops and window systems to the world. So his observation definitely did not date from before multitasking, GUIs, and window systems and desktops. Smalltalk and its current incarnation Squeak and SqueakNOS (for Squeak No Operating System) is a virtual machine written in a high level programming language that contains everything that we now call an operating system including its device drivers. It runs on bare metal hardware. Alan Kay his research group [2] attempted to make an entire personal computer operating system in just 10.000 lines of high level programming language and demonstrated that it is possible to do [3]. [3] answers the question "How would you build an operating system?" from OP. [1] https://archive.org/details/byte-magazine-1981-08 https://archive.org/details/byte-magazine-1981-08 [2] https://tinlizzie.org/IA/index.php/Papers_from_Viewpoints_Research_Institute https://tinlizzie.org/IA/index.php/Papers_from_Viewpoints_Re... [3] https://www.youtube.com/watch?v=ubaX1Smg6pY https://www.youtube.com/watch?v=ubaX1Smg6pY
- 3y ago
- cedws 3y agoSerenity is one of the only recent OS projects that I think has any potential. It's a very high velocity, high quality project and I think that's down to the community culture Andreas has fostered. They aren't a load of fuddy-duddies who insist on using mailing lists and IRC. Nobody wants to collaborate with a load of grumpy greybeards.
- asddubs 3y agoI think ladybird has unfortunately sucked most of the momentum out of the project.
- TehCorwiz 3y agoI disagree. It’s driving progress in GPU support and acceleration, Image format support, standard lib stability. These all happen under the hood of the os and aren’t particularly flashy. But, the architecture of Serenity and Ladybird means that a lot of the core lib stuff is immediately available to the rest of the is and user space.
- rujkking 3y agoYou think a throw back Win 95 clones with UNIX-like internals is what modern users want? We’ve moved on from “back in my day, the 80s…” computer UI nostalgia to 90s it seems. 10 more years we’ll have Mac OSX Snow Leopard clone project get abandoned (have had a lot of young Millennials and older GenZ hackers talk it up at me as peak computing) Just because they’re not grumpy doesn’t mean they aren’t nostalgia driven iconoclasts making serenity out of disdain for status quo
- r14c 3y agohttps://src.ix.cyb.red/tomo-el-fuego/ https://src.ix.cyb.red/tomo-el-fuego/ my hobby os is a fork of inferno, but mostly for dis and the ports structure. i've been slowly working towards converting the userspace into something akin to the old symbolics systems. basically a unikernel/language kernel with AOT bytecode and optional JIT. the project is heavily inspired by the s/m lang ideas originally proposed with LISP. i had never done osdev-level c-programming when i started on the project, but its been a really fun learning experience.
- m3kw9 3y agoHow would you build a minimal OS that can run just the simplest program you can think of? We start with the simplest program and work our way to what the OS needs. I’ll start: a program that asserts an 8bit number
- eska 3y agoThat’s how games used to work. The “OS” was just some middleware library.
- King1st 3y agoWith the power of God and Schizophrenia on my side. https://templeos.org/ https://templeos.org/
- skizm 3y agoHow does one get started on a toy OS? I don’t mean like concepts, but is there like a minimum viable OS .c file somewhere I can look at, with instructions for how to run it on “bare metal”? I assume I need a bootloader of some kind first? I’m just totally unsure where I’d even begin in a practical sense.
- tonetegeatinst 3y agoI think tinycore would be your best bet. Its linux but stripped down to the very bare essentials
- filleduchaos 3y agoTiny core Linux is far too heavy for what they're asking for. It's the very bare essentials for a user, not a bare metal programmer. The OSDev wiki has lots of actually minimal examples for different architectures and boards
- adhamsalama 3y agoxv6 is a small one built for operating systems courses. We added functionality to it in my college operating systems course as a project.
- gjvc 3y agohttps://www.osdev.org https://www.osdev.org
- eric__cartman 3y agoThat wiki was invaluable to me when I took computer organization courses. My recommendation would be to learn x86 assembly and C (also familiarize yourself with gdb!) by writing programs that run on a Linux system and only after target bare metal programming by using something like QEMU + gdb or Bochs for emulating bare metal hardware that's also debuggable. (I used x86 assembly as an example assuming you wanted to boot your OS on a PC. An alternative could be to target a well documented SBC like the BeagleBone Black)
- rujkking 3y agoWhat I want is to be free of the desktop metaphor and the notion my computer is an office job data entry terminal only. Am working on a GPU accelerated application to replace Gnome. Currently boots to a blank 3D canvas, like opening a new Blender project. Am working on using AI and a planner system to generate a viable user interface given user prompt. Users fill out the plan with possible states and relevant actions, go back and forth with AI to get there. All runs locally and is loaded by x11 instead of the usual desktop options.
- gjvc 3y agoGet some inspiration from something like https://www.haiku-os.org/ https://www.haiku-os.org/ https://www.vitanuova.com/inferno/ https://www.vitanuova.com/inferno/ https://9front.org/ https://9front.org/ or even https://vmssoftware.com/ https://vmssoftware.com/ and consider how one might re-imagine compute provisioning to scale across hardware but also provide services for workstation use. https://osdev.org/ https://osdev.org/ is an excellent starting point, and if you start to write code to sketch out the components, it's surprising how many features are required. (Not as monumental as writing a database, but nevertheless surprising.) Going in other directions, consider how Smalltalk, and the Lively Kernel do things. Gaze in wonder at the power of inertia, and network effects. Other operating systems are available.
- RecycledEle 3y agoI would have a UX processor that handles video, sound, and keyboard/mouse inputs. That system would be very fast so the user would never again click only to see the click register on something else.
- dang 3y agoRelated ongoing thread: The Ladybird browser project - https://news.ycombinator.com/item?id=39271449 https://news.ycombinator.com/item?id=39271449 - Feb 2024 (228 comments) OP discussed at the time: Serenity OS: Interview - https://news.ycombinator.com/item?id=30214371 https://news.ycombinator.com/item?id=30214371 - Feb 2022 (52 comments) Corecursive: Serenity OS with Andreas Kling [audio] - https://news.ycombinator.com/item?id=30176123 https://news.ycombinator.com/item?id=30176123 - Feb 2022 (3 comments) There's a big list of related links at https://news.ycombinator.com/item?id=30858768 https://news.ycombinator.com/item?id=30858768 but it's a couple years old now.