9 ms·
Reforming Unix
- deviantbit 3y ago[flagged]
- guerrilla 3y agoYeah, exactly... Opposed to?
- Qwertious 3y ago>How do you take someone seriously that opens with that? By reading the whole article and responding to it on its merits, and not on a single three-word sentence that isn't that relevant to the core point of the article.
- Ericson2314 3y agoThank you. The computing world loves to be relentless positive about things. Ergo all the over-promising and hype-cycles and whatnot. I don't want to descend into nihilism, but I think we need more criticism to balance it out. If people don't know what's bad, how can that appreciate what is good? A lot of people revere Bell Labs a lot. I dunno, I think the original version in Manhattan with a train running through the building might have been cooler! There was a huge diversity of computers and operating systems in the late 1960s and 1970s. It can easily be argued that technical merits had less to do with Unix's winning than the monopoly law that preventing AT&T from commercializing it. Starting with an elementary schooler's negative sentence is a good way as any to break from the relentless positivity and widespread Unix worship :)
- sam_bristow 3y agoI think the Unix model is a local maxima in the design space we've been living in for decades. It's incredible how well we've managed to scale and tweak the concepts but onto the systems we have today, but I'd love to see more experimental OS architectures again.
- Ericson2314 3y agoDefinitely more experimentation would be good. On the light experimentation front, a longstanding goal of mine is to see NixOS support multiple kernels, because the status quo of having to cobble together a userland from parts for each kernel is a huge productivity drag. Having a "userland assembly toolkit" that lets you plan out what software you want to support with what patches could dramatically lower the barrier to entry. Trying to carve out a driver layer would also be fantastic. This should be analogized to LLVM making a reusable compiler backend, and the dramatic effect that had on programming language diversity allowing things like Rust to come into existence. At the same time, kinda what I am saying here is that maybe it isn't quite a local maxima. Today most kernel work is motivated by performance, maybe security. The mistakes that are already worked around with more ugliness in other layers is usually not prioritized --- we have the workarounds, what's the problem? If more OS work was thought of in productivity terms --- we want to make writing correct, secure, performant, etc. software easier and more natural --- I think we would find the gradient might be bad in many directions, but still good in others. The stuff I include here is are all things that I think could be refactored into existing "heritage" kernels without much difficulty. I actually looked a bit at FreeBSD and Linux for the process-spawning parts, for example. Just need to slice up the fork/exec code and then call it in a different order.
- deleted 3y ago[deleted]
- pjmlp 3y agoLike this comment, anyone that is serious about UNIX, knows it is a pile of hacks, everything is a file except when it is not, and being free beer with source code is what has largely contributed to its adoption by universities and workstation vendors, saving costs on OS licenses and R&D. The UNIX creators did not come up with Plan 9 and Inferno just for fun. And to quote Rob Pike, someone that knows more about UNIX, than most folks here, maybe you will take him more seriously, "We really are using a 1970s era operating system well past its sell-by date. We get a lot done, and we have fun, but let's face it, the fundamental design of Unix is older than many of the readers of Slashdot, while lots of different, great ideas about computing and networks have been developed in the last 30 years. Using Unix is the computing equivalent of listening only to music by David Cassidy." -- https://interviews.slashdot.org/story/04/10/18/1153211/rob-pike-responds https://interviews.slashdot.org/story/04/10/18/1153211/rob-p...
- juped 3y agoYou think "true, but what does this author have to say about that?" and continue on to read the rest, that's how.
- LanzVonL 3y agoIt's simple - you don't. Just close the tab and move on.
- nialse 3y agoWhile these Unix reforms sound great in theory, they somewhat disregard Unix’s practical, evolved nature. It’s not just about clean ideas; it’s about what works in the real world, shaped by decades of use. Implementing such sweeping changes could take way longer than one would expect.
- Ericson2314 3y agoErr I meant the entire point of this to be that we can do nice things incrementally. E.g. I am linking LWN articles and whatnot to demonstrate that actually kernel devs thought some of this stuff is feasible. If anything, the heavy lift here is not even changing kernels, but reforming programming language's standard/popular libraries to make this stuff accessible and ergonomic --- most software today isn't written in C directly using some kernel-developer-maintained headers. If we can relatively make the OS changes, but it takes a bunch more slow steps for the new interfaces to percolate out "regular" developers, that adds uncertainty. "Latency" is a bigger enemy than "feasibility". You can find me jabbering away in many corners of the internet about trying to make standard libraries easier to evolve for precisely this reason. I want to keep evolving things, I am a big believe in https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/ https://www.joelonsoftware.com/2000/04/06/things-you-should-..., I just want things to evolve with a bit more foresight than the genetic algorithm can do on its own.
- pdimitar 3y agoThe author explicitly mentions Fuchsia as going too far i.e. advocates for "let's try improving things little by little". > While these Unix reforms sound great in theory, they somewhat disregard Unix’s practical, evolved nature. To me that's a polite language for: it's a wild west of "oh crap, we forgot X, guess it's time for ungodly hacks in Y and Z". I do somewhat agree with you though. Let's see what did our area's emergent behavior and hacks achieve and try to distill them in a more clean future API.
- IshKebab 3y agoThe things in this list are clearly mistakes. They aren't optimised, evolved solutions. They've just stuck around due to network effects. Nobody would design a process spawning API like fork/exec today.
- bakul 3y agoPlan9 gets a lot of this right. The author should flesh out why "Unix is bad".
- Ericson2314 3y agoPlan 9 made a number of improvements, but while "everything is a file" is an improvement, "everything is a file descriptor" think is better. namespaces aside, files as global variables, while file descriptors are local variables. It is much more easy to make an interface that "naturally" makes it easy to abide by https://en.wikipedia.org/wiki/Principle_of_least_privilege https://en.wikipedia.org/wiki/Principle_of_least_privilege with local variables. Unix < Plan 9 < Capiscum-like designs, for me. (Of course, if Capsicum is "capability-based security for Unix", maybe a "capability-based security for Plan 9" is better than that. :))
- bakul 3y agoPlan9 in fact uses file descriptors all over the place, including accessing synthetic or real filesystems (which are really namespaces). In fact the only "globals" are the #<dev> devices. IMHO you're better off starting with plan9 concepts + pure capabilities (KeyKOS as opposed to Capsicum). It would not enough to have a clean architecture unless a) it matches or improves upon performance and b) there is a way to run existing Unix software via some adapter layer or library or monitor. I suspect if you try to "reform" unix, there will be a lot of recidivism :-)
- ksjskskskkk 3y agocapros. keykos is so 1980. i miss when osnews was still decent.
- Ericson2314 3y ago> recidivism Yes, it is not enough to just add yet more interfaces. Using personalities to ban the use of deprecated systemcalls is an essential follow-up step. Fostering experimental kennels which don't bother to support the old interfaces at all is another.
- boricj 3y agoI've wondered if process creation on Unix could be overhauled by extending some of the existing syscalls to operate on a separate process rather than the current one: // dup2() a file descriptor to the destination process int rdup2(int procfd, int oldfd, int newfd); // mmap() at the destination process address space void *rmmap(int procfd, void *addr, size_t length, int prot, int flags, int fd, off_t offset); // create a thread remotely int rpthread_create(int procfd, pthread_t *restrict thread, const pthread_attr_t *restrict attr, void *(*start_routine)(void *), void *restrict arg); ... With this model, creating a new process would start by creating an empty, suspended one, then populating the various pieces with the returned process file descriptor: * file descriptor table, * address space, * stack, * threads... Once everything's in place, the process would be unsuspended with one final syscall and start execution.
- saurik 3y agoWelcome to mach!
- Joker_vD 3y agoYou mean, like DuplicateHandle, MapViewOfFile2, and CreateThreadRemote from Win32 API? Well, to be fair, it has some merit: UNIX processes are possibly the only resources/kernel objects across any existing OS that can be created only by copying an existing one and then fiddling with the copy to make it look right. That's quirky. And if we have "procfd"s, then the need for process 1 (and the rules about a parent process having to call wait() for its children) goes away, just like on Windows: if the parent P of a would-be zombie process Z closed the procfd of Z, then when Z terminates the refcount of its process entry in the process table drops to zero and it gets cleaned up, freeing the Z's globally-visible PID, along with other B's resources like disk files and sockets, without P having to do anything.
- boricj 3y agoI was actually inspired by the Zircon system calls from Fuchsia, which is a capability-based system. Operating on the current process or another one is just a matter of passing the correct handle to the relevant system calls. I'm not familiar with the Win32 API, but I'm not surprised to learn that it has similar facilities (although MapViewOfFile2 appears to be a fairly recent addition). I'm of the opinion that POSIX is a fossilized design kept on life support, no longer fit for purpose in the modern era. Modern Unix-like are trying to plug in the holes with mostly proprietary extensions and this proposal would be just another one added to the pile... Whatever the future is made of, I do hope we won't have to carry 50 years old design mistakes for another 50 years from now.
- gigatexal 3y agoSo plan9?
- speed_spread 3y agoWell yeah. You already could rebuild most of Plan9 using a standard Linux kernel and a mutated userspace to take advantage of advanced features that are usually ignored by glibc and friends.
- mcny 3y agoDid you accidentally a word here? > Remember, the file system is a flavour of database. We shoudn’t just it by other standards.
- defrost 3y agoFWiW, my best guess: s/just/judge/
- kaoD 3y agoPossibly just -> judge?
- Ericson2314 3y agoYup you are right. Will fix.
- rollcat 3y agoGo already implements many of these ideas. There's os/exec (and syscall.ForkExec), and io/fs (albeit it's a family of interfaces, rather than a descriptor you could pass between processes). It had to ban forking, because of the guarantees made by the runtime (and all the "weird" OSs it runs on, like Windows). The API is clean and easy to work with, but don't look at the implementation ;) I'm curious what would "UNIX v3" look like today, if made by the same teams that did UNIX, and then P9/Inferno/Go.
- Ericson2314 3y agoYes being able to seamlessly swap out the implementation of library functions is a crucial way to justify this sort of work on a shorter timescale. I'm very glad that many such libraries already have process creation interfaces that possible, and in fact easier, to implement with this approach.
- aryonoco 3y agoThis reminds me of the Unix Haters handbook. a favourite topic back in the 90s amongst certain groups. https://web.mit.edu/~simsong/www/ugh.pdf https://web.mit.edu/~simsong/www/ugh.pdf I'd encourage the author to have a look at Plan 9 and GNU Mach as well, which in some shape or form do implement many of these. But none of them ended up being a viable reality.
- Beijinger 3y agoThe best part is still: I have succumbed to the temptation you offered in your preface: I do write you off as envious malcontents and romantic keepers of memories. The systems you remember so fondly (TOPS-20, ITS, Multics, Lisp Machine, Cedar/Mesa, the Dorado) are not just out to pasture, they are fertilizing it from below. Your judgments are not keen, they are intoxicated by metaphor. In the Preface you suffer first from heat, lice, and malnourishment, then become prisoners in a Gulag. In Chapter 1 you are in turn infected by a virus, racked by drug addiction, and addled by puffiness of the genome. Yet your prison without coherent design continues to imprison you. How can this be, if it has no strong places? The rational prisoner exploits the weak places, creates order from chaos: instead, collectives like the FSF vindicate their jailers by building cells almost compatible with the existing ones, albeit with more features. The journalist with three undergraduate degrees from MIT, the researcher at Microsoft, and the senior scientist at Apple might volunteer a few words about the regulations of the prisons to which they have been transferred. Your sense of the possible is in no sense pure: sometimes you want the same thing you have, but wish you had done it yourselves; other times you want something different, but can't seem to get people to use it; sometimes one wonders why you just don't shut up and tell people to buy a PC with Windows or a Mac. No Gulag or lice, just a future whose intellectual tone and interaction style is set by Sonic the Hedgehog. You claim to seek progress, but you succeed mainly in whining. Here is my metaphor: your book is a pudding stuffed with apposite observations, many well-conceived. Like excrement, it contains enough undigested nuggets of nutrition to sustain life for some. But it is not a tasty pie: it reeks too much of contempt and of envy. Bon appetit!
- BarbaryCoast 3y agoMach was created at CMU, and later development moved to the University of Utah. This work became the basis of the NeXT operating system, which then turned into Mac OSX. So I'd call this a "viable reality". PS -- I think you meant "GNU/Hurd", not "GNU Mach".
- deleted 3y ago[deleted]
- amelius 3y ago> Unix is bad. But file descriptors are good. Modern Unix-derived operating systems (Linux, some BSDs) are already moving towards file descriptors for everything. Perhaps we should just call them "object handles" instead.
- Ericson2314 3y agoFine with me!
- alsetmusic 3y ago> Remember, the file system is a flavour of database. We shoudn't just it by other standards. A word appears to be missing here. A letter, too, but I don't care about that.
- Uvix 3y ago“Just” is probably supposed to be “judge”.
- Ericson2314 3y agoYup, and I received a PR for this :) thanks whoever sent it. Should be fixed now.
- gavinhoward 3y agoWow, this is awesome. I'm working on a design for a next-gen OS, and I already have some things that are in the article, like starting a blank process to be mutated before start [1], file descriptors as handles/capabilities, and such. But there are some other things I don't have that I like. For the bootstrap problem, I realized something that the author did: the better interface is generally more powerful: > When requests and responses are paired 1-1, the asynchronous APIs offer perhaps performance improvements but no expressive power. When they are not 1-1, however, things get more interesting. Then I realized that if an OS is designed right, it will generally have more expressive interfaces, and those expressive interfaces can then be used to implement the APIs from other operating systems. So I intend to make a microkernel that implements the APIs for the popular operating systems. Then I will make a userspace driver shim that implements the Linux driver API. Boom! Instant OS and driver availability. Done right, Linux drivers will work, and software from the major operating systems will work. And then I will encourage people to use the better APIs when on my OS until everybody is on my OS. Embrace, extend, extinguish, but in a good way! :) Will it work? Oh, heavens no. But as some comments have said, experimentation is a good thing. [1]: https://news.ycombinator.com/item?id=35681941 https://news.ycombinator.com/item?id=35681941
- ksjskskskkk 3y agobro is reinventing windowsNT/os2
- thecodedmessage 3y agoIf at first you don’t succeed with a big company’s resources, try again as a hobbyist!
- BirAdam 3y agoDefinitely more NT, but he’ll hopefully make better use of the “personalities” than Microsoft ever did. Also, NT is mostly just VMS. So much so, that DEC threatened litigation which resulted in Microsoft supporting Alpha for longer than they’d otherwise have cared for.
- Ericson2314 3y ago
- jasoneckert 3y agoI stopped reading after the line "Unix is bad." Bold sweeping statements like this are a red flag - they tell the reader that the remaining content is likely subjective and offers little value. From reading the other comments here, it appears that there are some valid Unix criticisms and improvements (all of which have been discussed previously on HN), but Unix is certainly anything from bad. And like anything, it can always be improved. Unfortunately the author didn't take this angle when writing the first line of this post.
- layer8 3y agoThis seems like a subjective comment of little value.
- jasoneckert 3y agoHow is this comment subjective and of little value?
- Koshkin 3y agoNever mind. The parent comment was subjective and of little value.
- MCUmaster 3y ago[flagged]
- bborud 3y agoMe 25 years ago, spending my days implementing mostly server applications in C and trying to design things in mechanical sympathy with the OS, would probably have had a lot of opinions on this. Mostly "don't make my life harder by changing things around too much". Me today, still writing server applications, but now in Go, lives somewhat more remotely from the low level details of the system. There are more abstractions between me and the system. It is rare that I even have to deal directly with, for instance, low level networking system calls (except the odd ioctl, which, as someone pointed out years ago, is the carpet under which we sweep all the things that we couldn't figure out how to do properly). During those 25 years the OS went from something I had to care about to something that I only cared about when something didn't work. I'd love to say that things got better, but somehow, unix managed to attract a lot of people who never understood what made unix good in the first place: simplicity. Which leads to three thoughts. The first thought is that I both think it is possible and desirable to evolve towards something that simplifies writing libraries and system software a bit. The vast majority of development work is not really affected since we don't typically spend our days obsessing over system calls. Heck, most people who develop on unix today are not going to be able to distinguish a system call from a library call anyway. We don't really spend a lot of time doing low level stuff. And those who do probably wouldn't mind as much as we think. Provided things don't get worse. Which leads me to my second thought. The second is that we should be careful about putting too much stuff in the OS that really belongs in libraries or applications. Not that I see this here, but talking about filesystems as databases and offering transaction semantics is right on the edge. It can easily become complex and hard to implement correctly. (Yes, I'd love to have better defined filesystem IO with certain transactional capabilities. But I really would rather we didn't even try if it means there is a risk people will overdo it and I have to spend the next decade tolerating that my filesystem craps out on me more often than is the case now) Operating systems shouldn't do anything too complicated. Complicated isn't robust. Do the complicated stuff in userspace where it doesn't matter as much when it blows up. Third, learn from the last couple of decades of poor system software. Managing unix used to be easy. Every part of the system used to have a single purpose and you could figure stuff out. These days I'm lucky if I can figure out how to configure which DNS servers to use and make that config stick. My system is full of overly complex system software that is poorly implemented and has horrific configuration regimens. I haven't been a sysadmin for 20 years. If I have to give a shit about these things, it means they are poorly designed, implemented or both. In the last 20 years a lot of people got it in their heads that they were going to improve linux by designing and implementing ambitious system software to try to configure networking and daemons. And while I understand the temptation and I'd probably have committed similar atrocities if I'd been sufficiently bored, it is okay to admit that for the most part, we didn't get much net benefit. So my third point kind of expands to: "don't try to be too clever" and "if possible, see if we can make more system software obsolete while we're at it". EDIT: fixed typo
- auselen 3y agoWouldn’t creating new processes from zero rather than inheriting from parent make it hard to do things like ‘sudo’?
- wmf 3y agoroot and suid are big hammers that should be eliminated. In a capability world maybe you can send a message to some more powerful process to ask it to do things for you.
- auselen 3y agoI also had in mind features like cpu affinity, and may be the next thing that this app which compiled a decade ago didn’t know about. I don’t like current forking for sure, just that I’m not against inheritance…
- Ericson2314 3y agoI think it would be possible. There's the version where you start with a blank file and mmap as you please, and the version where you still start with by specifying an executable and then set the other state. The latter version should work just fine with setid. Agreed with the others that setid is bad and ultimately should be gotten rid of, but we do need to support it for a transitional period. So thanks, good question.
- ajross 3y agoI dunno. I think I can get behind the ideas here, but a lot of it seems a little too informed by app requirements and not the bits (hardware abstraction) that are the actual job of an operating system. Discussing better ways to spawn a process is great, and "everything is a file descriptor" is a fine metaphor. But if there's no thought given to mapping memory, handling virtualized machines, driver interaction (ioctls, sysfs, et. al.), async signaling and IPC, etc... I don't know that this really gets very far. You could write these APIs on top of the Linux kernel today, but you'd never be able to do anything but toy code without then wrapping a vast quantity of system interfaces in this "fd's everywhere" model.
- LanzVonL 3y ago[flagged]
- hulitu 3y ago> Unix is bad. No. Doing one thing well and combining things is good. The freedesktop parody is bad. It is a solution looking for a problem. Why do i need d-bus and gnome-keyring to authenticate an authenticated user ?