4 ms·

Me 25 years ago, spending my days implementing mostly server applications in C and trying to design things in mechanical sympathy with the OS, would probably ha

by bborud 3y ago

Me 25 years ago, spending my days implementing mostly server applications in C and trying to design things in mechanical sympathy with the OS, would probably have had a lot of opinions on this. Mostly "don't make my life harder by changing things around too much". Me today, still writing server applications, but now in Go, lives somewhat more remotely from the low level details of the system. There are more abstractions between me and the system. It is rare that I even have to deal directly with, for instance, low level networking system calls (except the odd ioctl, which, as someone pointed out years ago, is the carpet under which we sweep all the things that we couldn't figure out how to do properly). During those 25 years the OS went from something I had to care about to something that I only cared about when something didn't work. I'd love to say that things got better, but somehow, unix managed to attract a lot of people who never understood what made unix good in the first place: simplicity. Which leads to three thoughts. The first thought is that I both think it is possible and desirable to evolve towards something that simplifies writing libraries and system software a bit. The vast majority of development work is not really affected since we don't typically spend our days obsessing over system calls. Heck, most people who develop on unix today are not going to be able to distinguish a system call from a library call anyway. We don't really spend a lot of time doing low level stuff. And those who do probably wouldn't mind as much as we think. Provided things don't get worse. Which leads me to my second thought. The second is that we should be careful about putting too much stuff in the OS that really belongs in libraries or applications. Not that I see this here, but talking about filesystems as databases and offering transaction semantics is right on the edge. It can easily become complex and hard to implement correctly. (Yes, I'd love to have better defined filesystem IO with certain transactional capabilities. But I really would rather we didn't even try if it means there is a risk people will overdo it and I have to spend the next decade tolerating that my filesystem craps out on me more often than is the case now) Operating systems shouldn't do anything too complicated. Complicated isn't robust. Do the complicated stuff in userspace where it doesn't matter as much when it blows up. Third, learn from the last couple of decades of poor system software. Managing unix used to be easy. Every part of the system used to have a single purpose and you could figure stuff out. These days I'm lucky if I can figure out how to configure which DNS servers to use and make that config stick. My system is full of overly complex system software that is poorly implemented and has horrific configuration regimens. I haven't been a sysadmin for 20 years. If I have to give a shit about these things, it means they are poorly designed, implemented or both. In the last 20 years a lot of people got it in their heads that they were going to improve linux by designing and implementing ambitious system software to try to configure networking and daemons. And while I understand the temptation and I'd probably have committed similar atrocities if I'd been sufficiently bored, it is okay to admit that for the most part, we didn't get much net benefit. So my third point kind of expands to: "don't try to be too clever" and "if possible, see if we can make more system software obsolete while we're at it". EDIT: fixed typo

BirAdam 3y ago
Same. The only thing I’d say in contrast is that while getting worse to manage, a lot of stuff works more often than it used to do. Now, I’ll expand and add a fourth point. Do not have so much hubris as to think that you can achieve in one year what took 40 years to build, and certainly do not think that just because you have better tooling you will be able to reimplement something better than what has already been done. The reality is that the new implementation will likely suck in comparison to the old one for many years, and it will require ceaseless and thankless effort. Prime example: Wayland (which is now really awesome btw) that took a decade or so. If, otoh, something sucks and has done for a long time, go for it. Prime example: Linux audio (OSS, Alsa, Pulse, Jack) where pipewire is a godsend.
Ericson2314 3y ago
> Me today, still writing server applications, but now in Go, lives somewhat more remotely from the low level details of the system. There are more abstractions between me and the system. It is rare that I even have to deal directly with, for instance, low level networking system calls (except the odd ioctl, which, as someone pointed out years ago, is the carpet under which we sweep all the things that we couldn't figure out how to do properly). This is both a disadvantage and a major advantage. The disadvantage is that yes, changing syscalls / libc is not enough. No one is going to hand-write the FFI for your fancy new syscall; it needs to percolate down to the library they are actually going to use. That percolation takes time, and the extra latency makes this sort of reform project less enticing. The advantage is when the library interface already works with the new system call. For example, someone pointed out the Go already doesn't expose fork/exec because it wouldn't work on Windows. Well, the proposed process spawning interface might not only support Go's interface, but make it easier to implement. If we can swap out a standard library implementation once, and then a huge number of applications instantly benefit, that really boosts the cost/benefit ratio! I myself do not write applications in C either. I mainly notice these bad syscalls when the higher level library interfaces all of the sudden get really awkward for no good reason. > The second is that we should be careful about putting too much stuff in the OS that really belongs in libraries or applications. Yes absolutely agree. I meant to talk to about interfaces while being agnostic about what thing implements them. I would be perfectly happy if the OS didn't implement a file system at all. The reform approach probably does mean adding yet more system calls to existing heritage kernels. But that must not be the entire story. Ideally things look like this: 1. Implement more system calls in heritage kernels with better semantics. 2. Userland software can switch to new system calls and become simpler. Ideally it uses fewer different system calls than before. 3. New greenfield kernels can support the improved/simplified userland software with less work / fewer constraints. In this manner I hope we can "pivot" to better things. (The pivoting reminds me of parallel parking n-point turn a bit, in that a bunch of small steps with a lot of "wash" somehow combine into something more impressive.)

by bborud 3y ago

BirAdam 3y ago

Same. The only thing I’d say in contrast is that while getting worse to manage, a lot of stuff works more often than it used to do. Now, I’ll expand and add a fourth point. Do not have so much hubris as to think that you can achieve in one year what took 40 years to build, and certainly do not think that just because you have better tooling you will be able to reimplement something better than what has already been done. The reality is that the new implementation will likely suck in comparison to the old one for many years, and it will require ceaseless and thankless effort. Prime example: Wayland (which is now really awesome btw) that took a decade or so. If, otoh, something sucks and has done for a long time, go for it. Prime example: Linux audio (OSS, Alsa, Pulse, Jack) where pipewire is a godsend.

Ericson2314 3y ago

> Me today, still writing server applications, but now in Go, lives somewhat more remotely from the low level details of the system. There are more abstractions between me and the system. It is rare that I even have to deal directly with, for instance, low level networking system calls (except the odd ioctl, which, as someone pointed out years ago, is the carpet under which we sweep all the things that we couldn't figure out how to do properly). This is both a disadvantage and a major advantage. The disadvantage is that yes, changing syscalls / libc is not enough. No one is going to hand-write the FFI for your fancy new syscall; it needs to percolate down to the library they are actually going to use. That percolation takes time, and the extra latency makes this sort of reform project less enticing. The advantage is when the library interface already works with the new system call. For example, someone pointed out the Go already doesn't expose fork/exec because it wouldn't work on Windows. Well, the proposed process spawning interface might not only support Go's interface, but make it easier to implement. If we can swap out a standard library implementation once, and then a huge number of applications instantly benefit, that really boosts the cost/benefit ratio! I myself do not write applications in C either. I mainly notice these bad syscalls when the higher level library interfaces all of the sudden get really awkward for no good reason. > The second is that we should be careful about putting too much stuff in the OS that really belongs in libraries or applications. Yes absolutely agree. I meant to talk to about interfaces while being agnostic about what thing implements them. I would be perfectly happy if the OS didn't implement a file system at all. The reform approach probably does mean adding yet more system calls to existing heritage kernels. But that must not be the entire story. Ideally things look like this: 1. Implement more system calls in heritage kernels with better semantics. 2. Userland software can switch to new system calls and become simpler. Ideally it uses fewer different system calls than before. 3. New greenfield kernels can support the improved/simplified userland software with less work / fewer constraints. In this manner I hope we can "pivot" to better things. (The pivoting reminds me of parallel parking n-point turn a bit, in that a bunch of small steps with a lot of "wash" somehow combine into something more impressive.)