8 ms·

On the contrary, the widespread obsession with "Unix philosophy" often provides quasi-religious objections to suggestions we ever try new approaches to things.

On the contrary, the widespread obsession with "Unix philosophy" often provides quasi-religious objections to suggestions we ever try new approaches to things. In particular, the desire to cram increasingly diverse and rich interfaces into 70s file IO is a strange sacrament

seydar 3y ago
What are some alternatives that you'd recommend people look at? I'm keen for any potential successor, but a lot things come back to effectively text, and anything we could build on top of it. Binary object formats seem like an alternative for faster parsing of structured data, so while that ability has always been it, it's a matter of people actually sticking to it. Maybe some coordination between the program and the shell?
- LeonidasXIV 3y ago
  Some alternatives is a structured data interface, a bit like PowerShell where not everything needs to be serialized and deserialized and parsed in every program in a pipeline. Another approach is that the OS is basically a complete environment where everything is code. You can see the idea in the Smalltalk environments where you can theoretically interact with every object by sending messages. Lisp machines come to mind as well and one could even consider early personal computers that booted into BASIC as an idea of this (though in BASIC instead ov everything is a file everything is memory)
  - traverseda 3y ago
    Generally I think people are happy piping structured data around, the problem is that ties you into a particular data structure. JSON seems to be winning there, and I think people would be very happy if more commands had a JSON environment variable, or even if it was possible to output data on a /dev/stdjson pipe. Anyone know how to add a new stdout-like interface to unix-like OSes?
    - dale_glass 3y ago
      JSON is a good start, but Powershell is great in that a date is actually a date object, which means that I can do operations on that without faffing around with parsing and worrying about whether what comes in the JSON might depend on the locale of some system, or that somebody didn't take into account that I might want microseconds and truncated the timestamp.
      - traverseda 3y ago
        Right, but at that point you're tying it to a universal type system. Personally I like YAML, since you can add type-hints to data which you can use to turn simple text types into more complex types, but I can see why that standard wouldn't take off. Most of the unix-philosophy people I know are interested in stuff like that, it just has to be implemented in a thoughtful way.
        deleted 3y ago
        [deleted]
        wry_discontent 3y ago
        YAML is a crime.
        traverseda 3y ago
        I can see why you'd be worried about it, it's a lot more powerful than what I'd actually want for this. But that's sort of part of the problem, no one is going to agree on a common universal data-type.
      - convolvatron 3y ago
        if you're going to use something JSONesque as your common language, I think you do need to include a schema
    - stvltvs 3y ago
      What value would a new standard pipe for JSON bring? It's already serialized text that can be sent to stdout. The real rub is getting programs to speak JSON natively.
- delta_p_delta_x 3y ago
  > What are some alternatives that you'd recommend people look at? For me, this would mean an OS that supports both static and runtime reflection, and therefore code-generation and type generation. Strong typing should become a fundamental part of OS and system design. This probably means having an OS-wide thin runtime (something like .NET?) that all programs are written against. UNIX's composability was a good idea, but is an outdated implementation, which is built on string-typing everything, including command-line options, environment variables, program input and output, and even memory and data itself ('a file is a bag of bytes'). The same thing has happened to C (and therefore C++) itself. The way to use a library in C (and C++) is to literally textually include function and type declarations, and then compile disparate translation units together. If we want runtime library loading, we have to faff with dlopen/dlsym/LoadLibrary/GetProcAddress, and we have to know the names of symbols beforehand. A better implementation would use full-fledged reflection to import new types, methods, and free functions into the namespace of the currently-executing program. James Mickens in The Night Watch[1] says 'you can’t just place a LISP book on top of an x86 chip and hope that the hardware learns about lambda calculus by osmosis'. Fair point. But the next-best thing would have been to define useful types around pointers as close to the hardware as possible, rather than just saying 'here, this is the address for the memory you want from HeapAlloc; go do what you want with it and I will only complain at runtime when you break it'. Pointers are a badly leaky abstraction that don't really map to the real thing anyway, given MMUs, memory mapping, etc. There are so many ways we could make things a bit more disciplined in system and OS design, and drastically improve life for everyone using it. Of the major vendors, I'd say only Windows has taken merely half-hearted steps towards an OS-wide 'type system'. I say half-hearted, because PowerShell is fantastic and handles are significantly more user-friendly than UNIX's file descriptors, thread pointers, and pipes. But this is still a small oasis in a desert of native programs that have to mess with string-typing. [1]: https://www.usenix.org/system/files/1311_05-08_mickens.pdf https://www.usenix.org/system/files/1311_05-08_mickens.pdf
- nindalf 3y ago
  One alternative is Apache Arrow. It's an efficient in-memory format used to represent flat or hierarchal data. Multiple databases and programs like pandas support this, so they become compatible with each other.
traverseda 3y ago
You're not inventing new things, take a look at CORBA and compare that to COM objects. It's not "no new things", it's "build things that are actually principled, don't just cram a bunch of garbage together because it's the path of least resistance". >Every increase in expressiveness brings an increased burden on all who care to understand the message. There are plenty of new things that are largely compatible with unix philosophy. MQTT is one of my favorite examples, it's a message bus that follows a lot of unix philosophy and I find it a joy to work with compared to stuff like DBUS. Obviously it doesn't fill the same role as DBUS, but still.
stefan_ 3y ago
Recommended reading: the Unix Haters Handbook: https://web.mit.edu/~simsong/www/ugh.pdf https://web.mit.edu/~simsong/www/ugh.pdf
- lproven 3y ago
  Big fan, and I mentioned it in the article I pointed out downthread: https://www.theregister.com/2023/12/25/the_war_of_the_workstations/ https://www.theregister.com/2023/12/25/the_war_of_the_workst... Note that this is loosely based on part of a FOSDEM talk I gave 4 years ago.
HackerThemAll 3y ago
That approach works so well that a program written in '70s is so well abstracted away from the intricacies of the system that it can input data from a network socket, a pipe, a file, or output to whatnot, and doesn't need to be touched for all that to work. A plain program reading from stdin and writing to stdout can be turned into a network service via inetd or similar by just redirecting file descriptors. But for the contemporary software coders the ingenious idea is of course rubbish. They would replace everything by node_modules.
- 01HNNWZ0MV43FF 3y ago
  They say io_uring is faster. The cost of modularity is often overhead.
- bluetomcat 3y ago
  It was originally developed as a multi-user time-sharing system. Multiple users would access the system simultaneously via the attached teletype terminals. The "shell" program handled such an interactive user session. All IO was blocking, and programs were written in a sequential manner. Piping enabled them to feed the standard output of one process to the standard input of another. It's a good framework for text-based computing, but nowadays we have so much more problems to cope with. This original idea never addressed networking, binary versioning and control, configuration management, async IO, multithreading, GUI, etc.
- jeffbee 3y ago
  It's great that it is easy to do simple things with ancient tools, and it's completely true that you can inetd and CGI your way to covering numerous use cases on small systems. The problem is the lack of innovation holds us back from solving complex use cases. The sockets API in particular is a gigantic mess, because the protocols are more complicated than people imagined forty years ago. For example it is simple to send a datagram. What if you want to get notified of the timestamp when the datagram hits the wire? Now you will 1) read a lot of man pages, 2) write 100s of lines of gross C code, 3) read the source code of the Linux kernel to figure out which parts of the manual were lies, then 4) fix your program with some different code and a long, angry comment. None of they would be necessary if the last quarter-century of network features hadn't been jammed into the ioctl and setsockopt hack interfaces. In other words there's plenty of room for operating systems to reconsider interfaces in light of the systems and protocols we have now, instead of Space Age imaginary use cases.
gregjor 3y ago
You can't blame Unix for getting popular, or for almost no research and progress in operating systems since Unix got popular. Rob Pike wrote a good paper about that. http://doc.cat-v.org/bell_labs/utah2000/ http://doc.cat-v.org/bell_labs/utah2000/ At this point it's network effect, familiarity, and the free price tag, not a quasi-religious sacrament. Anyone can try to take that different approach, but that doesn't mean it will go anywhere. Some very smart people have worked on alternatives to Unix, or different directions for operating systems -- including the original Unix team with Plan 9, Wirth with Oberon, the Lisp machines. No Unix Inquisition shut those down, they just didn't offer a 10x improvement along enough axes.
kazinator 3y ago
The stream of bytes is democratizing; you can put anything into it you want. If you impose any structure, you're telling hackers what to do at the system level. If that clashes with their favorite language run-time or whatever, you're effectively declaring war.
jjav 3y ago
> In particular, the desire to cram increasingly diverse and rich interfaces into 70s file IO is a strange sacrament Long time ago I asked my karate sensei how come I see some of the black belts doing katas in a different way. His answer was that to improve on something one must first have deep mastery of how it is done. (i.e. "Shut up and do the katas correctly. If you ever get to black belt then you can improvise.") Always found that answer to apply to everything. In particular it applies to a lot of the poor quality libraries and code we see today as a direct consequence of not understanding the past and what has been tried and failed and why. The fact that we're talking in 2024 about using the file abstractions of Unix created in the 70s is proof of how extremely powerful they are. Nothing is perfect, but that's quite an achievement.
- galaxyLogic 3y ago
  A mistake we often make I believe is to focus on code-quality, rather than code-content-quality. Code is basically mathematics. Is it good math or bad that is the question. Does it do something useful, in an economical way. To write great code, I believe, means you have to be a domain expert of the domain for which your code solves problems.

seydar 3y ago

What are some alternatives that you'd recommend people look at? I'm keen for any potential successor, but a lot things come back to effectively text, and anything we could build on top of it. Binary object formats seem like an alternative for faster parsing of structured data, so while that ability has always been it, it's a matter of people actually sticking to it. Maybe some coordination between the program and the shell?

LeonidasXIV 3y ago
Some alternatives is a structured data interface, a bit like PowerShell where not everything needs to be serialized and deserialized and parsed in every program in a pipeline. Another approach is that the OS is basically a complete environment where everything is code. You can see the idea in the Smalltalk environments where you can theoretically interact with every object by sending messages. Lisp machines come to mind as well and one could even consider early personal computers that booted into BASIC as an idea of this (though in BASIC instead ov everything is a file everything is memory)
- traverseda 3y ago
  Generally I think people are happy piping structured data around, the problem is that ties you into a particular data structure. JSON seems to be winning there, and I think people would be very happy if more commands had a JSON environment variable, or even if it was possible to output data on a /dev/stdjson pipe. Anyone know how to add a new stdout-like interface to unix-like OSes?
  - dale_glass 3y ago
    JSON is a good start, but Powershell is great in that a date is actually a date object, which means that I can do operations on that without faffing around with parsing and worrying about whether what comes in the JSON might depend on the locale of some system, or that somebody didn't take into account that I might want microseconds and truncated the timestamp.
    - traverseda 3y ago
      Right, but at that point you're tying it to a universal type system. Personally I like YAML, since you can add type-hints to data which you can use to turn simple text types into more complex types, but I can see why that standard wouldn't take off. Most of the unix-philosophy people I know are interested in stuff like that, it just has to be implemented in a thoughtful way.
      - deleted 3y ago
        [deleted]
      - wry_discontent 3y ago
        YAML is a crime.
        traverseda 3y ago
        I can see why you'd be worried about it, it's a lot more powerful than what I'd actually want for this. But that's sort of part of the problem, no one is going to agree on a common universal data-type.
    - convolvatron 3y ago
      if you're going to use something JSONesque as your common language, I think you do need to include a schema
  - stvltvs 3y ago
    What value would a new standard pipe for JSON bring? It's already serialized text that can be sent to stdout. The real rub is getting programs to speak JSON natively.
delta_p_delta_x 3y ago
> What are some alternatives that you'd recommend people look at? For me, this would mean an OS that supports both static and runtime reflection, and therefore code-generation and type generation. Strong typing should become a fundamental part of OS and system design. This probably means having an OS-wide thin runtime (something like .NET?) that all programs are written against. UNIX's composability was a good idea, but is an outdated implementation, which is built on string-typing everything, including command-line options, environment variables, program input and output, and even memory and data itself ('a file is a bag of bytes'). The same thing has happened to C (and therefore C++) itself. The way to use a library in C (and C++) is to literally textually include function and type declarations, and then compile disparate translation units together. If we want runtime library loading, we have to faff with dlopen/dlsym/LoadLibrary/GetProcAddress, and we have to know the names of symbols beforehand. A better implementation would use full-fledged reflection to import new types, methods, and free functions into the namespace of the currently-executing program. James Mickens in The Night Watch[1] says 'you can’t just place a LISP book on top of an x86 chip and hope that the hardware learns about lambda calculus by osmosis'. Fair point. But the next-best thing would have been to define useful types around pointers as close to the hardware as possible, rather than just saying 'here, this is the address for the memory you want from HeapAlloc; go do what you want with it and I will only complain at runtime when you break it'. Pointers are a badly leaky abstraction that don't really map to the real thing anyway, given MMUs, memory mapping, etc. There are so many ways we could make things a bit more disciplined in system and OS design, and drastically improve life for everyone using it. Of the major vendors, I'd say only Windows has taken merely half-hearted steps towards an OS-wide 'type system'. I say half-hearted, because PowerShell is fantastic and handles are significantly more user-friendly than UNIX's file descriptors, thread pointers, and pipes. But this is still a small oasis in a desert of native programs that have to mess with string-typing. [1]: https://www.usenix.org/system/files/1311_05-08_mickens.pdf https://www.usenix.org/system/files/1311_05-08_mickens.pdf
nindalf 3y ago
One alternative is Apache Arrow. It's an efficient in-memory format used to represent flat or hierarchal data. Multiple databases and programs like pandas support this, so they become compatible with each other.

traverseda 3y ago

You're not inventing new things, take a look at CORBA and compare that to COM objects. It's not "no new things", it's "build things that are actually principled, don't just cram a bunch of garbage together because it's the path of least resistance". >Every increase in expressiveness brings an increased burden on all who care to understand the message. There are plenty of new things that are largely compatible with unix philosophy. MQTT is one of my favorite examples, it's a message bus that follows a lot of unix philosophy and I find it a joy to work with compared to stuff like DBUS. Obviously it doesn't fill the same role as DBUS, but still.

stefan_ 3y ago

Recommended reading: the Unix Haters Handbook: https://web.mit.edu/~simsong/www/ugh.pdf https://web.mit.edu/~simsong/www/ugh.pdf

lproven 3y ago
Big fan, and I mentioned it in the article I pointed out downthread: https://www.theregister.com/2023/12/25/the_war_of_the_workstations/ https://www.theregister.com/2023/12/25/the_war_of_the_workst... Note that this is loosely based on part of a FOSDEM talk I gave 4 years ago.

HackerThemAll 3y ago

That approach works so well that a program written in '70s is so well abstracted away from the intricacies of the system that it can input data from a network socket, a pipe, a file, or output to whatnot, and doesn't need to be touched for all that to work. A plain program reading from stdin and writing to stdout can be turned into a network service via inetd or similar by just redirecting file descriptors. But for the contemporary software coders the ingenious idea is of course rubbish. They would replace everything by node_modules.

01HNNWZ0MV43FF 3y ago
They say io_uring is faster. The cost of modularity is often overhead.
bluetomcat 3y ago
It was originally developed as a multi-user time-sharing system. Multiple users would access the system simultaneously via the attached teletype terminals. The "shell" program handled such an interactive user session. All IO was blocking, and programs were written in a sequential manner. Piping enabled them to feed the standard output of one process to the standard input of another. It's a good framework for text-based computing, but nowadays we have so much more problems to cope with. This original idea never addressed networking, binary versioning and control, configuration management, async IO, multithreading, GUI, etc.
jeffbee 3y ago
It's great that it is easy to do simple things with ancient tools, and it's completely true that you can inetd and CGI your way to covering numerous use cases on small systems. The problem is the lack of innovation holds us back from solving complex use cases. The sockets API in particular is a gigantic mess, because the protocols are more complicated than people imagined forty years ago. For example it is simple to send a datagram. What if you want to get notified of the timestamp when the datagram hits the wire? Now you will 1) read a lot of man pages, 2) write 100s of lines of gross C code, 3) read the source code of the Linux kernel to figure out which parts of the manual were lies, then 4) fix your program with some different code and a long, angry comment. None of they would be necessary if the last quarter-century of network features hadn't been jammed into the ioctl and setsockopt hack interfaces. In other words there's plenty of room for operating systems to reconsider interfaces in light of the systems and protocols we have now, instead of Space Age imaginary use cases.

gregjor 3y ago

You can't blame Unix for getting popular, or for almost no research and progress in operating systems since Unix got popular. Rob Pike wrote a good paper about that. http://doc.cat-v.org/bell_labs/utah2000/ http://doc.cat-v.org/bell_labs/utah2000/ At this point it's network effect, familiarity, and the free price tag, not a quasi-religious sacrament. Anyone can try to take that different approach, but that doesn't mean it will go anywhere. Some very smart people have worked on alternatives to Unix, or different directions for operating systems -- including the original Unix team with Plan 9, Wirth with Oberon, the Lisp machines. No Unix Inquisition shut those down, they just didn't offer a 10x improvement along enough axes.

kazinator 3y ago

The stream of bytes is democratizing; you can put anything into it you want. If you impose any structure, you're telling hackers what to do at the system level. If that clashes with their favorite language run-time or whatever, you're effectively declaring war.

jjav 3y ago

> In particular, the desire to cram increasingly diverse and rich interfaces into 70s file IO is a strange sacrament Long time ago I asked my karate sensei how come I see some of the black belts doing katas in a different way. His answer was that to improve on something one must first have deep mastery of how it is done. (i.e. "Shut up and do the katas correctly. If you ever get to black belt then you can improvise.") Always found that answer to apply to everything. In particular it applies to a lot of the poor quality libraries and code we see today as a direct consequence of not understanding the past and what has been tried and failed and why. The fact that we're talking in 2024 about using the file abstractions of Unix created in the 70s is proof of how extremely powerful they are. Nothing is perfect, but that's quite an achievement.

galaxyLogic 3y ago
A mistake we often make I believe is to focus on code-quality, rather than code-content-quality. Code is basically mathematics. Is it good math or bad that is the question. Does it do something useful, in an economical way. To write great code, I believe, means you have to be a domain expert of the domain for which your code solves problems.