4 ms·

Am I the one who doesn’t want this? The entire point of UNIX philosophy (Which seems to be something they aren’t teaching in software development these days) i

Am I the one who doesn’t want this? The entire point of UNIX philosophy (Which seems to be something they aren’t teaching in software development these days) is to do one thing and do it well. We don’t need Linux operating operating as a big declarative distributed system with a distributed scheduling systems and a million half-baked APIs to interact with it, the way K8s works. If you want that you should build something to your specific requirements, not shove more things into the kernel.

goodpoint 5y ago
Linux/UNIX does not have to turn into a mess like k8s to be natively distributed. Plan9 was doing it with a tiny codebase in comparison.
Karrot_Kream 5y ago
The UNIX philosophy made more sense as an abstraction for a computer when computers were simpler. Computers nowadays (well at least since 2006-ish) have multiple cores executing simultaneously with complicated amounts of background logic, interrupt-driven logic, shared caches, etc. The UNIX philosophy doesn't map to this reality at all. Right now there's no set of abstractions except machine code that exposes the machine's distributed systems' in a coherent abstraction. Nothing is stopping someone else from writing a UNIX abstraction atop this though.
- generalizations 5y ago
  The idea of doing one thing, and doing it well, isn't dependent on the simplicity of the underlying system (I imagine that PDP-11 systems seemed impressively complicated in their time, too). The UNIX philosophy is a paradigm for managing complexity. To me, that seems more relevant with modern computers, not less. > “A program is generally exponentially complicated by the number of notions that it invents for itself. To reduce this complication to a minimum, you have to make the number of notions zero or one, which are two numbers that can be raised to any power without disturbing this concept. Since you cannot achieve much with zero notions, it is my belief that you should base systems on a single notion.” - Ken Thompson
random314 5y ago
The Unix philosophy was a reasonably good model decades ago. But I think it is over romanticized. It's binary blob design is no good for security, as opposed to a byte code design like Forth. Its user security model was poor and doesn't help with modern devices like phones. Its multiprocess model was ham fisted into a multithreading model to compete with windows NT. Its asynchronous i/o model has always been a train wreck even compared to NT. Its design creates performance issues, especially in multiproc networking code with needless amount of memcopys. Now folks are rewriting the networking stack in user space. Its software abstraction layer was some simple scheme from the 70s which has fragmented into crazy number of implementations now. Open source developers still complain about how much easier it is to build a package for windows, as opposed to linux. It was never meant to be a distributed system either. Modern enterprise compute cannot scale by treating and managing each individual VM as it's own thing with clusters held together by some sysadmins batch scripts.
- anthk 5y ago
  And yet Linux manages better doing heavy I/O stuff over filesystems than Windows NT.
  - pjmlp 5y ago
    Because it doesn't provide the abstraction capabilites that NTFS allows for third parties, so naturally it is faster doing less.
    - anthk 5y ago
      Ever checked xattr's and ACL's?
      - random314 5y ago
        He is talking about the NTFS file filter stack.
        anthk 5y ago
aseipp 5y ago
All of this was possible with QNX literally decades ago, and it didn't need whatever strawman argument you're making up in your head in order to accomplish it. QNX was small, fast, lean, real-time, distributed, and very powerful for the time. Don't worry, it even had POSIX support. A modern QNX would be very well received, I think, precisely because taking a distributed-first approach would dramatically simplify the whole system design versus tacking on a distributed layer on top of one designed for single computers. > Which seems to be something they aren’t teaching in software development these days This is funny. Perhaps the thing you should have been taught instead is history, my friend.
- jeffreygoesto 5y ago
  You mean QNet [0]? That is still alive... It is for LAN use ("Qnet is intended for a network of trusted machines that are all running QNX Neutrino and that all use the same endianness."), so extra care is needed to secure this group of machines when exposed to the internet. [0] https://www.qnx.com/developers/docs/7.0.0///index.html#com.qnx.doc.neutrino.user_guide/topic/qnet.html https://www.qnx.com/developers/docs/7.0.0///index.html#com.q... [1] https://recon.cx/2018/brussels/resources/slides/RECON-BRX-2018-Dissecting-QNX.pdf https://recon.cx/2018/brussels/resources/slides/RECON-BRX-20...
  - aseipp 5y ago
    Correct. Thought QNet itself is only one possible implementation, in a sense (but obviously the one shipped with QNX.) And the more important part of the whole thing is the message-passing API design built into the system, which enables said networking transparency, because it means your programs are abstracted over the underlying transport mechanism. "LAN use" I think would qualify roughly 95% of the need for a "distributed OS," including a lot of usage of K8s, frankly. Systems with WAN latency impose a different set of challenges for efficient comms at the OS layer. But even then you also have to design your apps themselves to handle WAN-scale latencies, failover, etc too. So it isn't like QNX is going to make your single-executable app magic or whatever bullshit. But it exposes a set of primitives that are much more tightly woven into the core system design and much more flexible for IPC. Which is what a distributed system is; a large chattery IPC system. The RECON PDF is a very good illustration of where such a design needs to go, though. It doesn't surprise me QNX is simply behind modern OS's exploit mitigations. But on top of that, a modern take on this would have to blend in a better security model. You'd really just need to throw out the whole UNIX permission model frankly, it's simply terrible as far as modern security design is concerned. QNet would obviously have to change as well. You'd at minimum want something like a capability-based RPC layer I'd think. Every "application server" is like an addressable object you can refer to, invoke methods on, etc. (Cap'n Proto is a good way to get a "feel" for this kind of object-based server design without abandoning Linux, if you use its RPC layer.) I desperately wish someone would reinvent QNX but with all the nice trappings and avoiding the missteps we've accumulated over the past 10 to 15 years. Alas, it's much more profitable to simply re-invent its features poorly every couple of years and sell that instead. This overview of the QNX architecture (from 1992!) is one of my favorite papers for its simplicity and straightforward prose. Worth a read for anyone who like OS design. https://cseweb.ucsd.edu/~voelker/cse221/papers/qnx-paper92.pdf https://cseweb.ucsd.edu/~voelker/cse221/papers/qnx-paper92.p...
pjmlp 5y ago
The philosophy that is cargo culted and was never taken seriously by any commercial UNIX.
- random314 5y ago
  Moreover when folks talk about doing only one thing and one thing well, they are referring to command line utilities and pipes. And pipes were an invention of Doug mclroy, not Ritchie or Thompson. And the unix command-pipe philosophy is realized much better as ordinary functions in a functional programming language.
  - pjmlp 5y ago
    Indeed, the UNIX CLI is a poor man's experience compared with a Lisp REPL, Smalltalk transcript or Oberon commands session.

goodpoint 5y ago

Linux/UNIX does not have to turn into a mess like k8s to be natively distributed. Plan9 was doing it with a tiny codebase in comparison.

Karrot_Kream 5y ago

The UNIX philosophy made more sense as an abstraction for a computer when computers were simpler. Computers nowadays (well at least since 2006-ish) have multiple cores executing simultaneously with complicated amounts of background logic, interrupt-driven logic, shared caches, etc. The UNIX philosophy doesn't map to this reality at all. Right now there's no set of abstractions except machine code that exposes the machine's distributed systems' in a coherent abstraction. Nothing is stopping someone else from writing a UNIX abstraction atop this though.

generalizations 5y ago
The idea of doing one thing, and doing it well, isn't dependent on the simplicity of the underlying system (I imagine that PDP-11 systems seemed impressively complicated in their time, too). The UNIX philosophy is a paradigm for managing complexity. To me, that seems more relevant with modern computers, not less. > “A program is generally exponentially complicated by the number of notions that it invents for itself. To reduce this complication to a minimum, you have to make the number of notions zero or one, which are two numbers that can be raised to any power without disturbing this concept. Since you cannot achieve much with zero notions, it is my belief that you should base systems on a single notion.” - Ken Thompson

random314 5y ago

The Unix philosophy was a reasonably good model decades ago. But I think it is over romanticized. It's binary blob design is no good for security, as opposed to a byte code design like Forth. Its user security model was poor and doesn't help with modern devices like phones. Its multiprocess model was ham fisted into a multithreading model to compete with windows NT. Its asynchronous i/o model has always been a train wreck even compared to NT. Its design creates performance issues, especially in multiproc networking code with needless amount of memcopys. Now folks are rewriting the networking stack in user space. Its software abstraction layer was some simple scheme from the 70s which has fragmented into crazy number of implementations now. Open source developers still complain about how much easier it is to build a package for windows, as opposed to linux. It was never meant to be a distributed system either. Modern enterprise compute cannot scale by treating and managing each individual VM as it's own thing with clusters held together by some sysadmins batch scripts.

anthk 5y ago
And yet Linux manages better doing heavy I/O stuff over filesystems than Windows NT.
- pjmlp 5y ago
  Because it doesn't provide the abstraction capabilites that NTFS allows for third parties, so naturally it is faster doing less.
  - anthk 5y ago
    Ever checked xattr's and ACL's?
    - random314 5y ago
      He is talking about the NTFS file filter stack.
      - anthk 5y ago

aseipp 5y ago

All of this was possible with QNX literally decades ago, and it didn't need whatever strawman argument you're making up in your head in order to accomplish it. QNX was small, fast, lean, real-time, distributed, and very powerful for the time. Don't worry, it even had POSIX support. A modern QNX would be very well received, I think, precisely because taking a distributed-first approach would dramatically simplify the whole system design versus tacking on a distributed layer on top of one designed for single computers. > Which seems to be something they aren’t teaching in software development these days This is funny. Perhaps the thing you should have been taught instead is history, my friend.

jeffreygoesto 5y ago
You mean QNet [0]? That is still alive... It is for LAN use ("Qnet is intended for a network of trusted machines that are all running QNX Neutrino and that all use the same endianness."), so extra care is needed to secure this group of machines when exposed to the internet. [0] https://www.qnx.com/developers/docs/7.0.0///index.html#com.qnx.doc.neutrino.user_guide/topic/qnet.html https://www.qnx.com/developers/docs/7.0.0///index.html#com.q... [1] https://recon.cx/2018/brussels/resources/slides/RECON-BRX-2018-Dissecting-QNX.pdf https://recon.cx/2018/brussels/resources/slides/RECON-BRX-20...
- aseipp 5y ago
  Correct. Thought QNet itself is only one possible implementation, in a sense (but obviously the one shipped with QNX.) And the more important part of the whole thing is the message-passing API design built into the system, which enables said networking transparency, because it means your programs are abstracted over the underlying transport mechanism. "LAN use" I think would qualify roughly 95% of the need for a "distributed OS," including a lot of usage of K8s, frankly. Systems with WAN latency impose a different set of challenges for efficient comms at the OS layer. But even then you also have to design your apps themselves to handle WAN-scale latencies, failover, etc too. So it isn't like QNX is going to make your single-executable app magic or whatever bullshit. But it exposes a set of primitives that are much more tightly woven into the core system design and much more flexible for IPC. Which is what a distributed system is; a large chattery IPC system. The RECON PDF is a very good illustration of where such a design needs to go, though. It doesn't surprise me QNX is simply behind modern OS's exploit mitigations. But on top of that, a modern take on this would have to blend in a better security model. You'd really just need to throw out the whole UNIX permission model frankly, it's simply terrible as far as modern security design is concerned. QNet would obviously have to change as well. You'd at minimum want something like a capability-based RPC layer I'd think. Every "application server" is like an addressable object you can refer to, invoke methods on, etc. (Cap'n Proto is a good way to get a "feel" for this kind of object-based server design without abandoning Linux, if you use its RPC layer.) I desperately wish someone would reinvent QNX but with all the nice trappings and avoiding the missteps we've accumulated over the past 10 to 15 years. Alas, it's much more profitable to simply re-invent its features poorly every couple of years and sell that instead. This overview of the QNX architecture (from 1992!) is one of my favorite papers for its simplicity and straightforward prose. Worth a read for anyone who like OS design. https://cseweb.ucsd.edu/~voelker/cse221/papers/qnx-paper92.pdf https://cseweb.ucsd.edu/~voelker/cse221/papers/qnx-paper92.p...

pjmlp 5y ago

The philosophy that is cargo culted and was never taken seriously by any commercial UNIX.

random314 5y ago
Moreover when folks talk about doing only one thing and one thing well, they are referring to command line utilities and pipes. And pipes were an invention of Doug mclroy, not Ritchie or Thompson. And the unix command-pipe philosophy is realized much better as ordinary functions in a functional programming language.
- pjmlp 5y ago
  Indeed, the UNIX CLI is a poor man's experience compared with a Lisp REPL, Smalltalk transcript or Oberon commands session.