3 ms·

The pipeline pattern has its uses, but it's not the most important (or even the most well known?) tenant of the Unix philosophy. It's "Everything is a File"[0]

The pipeline pattern has its uses, but it's not the most important (or even the most well known?) tenant of the Unix philosophy. It's "Everything is a File"[0]. That one has really stood the test of time. And it's often misunderstood to mean that everything implements {Read,Write,Seek,Truncate, etc.}. Then when a TCP socket or character device or whatever shows up, the whole abstraction leaks, and it seems like not everything is a file. The Good Parts of Everything is a File: 1. Everything exists in a Filesystem, meaning a directory tree. There are directories, they list things, either resources, or more directories. A process gets access to all additional resources through the filesystem. We've really screwed up by putting networking in it's own place, environment variables, etc. 2. File descriptors. If you squint, these are capabilities[1], which is probably the best way to manage resource permissions. They are fine grained, they handle delegation, there is a tree of legitimacy back to the root. We were so close, but we got this one wrong too, with a single global root, instead of a chroot per process by default. [0] https://en.wikipedia.org/wiki/Everything_is_a_file https://en.wikipedia.org/wiki/Everything_is_a_file [1] https://en.wikipedia.org/wiki/Capability-based_security https://en.wikipedia.org/wiki/Capability-based_security

fyrn_ 3y ago
They talk pretty extensively about everything is a file? They even mention the "everything is a file" description of that concept
- colordrops 3y ago
  Yeah seems the GP didn't watch the video.
  - xena 3y ago
    At some point I need to make a really misleading title for my talks or articles just to catch people like the GP that don't read the article or watch the talk and only comment based on what the title makes them assume it's about.
  - alphazard 3y ago
    I did not watch the video. I assumed the blog post was a transcription since it contains parentheticals like "(audience laughs)". My takeaway from the transcript was that the author was chaining together webassembly programs like a pipeline in a Unix shell. Is the video about something else entirely?
    - deleted 3y ago
      [deleted]
    - deleted 3y ago
      [deleted]
    - guiambros 3y ago
      The post seems indeed a full transcript of the video, and it does mention the everything-is-a-file: "So, going back to where we were with Unix, what does it mean for everything to be a file? What is a file in the first place?" The talk is fantastic; Xe is a prolific hacker, low-level OS engineer, and engaging speaker. If anyone prefers YT, here's the direct link: https://www.youtube.com/watch?v=QNDvfez6QL0 https://www.youtube.com/watch?v=QNDvfez6QL0
zzo38computer 3y ago
The pipeline pattern is still very useful, even if it isn't the most important or most well known tenant (I am not sure of this, but maybe you are right). Still, of course it is not perfect. "Everything is a file" has its benefits too, but also is not perfect either. However, even if not all of {Read,Write,Seek,Truncate,etc} are implemented for all objects, some of them will be implemented, e.g. a TCP socket or character device can use read/write (or some might be read-only or write-only) but is not seekable. This is useful to use programs expecting other files and they will still work. For example, once I had a USB with a exfat file system, which I could not mount, but I knew it contained a ZIP file, so I tried that it it was able to extract the ZIP archive even though it could not be mounted; that is it could treat the USB device itself as a file. And, commonly, it is useful to write stuff that could be written to files, to pipes; for example in Heirloom-mailx you can write attachments to pipes instead of files, and I find this very helpful. I think that the file system directory tree is not the best way. File descriptors are like capabilities, and that is good (and I agree that it is probably the best way to manage resource permissions), but I think that it could be done better as better capabilities. I think that a single global root and chroot per process are both not the best way; I have a (what I think, at least) better way. My own design has a file system but does not have directory structures nor file names, but it is a hypertext file system, and files can have multiple forks, and the data streams can contain links (similar to UNIX hard links) to other files. Links can optionally be to a fixed version, or to a changeable version; copy on write can be used if you have both kinds of links to the same file. There is also journaling, and can have locks and transactions that can consist of multiple objects at once (this is necessary in the core system, so that you do not make a mess trying to do such things in user code like SQLite (and any other SQL database engine) does). My own design also uses "proxy capabilities". Messages can be passed using capabilities, and these messages can contain sequences of bytes and/or capabilities. A program might also create its own capabilities, which can be proxies of others; you can implement fine grained security and also allow fault simulation and many other purposes. All I/O and system calls (except Yield and Quit) must use these capabilities (for full security). A program will receive an initial message when it starts, so it will start up with some capabilities that were passed in that message (which can be whatever capabilities the caller decided to give it). Also, the multiple objects transaction/locking mentioned above is actually a general feature of capabilities and is not specific to disk files; you can make a transaction or lock of any set of objects (if supported; some combinations might not be possible). Proxy capabilities are actually very useful and many of the high-level features of the system are implemented in terms of proxy capabilities, so the kernel does not need to know all of the possible uses. In this way, you can easily emulate a "chroot per process", although it does not actually work like that. Such an initial message could as easily be used to emulate environment variables or whatever else you might want, too; a POSIX compatibility layer can be possible in user code if needed, although the low-level and high-level design of this system are not designed to be POSIX but rather something different, which does (what I am considering) working better in many ways. (My own operating system design does not currently have a name.)
deleted 3y ago
[deleted]
- deleted 3y ago
  [deleted]

fyrn_ 3y ago

They talk pretty extensively about everything is a file? They even mention the "everything is a file" description of that concept

colordrops 3y ago
Yeah seems the GP didn't watch the video.
- xena 3y ago
  At some point I need to make a really misleading title for my talks or articles just to catch people like the GP that don't read the article or watch the talk and only comment based on what the title makes them assume it's about.
- alphazard 3y ago
  I did not watch the video. I assumed the blog post was a transcription since it contains parentheticals like "(audience laughs)". My takeaway from the transcript was that the author was chaining together webassembly programs like a pipeline in a Unix shell. Is the video about something else entirely?
  - deleted 3y ago
    [deleted]
  - deleted 3y ago
    [deleted]
  - guiambros 3y ago
    The post seems indeed a full transcript of the video, and it does mention the everything-is-a-file: "So, going back to where we were with Unix, what does it mean for everything to be a file? What is a file in the first place?" The talk is fantastic; Xe is a prolific hacker, low-level OS engineer, and engaging speaker. If anyone prefers YT, here's the direct link: https://www.youtube.com/watch?v=QNDvfez6QL0 https://www.youtube.com/watch?v=QNDvfez6QL0

zzo38computer 3y ago

The pipeline pattern is still very useful, even if it isn't the most important or most well known tenant (I am not sure of this, but maybe you are right). Still, of course it is not perfect. "Everything is a file" has its benefits too, but also is not perfect either. However, even if not all of {Read,Write,Seek,Truncate,etc} are implemented for all objects, some of them will be implemented, e.g. a TCP socket or character device can use read/write (or some might be read-only or write-only) but is not seekable. This is useful to use programs expecting other files and they will still work. For example, once I had a USB with a exfat file system, which I could not mount, but I knew it contained a ZIP file, so I tried that it it was able to extract the ZIP archive even though it could not be mounted; that is it could treat the USB device itself as a file. And, commonly, it is useful to write stuff that could be written to files, to pipes; for example in Heirloom-mailx you can write attachments to pipes instead of files, and I find this very helpful. I think that the file system directory tree is not the best way. File descriptors are like capabilities, and that is good (and I agree that it is probably the best way to manage resource permissions), but I think that it could be done better as better capabilities. I think that a single global root and chroot per process are both not the best way; I have a (what I think, at least) better way. My own design has a file system but does not have directory structures nor file names, but it is a hypertext file system, and files can have multiple forks, and the data streams can contain links (similar to UNIX hard links) to other files. Links can optionally be to a fixed version, or to a changeable version; copy on write can be used if you have both kinds of links to the same file. There is also journaling, and can have locks and transactions that can consist of multiple objects at once (this is necessary in the core system, so that you do not make a mess trying to do such things in user code like SQLite (and any other SQL database engine) does). My own design also uses "proxy capabilities". Messages can be passed using capabilities, and these messages can contain sequences of bytes and/or capabilities. A program might also create its own capabilities, which can be proxies of others; you can implement fine grained security and also allow fault simulation and many other purposes. All I/O and system calls (except Yield and Quit) must use these capabilities (for full security). A program will receive an initial message when it starts, so it will start up with some capabilities that were passed in that message (which can be whatever capabilities the caller decided to give it). Also, the multiple objects transaction/locking mentioned above is actually a general feature of capabilities and is not specific to disk files; you can make a transaction or lock of any set of objects (if supported; some combinations might not be possible). Proxy capabilities are actually very useful and many of the high-level features of the system are implemented in terms of proxy capabilities, so the kernel does not need to know all of the possible uses. In this way, you can easily emulate a "chroot per process", although it does not actually work like that. Such an initial message could as easily be used to emulate environment variables or whatever else you might want, too; a POSIX compatibility layer can be possible in user code if needed, although the low-level and high-level design of this system are not designed to be POSIX but rather something different, which does (what I am considering) working better in many ways. (My own operating system design does not currently have a name.)

deleted 3y ago

[deleted]

deleted 3y ago
[deleted]