4 ms·
Everything Is a File
- edward 3y agohttps://concepts.4angle.com/edward/everything_is_a_file https://concepts.4angle.com/edward/everything_is_a_file
- deathanatos 3y agoWhere I think many people get weaker abstractions of this in their mind is where the OP hints: > and therefore a more accurate description of this feature is Everything is a file descriptor The filesystem is where files go to get names, and to be organized. But critically, not everything is a normal file, which I think is where people experience the most "the abstraction is leaky!" feelings. I don't think the abstraction needs everything to be a normal file, but I think there is still a lot of use in having a unified namespace for all the various types of files, normal or otherwise. (As opposed to having disparate namespaces for files of different types.) For the "file descriptor" half, the value is being able to wait on events/event completion for a set of heterogeneous file descriptors, but boy did it take a while to get there with usable interfaces for that.
- charcircuit 3y agoopen, read, write, seek, ioctl, etc are a weird interface to use for IPC. I would rather there be a proper API for remote objects rather then every remote object having to roll its own protocol for doing the same.
- kazinator 3y agoIt's fine for local IPC like Unix domain sockets. For remote IPC, obviously the machines don't share a common space they can just access as a file for rendezvous, and so we don't use that API. We have socket, connect|{bind, listen, accept}, read, write. For connectionless, send, recv. Sockets are not part of "everything is a file" but of "everything is a file descriptor".
- charcircuit 3y ago>It's fine for local IPC like Unix domain sockets. I disagree. I should be able to just call a function. I don't want to bother figuring out the network protocol that the process decided on. It's too low level to expose.
- duped 3y agoThat's why 9P was invented.
- mjb 3y agoYes! There are a lot of benefits to the 'everything is a file' philosophy, but files aren't very good APIs for a structured data. The perfect example is the stuff in /proc and /sys. Making it all 'just a file' is super nice and flexible at first, but either makes stuff very difficult to change (the file becomes a poorly-specified inconvenient ad-hoc API), or makes it very hard to build applications that work in the long term (because the application has to write a change-tolerant parser). It makes a very cool demo (look, I can do stuff to TCP sockets using bash!), and really convenient for ad-hoc stuff, but is inferior to a real API for applications that need to keep working year after year after year while still being able to patch the rest of the system.
- duped 3y agoThe file tree is just a way of discovering endpoints to the API. You still need an encoding to read/write to those endpoints. The same is true over a "real" API with one big difference: there's not a small set of syscalls to access the endpoints and read/write binary data to them. Thats more stable than any FFI or encoding scheme. I'll take "everything is a file" over "everything is an object that has a special mechanism for querying interfaces and dispatching to handlers" or "everything is RPC" or "everything is REST" because it has the smallest surface area.
- charcircuit 3y ago>The same is true over a "real" API with one big difference: there's not a small set of syscalls to access the endpoints and read/write binary data to them. What do you mean? Binder from Android uses just the ioctl syscall. >because it has the smallest surface area. I assure you the 1 syscalls for using binder (unless you count open and close of /dev/binder) is less syscalls than what it takes when everything is a file (read, write, ioctl, lseek, lstat, etc).
- DonHopkins 3y agoIt's the job of the operating system to provide a stable foreign function interface, and dynamically linked libraries to access common operating system apis that aren't covered by system calls. Smallest surface area??! There's no excuse for Unix forcing you to use the fractally complex sucking chest wound of ioctl and brittle binary raw pointerless C structures in memory, instead of simply calling functions and passing normal parameters and pointers and results in and out. Computers are great at and have a long history of calling functions, so why not just support that, huh? "Everything is a file" is pointlessly performative but hopelessly incomplete Bauhaus Minimalism that fails to reduce but instead multiplies and relocates all the complexity to the Brutalist Rawness of ioctl's uncanny leaky abstraction of a kitchen sink, which undermines the entire Unix presumptuous philosophy of simplicity and consistency.
- deathanatos 3y agoI think the big problem with objects is that they're just not the common denominator: e.g., image files/formats are going to be byte streams. That, and getting the world to agree on object formats is like pulling teeth, it seems. E.g., WASM just wanted to add a string type, which OUGHT TO¹ be simple (use UTF-8) … but alas. JSON? Not enough types. YAML? Doesn't matter if it's good, nobody likes YAML. Etc. And you have to serialize them to bytes somewhere along the line; either to transfer their memory representation to another process, or to put them on the network. I start thinking "Perhaps something could be built on top of byte streams" but that's YAML, JSON, CBOR, take your pick. ¹(this is an RFC 6919 OUGHT TO)
- charcircuit 3y agoI don't understand. Objects existing doesn't mean files can no longer exist. >getting the world to agree on object formats is like pulling teeth In Windows apps agree, Mac app agree, iOS app agree, Android apps agree, Java apps agree. Sending or receiving strings to a system API is common. >And you have to serialize them to bytes somewhere along the line; either to transfer their memory representation to another process, or to put them on the network. Yes, but I'd much prefer if there was the 1 true way an os chooses it to be than each thing inventing its own way.
- GoblinSlayer 3y agoWell, there's dbus. On the other hand if you want a function and not protocol, why would you care what protocol it uses? It might as well be a file. When you call getaddrinfo("localhost"), it reads /etc/hosts - a text file.
- blondin 3y agopowershell wanted to expose input and output as objects. liked the concept. the execution could have been better though.
- DonHopkins 3y agoEven PowerShell's flawed and limited execution definitively proves how terribly misguided and morally bankrupt the pin-headed one-dimensional "everything is a file" Unix philosophy is, especially with its the ridiculously hard-to-parse ad-hoc non-standard mixed-bag syntax of /proc and /etc files, which would have been much better simply using xml or json, yet which don't even come close to achieving that low bar of machine readability and ease of scripting. If everything is a file, then why the fuck is the insufferable abomination of ioctl so crucial to haphazardly papier-mâché and duct tape over all the uncanny gaps and gaping cracks? Yet another strike against the mindless cargo cult nonsense and unquestioning sycophantic hero worship of TAOUP. https://web.mit.edu/~simsong/www/ugh.pdf https://web.mit.edu/~simsong/www/ugh.pdf 13 The File System Sure It Corrupts Your Files, But Look How Fast It Is! Pretty daring of you to be storing important files on a Unix system. —Robert E. Seastrom The traditional Unix file system is a grotesque hack that, over the years, has been enshrined as a “standard” by virtue of its widespread use. Indeed, after years of indoctrination and brainwashing, people now accept Unix’s flaws as desired features. It’s like a cancer victim’s immune system enshrining the carcinoma cell as ideal because the body is so good at making them. Way back in the chapter “Welcome, New User” we started a list of what’s wrong with the Unix file systems. For users, we wrote, the the most obvious failing is that the file systems don’t have version numbers and Unix doesn’t have an “undelete” capability—two faults that combine like sodium and water in the hands of most users. But the real faults of Unix file systems run far deeper than these two missing features. The faults are not faults of execution, but of ideology. With Unix, we often are told that “everything is a file.” Thus, it’s not surprising that many of Unix’s fundamental faults lie with the file system as well. https://news.ycombinator.com/item?id=26604833 https://news.ycombinator.com/item?id=26604833 Ask HN: What are the bad parts of Unix? https://news.ycombinator.com/item?id=26613607 https://news.ycombinator.com/item?id=26613607 lapsed_lisper on March 28, 2021 | prev | next [–] [...] But I also think there's a deeper set of problems in the "genetics" of Unix, in that it supports a "reductive" form of problem solving, but doesn't help at all if you want to build abstractions. Let's say one of the core ideas in Unix is "everything is a file", i.e., read/write/seek/etc. is the universal interface across devices, files, pipes, etc.). "Everything is a file" insulates a program from some (but not all!) irrelevant details of the mechanics of moving bytes into and out of RAM... by forcing all programs to contend with even more profoundly irrelevant details about how those bytes in RAM should be interpreted as data in the program! While it is sometimes useful to be able to peek or poke at bits in stray spots, most programs implicitly or explicitly traffic in in data relevant to that program. While every such datum must be /realized/ as bytes somewhere, operating on some datum's realization /as bytes/ (or by convention, as text) is mostly a place to make mistakes. Here's an example: consider the question "who uses bash as their login shell?" A classical "Unixy" methodology to attacking such a problem is supposed to be to (a) figure out how to get a byte stream containing the information you want, and then (b) figure out how to apply some tools to extract and perhaps transform that stream into the desired stream. So maybe you know that /etc/passwd one way to get that stream on your system, and you decide to use awk for this problem, and type awk -F: '$6 ~ "bash$" { print $1 }' /etc/passwd That's a nicely compact expression! Sadly, it's an incorrect one to apply to /etc/passwd to get the desired answer (at least on my hosts), because the login shell in the 7th field, not the 6th. Now, this is just a trivial little error, but that's why I like it as an example. Even in the most trivial cases, reducing anything to a byte stream does mean you can use any general purpose tool to a problem, but it also means that any such usage is going to reinvent the wheel in exact proportion to how directly it's using that byte stream; and that reinvention is a source of needless error. Of course the sensible thing to do in all but the most contrived cases is to perform your handling of byte-level representations with a dedicated a library that provides at least some abstraction over the representation details; even thin and unsafe abstractions like C structs are better than nothing. (Anything less than a library is imperfect: if all you've got is a separate process on a pipe, you've just traded one byte stream problem for another. Maybe the one you get is easier than the one you started with, but still admits the same kinds of incorrect byte interpretation errors.) And so "everything is a file", which was supposed to be great faciltiy to help put things together, is usually just an utterly irrelevant implementation detail beneath libraries. [...] https://en.wikipedia.org/wiki/Talk:Everything_is_a_file https://en.wikipedia.org/wiki/Talk:Everything_is_a_file Origin of the phrase "Everything is a File" The writings of the original UNIX people are well-scrutinized yet there is no sign of them having written this phrase ever. I think the phrase is not authentic. The article is self-defeating in that it states "some IPC" is a file, meaning "not everything is a file as some IPC is not. The phrase is used as a joke in at least one presentation. "What UNIX Cost Us" - Benno Rice (LCA 2020): https://www.youtube.com/watch?v=9-IWMbJXoLM#t=24m35s https://www.youtube.com/watch?v=9-IWMbJXoLM#t=24m35s >UNIX is a hell of a thing. From starting as a skunkworks project in Bell Labs to accidentally dominating the computer industry it's a huge part of the landscape that we work within. The thing is, was it the best thing we could have had? What could have been done better? >Join me for a bit of meditation on what else existed then, what was gained, what was lost, and what could (and should) be re-learned. >Complex problem have simple, easy to understand, wrong answers. >Understand the past, but don't let it bind the future.
- kazinator 3y agoThe discussion about /proc is a red herring. A /proc-like filesystem could use binary formats that would have better security (no formatted printing and scanning bugs) as well as performance. (While not to mention having disadvantages also.) The article somewhat conflates "everything is a file" with "everything is a text file". Also, "everything is a file descriptor" is usefully different from "everything is a file". Not every resource bound to a file descriptor is a file (in the sense that it was obtained from a name resolution in the filesystem). Things that are not files, but part of "everything is a file descriptor": socket(), socketpair(), pipe().
- deathanatos 3y agoBoth pipes and sockets are files, in the sense that they can appear on the filesystem. Some people refer to pipe(2) pipes as anonymous pipes — they're anonymous because they're not in the filesystem. (One in there is a "named" pipe.) (I do sort of wish most object creation in the FS was a pair of, int fd = create_a_file_descriptor_of_some_type(); link_fd(fd, "/the/name/for/it"); I think that'd be a cleaner abstraction, and then introduce syscalls on top of that for doing them together in case you think that's too much user/kernel traffic.)
- kevincox 3y agoLinux does have the O_TMPFILE flag to open. This creates a file with no name. But importantly it still needs a directory path because it needs to know which filesystem the data should be stored on. With your example if you wrote a few gigs of data to that fd before linking it the kernel would need to either buffer it (maybe swapping it out) or guess and then copy the data when it guesses wrong.
- ChrisArchitect 3y agoAside: why does this have a wikipedia article? A Unix concept like this should simply be a section on the base Unix article no? Surprised the editors didn't squash it
- _annum 3y agoEverything is a file :)
- NoZebra120vClip 3y agoThe article is about 12 years old. Unix is one of those topics on Wikipedia where experts wrote a lot of trivial articles, for every shell builtin and implementation details. Gradually, these are challenged for independent notability and merged into larger articles, or deemed to be no more than dictionary definitions. There are some interesting points raised on the Talk page: that this alleged philosophy wasn't really espoused or coined by the implementors, so there's one angle for you.