6 ms·
In Unix Everything Is a File
- lvh 8y agoAnd it looks like the server may have run out of files (well, descriptors) :-) google cache link: https://webcache.googleusercontent.com/search?q=cache:L6aopSQWczwJ:https://ph7spot.com/musings/in-unix-everything-is-a-file+&cd=1&hl=en&ct=clnk&gl=us&client=firefox-b-1-ab https://webcache.googleusercontent.com/search?q=cache:L6aopS...
- lucb1e 8y agoWordpress strikes again. Page is from 2014. Archive link: https://web.archive.org/web/20091119184152/https://ph7spot.com/musings/in-unix-everything-is-a-file https://web.archive.org/web/20091119184152/https://ph7spot.c... Edit: archive of 6 years ago with some styling: http://archive.is/B9He http://archive.is/B9He (The page actually loaded for me on the second try, and the design looks a lot more modern now.)
- rzzzt 8y agoWhich was the last Unix release/flavor where you could still read(2) a directory? All manpages that I've found say that you get EISDIR if you try it.
- Pete_D 8y agoI certainly remember being surprised when I accidentally cat'd a directory on AIX and it worked. I can't remember exactly what you get back; I think it's a NUL-separated list of the files in it.
- Someone 8y agoThat depends on the file system (and, possibly, the OS, but I think the OS should just forward the call to the relevant driver and return its result) FreeBSD’s man page says (https://www.freebsd.org/cgi/man.cgi?sektion=2&query=read https://www.freebsd.org/cgi/man.cgi?sektion=2&query=read): [EISDIR] The file descriptor is associated with a directory residing on a file system that does not allow regular read operations on directories (e.g. NFS) NetBSD says something similar, adding ”The readdir() function should be used instead.” I also am not convinced that, for example, the Linux man page (http://man7.org/linux/man-pages/man2/read.2.html http://man7.org/linux/man-pages/man2/read.2.html) guarantees you get EISDIR when reading from a directory. It says that if you get EISDIR, fd refers to a directory, but isn’t explicit about the converse.
- rzzzt 8y agoYou are right. The default implementation in Linux returns -EISDIR, but filesystems are free to change it to something else: https://github.com/torvalds/linux/blob/2a56bb596b2c1fb612f9988afda9655c8c872a6e/fs/libfs.c#L211-L214 https://github.com/torvalds/linux/blob/2a56bb596b2c1fb612f99... https://github.com/torvalds/linux/search?p=1&q=generic_read_dir&unscoped_q=generic_read_dir https://github.com/torvalds/linux/search?p=1&q=generic_read_...
- koenigdavidmj 8y agoFreeBSD would allow this as of a few years ago (2007 at the earliest).
- skibo 8y agoStill does. I'm running 12-CURRENT and it looks like UFS allows it. I occasionally cat or more a directory by accident and get a screen full of junk.
- j-pb 8y agoImagine if unix was invented today and you would navigate a folder by reading it, and having a JSON encoded file of its content returned. Slow but so elegant and pretty.
- arendtio 8y agoIf unix was invented today, you would have 3rd party files everywhere, executing randomly and typing the newest offers to your prompt automatically.
- LukeShu 8y agoI know that SunOS 5.10 could read(2) a directory (that's what lore.cs.purdue.edu was running).
- ArtWomb 8y agoIs the sendfile system call still the fastest way to transfer data locally on Linux systems? For communicating between processes running locally on system. Or, for example, a server running locally on an android device. Unix domain sockets typically beat TCP connections for most large file transfers. But is there anything faster? And what about transfers over the public net from machine to machine? I usually just use netcat, ssh, or scp.
- Pete_D 8y agoConsider trying rsync instead of scp - I've found it to be faster for multiple small files.
- SteveNuts 8y agoShared memory is faster than sockets
- rurban 8y agoBut unsafe.
- the8472 8y agoThat depends on what you're doing. Sendfile, splice and copy_file_range are the fastest if you want to shovel things back and forth between file descriptors. The latter supports reflink copy if the the underlying filesystem does. And I think there's some special casing in vmsplice that lets you transfer memory pages from process to process over a pipe. At least I've seen some patches for that, not sure if they made it into the kernel. Alternatively there's memfd. But if you want to coordinate processes you'll use a shared memory and futexes. If you just want to hand over an entire file to a different process you can send the file descriptor itself over a unix socket.
- zokier 8y agoI think the author might have confused Plan9 and UNIX.
- DonHopkins 8y agoHe also confused "The Art of UNIX Programming" with a "fantastic book".
- Lordarminius 8y agoWhat's wrong with the book ? I have read it and I think it provides a great introduction to Unix.
- DonHopkins 8y agoOne obvious problem with the book which this article demonstrates is that it leaves the reader with the mistaken impression that "In Unix Everything Is a File". I assumed this was common knowledge, but apparently not, so here goes: Another reason not to buy the book is that the author, Eric S Raymond, is a racist sexist Islamophobic conspiracy theory spreading global warming denying gun nut, who threatened to kill Bruce Perens. In summary, and just because you asked, Eric S Raymond is a mediocre programmer who has made a career of trying to pose as a hacker, define what a hacker is, tell people how to become a hacker, all while viciously, personally, professionally, and unfairly attacking actual successful influential hackers like Richard Stallman, including his philosophy and his life's work, by hijacking and distorting the definition of Free Software and the very definitions of words in the hacker's dictionary, to reflect his own extremist narcissistic racist political ideology, which most certainly does not align with the non-bigoted ideology in the Hacker's Manifesto: "We exist without skin color, without nationality, without religious bias." https://en.wikiquote.org/wiki/Eric_S._Raymond https://en.wikiquote.org/wiki/Eric_S._Raymond "In the U.S., blacks are 12% of the population but commit 50% of violent crimes; can anyone honestly think this is unconnected to the fact that they average 15 points of IQ lower than the general population? That stupid people are more violent is a fact independent of skin color." -Eric S Raymond "A clash of civilizations driven by the failure of Islamic/Arab culture (though I would stress the problem of the Islamic commandment to jihad more than he does). I think he [Steven den Beste] is also right to say that our long-term objective must be to break, crush and eventually destroy this culture, because we can't live on the same planet with people who both carry those memes and have access to weapons of mass destruction. They will hate us and seek to destroy us not for what we've done but for what we are." -Eric S Raymond "And for any agents or proxy of the regime interested in asking me questions face to face, I’ve got some bullets slathered in pork fat to make you feel extra special welcome." -Eric S Raymond "When I hear the words "social responsibility", I want to reach for my gun." -Eric S Raymond As much as he thinks he and everyone else should own lots guns, Eric S Raymond has recently stated (2017) that he believes blacks are too unintelligent to own or to be trained to use guns: https://twitter.com/tqbf/status/816449724127608833 https://twitter.com/tqbf/status/816449724127608833 >Unfortunately, this doesn't cover the BLM crowd, which would have an average IQ of about 85 if it's statistically representative of American blacks as a whole. I've never tried to train anyone that dim and wouldn't want to. https://rationalwiki.org/wiki/Eric_S._Raymond https://rationalwiki.org/wiki/Eric_S._Raymond Eric S. Raymond (born 1957) is a computer programmer, advocate of open source software, and author or editor of several written works, including The Cathedral and the Bazaar and the Jargon File. Coincidentally, "Yonder Racism" is an anagram of his name. Unfortunately, his work and expertise in computer technology[1][2] has been all but overshadowed by his batshit insane wingnut tendencies in the wake of 9/11, his misogynistic claims,[3] and his increasingly wanky ego-gazing,[4][5] as he acknowledges that people aren't so crazy about him anymore. https://en.wikipedia.org/wiki/Talk:Russ_Nelson#Blacks_are_lazy https://en.wikipedia.org/wiki/Talk:Russ_Nelson#Blacks_are_la... Eric S Raymond threw down the gauntlet and defended Russ Nelson's infamous "Blacks are Lazy" article by calling everyone who wanted him to resign his position as president of OSI "fools and thugs", and demanded OSI spend their precious time and resources fighting critics of racism and defending Russ Nelson's racist beliefs which ESR shares, instead of promoting open source software. http://www.eweek.com/servers/new-osi-president-steps-down http://www.eweek.com/servers/new-osi-president-steps-down >“Abetting somebody elses witch hunt is no less disgusting than starting your own.” “Personally, I wanted to fight this on principle,” Raymond said. “Russ resigned the presidency rather than get OSI into that fight, and the board quite properly respected his wishes in the matter. That sacrifice makes me angrier at the fools and thugs who pulled him down.” This would be funny if it weren't exactly like Trump's recent "'nt" flip-flop on Putin: >Nelson subsequently sought to explain himself in an essay titled "Blacks are Not Lazy." https://encyclopediadramatica.rs/Eric_S._Raymond https://encyclopediadramatica.rs/Eric_S._Raymond In 1999, Debian developer Bruce Perens published an "email threat" that he allegedly received from Raymond. Raymond then "clarified" that he only meant to defame Perens. From this we can assume that he is batshit insane and will fucking kill and or write to anyone that says anything about him or his software. If you are lucky you might get an O'Rielly book about you. https://lists.debian.org/debian-user/1999/04/msg00623.html https://lists.debian.org/debian-user/1999/04/msg00623.html To: debian-user@lists.debian.org Subject: email threat From: bruce@perens.com Date: 5 Apr 1999 22:48:42 -0000 Message-id: <19990405224842.2386.qmail@perens.com> Today I received the following threat in e-mail from Eric Raymond. The message was copied to the Silicon Valley Linux User's Group officers, who you may consult regarding its authenticity. The police have been notified. Because I know that Eric is a firearms enthusiast, for my own protection, I feel the best strategy is for me to publicize the threat widely. Thanks Bruce Perens > Damn straight I took it personally. And if you ever again behave like > that kind of disruptive asshole in public, insult me, and jeopardize > the interests of our entire tribe, I'll take it just as personally -- > and I will find a way to make you regret it. Watch your step. The irony of Eric S Raymond threatening someone else for behaving "like that kind of disruptive asshole in public" is rich -- very rich. https://geekz.co.uk/lovesraymond/cat/bruce-perens/page/10 https://geekz.co.uk/lovesraymond/cat/bruce-perens/page/10 https://geekz.co.uk/lovesraymond/archive/terrorismistic https://geekz.co.uk/lovesraymond/archive/terrorismistic
- Pete_D 8y ago"Everything is a file" is a nice idea not taken far enough. The execution is full of frustrating omissions and holes at the edges, e.g.: - unless you are using Plan 9 or gpm, reading mouse events is almost certainly going to be a lot more involved than opening /dev/mouse - you can't portably use select() to wait on pthread conditions - lots of useful things (like POSIX shared memory objects) are file descriptors, but aren't filesystem namespaced Sometimes I wonder what the world would look like if Plan 9 had won.
- pjmlp 8y agoThat is easy, microservices über all. I never had such appreciation for Plan 9, as I consider Inferno the best design of what they were trying to achieve. The only caveat was that they never came around to add an ahead of time compilation support to DisVM.
- pjmlp 8y agoAny UNIX old timer knows that actually: In UNIX everything is a file, except when it is not.
- majewsky 8y ago> In Plan 9, everything is a file Fixed that.
- rdtsc 8y ago> > In Plan 9, everything is a file > Fixed that. It was a short snarky comment. Also the article mentions that too at the bottom. Yeah they "fixed it" but by that time not enough people cared. The abstraction purity was nice and developers and OS fans loved it, but it just didn't matter for most application. Another way to put it is that Linux was already fairly decent and it wasn't worth switching to a whole new OS just to have everything be a file. Another obstacle was that Plan 9 wasn't marketed or pushed enough. Imagine if Google backed Plan 9 and pushed it as the Android OS, put it on Chromebooks, updated it regularly etc. That would be enough for people to switch to it.
- pjmlp 8y agoAs we discussed the other day, there was still some issues with stuff like graphical performance with such a concept, suitable for gaming and such. In any case, Inferno with Limbo would be much better, and to pick on your example not much different from Android's architecture.
- yepguy 8y agoI understand why it didn't take off, but I don't think it's true that Plan 9 was just a tiny incremental improvement over UNIX's architecture. Modern networking and distributed systems would be a LOT nicer today if Plan 9 had taken off in the 90s. I think the economic benefit over today's state of affairs would have been massive, not just something that appealed to computer geeks' sense of elegance.
- 8y ago
- zmix 8y agoAnd if you use XML, all you need to do, is to add another forward slash after the filename. And now, the network, the host and the data have become one: Everything is a node (or an atomic value). But hey, that's stupid. Let's not do that...!
- techbio 8y agoBy way of XPath. I don't disagree but--that's stupid because of... using XML at all? Complications of altering/documenting ad hoc taxonomies? Mixing paradigms under a unified syntax? Something else?
- zmix 8y agoWould you mind giving me the one or other example? Or references to some documents about this (in regard to XML), that I could reach?
- nailer 8y agoAre there Unixes where TCP / UDP ports are files? setcap on Linux gets really old when you just want to give a user access to a porf.
- squirrelicus 8y agoI can't say for sure, but i do know ucspi-tcp is a great tool that wraps the noise for you into FDs. But it's only useful if you can tank the cost of invoking a process per tcp connection. https://cr.yp.to/ucspi-tcp.html https://cr.yp.to/ucspi-tcp.html
- epx 8y agoI like the Linus Torvalds' interpretation: that (almost) everything is actually a file descriptor. The few things that are not, like fork() returns, are subject to criticism (DJB wrote about how fork() should return an fd, I just couldn't find the URL right now.)
- severine 8y agoThis? http://cr.yp.to/docs/selfpipe.html http://cr.yp.to/docs/selfpipe.html > Richard Stevens's 1992 book ``Advanced programming in the UNIX environment'' says that you can't safely mix select() or poll() with SIGCHLD (or other signals). The SIGCHLD might go off while select() is starting, too early to interrupt it, too late to change its timeout. Solution: the self-pipe trick. Maintain a pipe and select for readability on the pipe input. Inside the SIGCHLD handler, write a byte (non-blocking, just in case) to the pipe output. Done. Of course, the Right Thing would be to have fork() return a file descriptor, not a process ID.
- jiveturkey 8y agothis needs a 2009 tag. it should actually be removed as the article is wrong. even in 2009, how could someone get this wrong? pretty sure 2009 was finally going to be the year of the linux desktop! IOW unix had reached the masses by then.