10 ms·

What Unix Cost Us [video]

152 points by stargrave 7y ago

kccqzy 7y ago
I watched the video. A few good points: should everything really be a file? He used the example of simply talking to a USB device on Linux without libusb involving a bunch of snprintf to construct a file path, and then a big bunch of ioctl. Setting up a device apparently requires creating a bunch of magic directories and magic files and magic symlinks and mounting magic filesystems... I feel that he has a good point about this. Not sure how Plan 9 does this better though. Besides, he also talked about how I/O was blocking, and even when non-blocking I/O was available it's still synchronous until the recent introduction of io_uring. Windows appears to do this "right" with its completion ports. This part of the video was much less convincing, IMO.
- h1x 7y ago
  Also thought that some of the technical points were good, but... Mixing together Linux and Unix? Saying that killing a process from command line in Linux is bad because MacOS has GUI for that? Not explaining in more detail a claim that C is responsible for Spectre? Topping a technical presentation with political statements? That's too much for 30 minutes I think.
  - catalogia 7y ago
    The Activity Monitor is hardly even the fastest way to kill a process in macOS. Usually when I'm working on something that goes haywire needs killing, I already know the process name and a simple `killall example` can be typed out faster than he can ⌘-space his way into the Activity Monitor and navigate that GUI. I don't think that proves anything profound though, just that his example was weak. I went into this video agreeing with the stated premise (and still do), but this presentation was a bit underwhelming.
    - celticmusic 7y ago
      linux has pkill as well.
  - H_Pylori 7y ago
    >Mixing together Linux and Unix? Oh, the horror. For all intent and purposes they are exactly the same and they share the same philosophy.
  - celticmusic 7y ago
    yeah, the whole conduct thing was weird and felt like part of a different talk. I kept expecting him to try and bring it back to the rest of his talk and he never really did outside of it being somewhat related to the idea of "lets do things differently in the future".
  - kragen 7y ago
    > Not explaining in more detail a claim that C is responsible for Spectre? He said trying to paper over the concurrency of the hardware to provide the simple in-order model C expects is responsible for Spectre (and he did explain how Spectre works). This is true but I don't think C is really the reason for this; designing a language that productively exposes parallelism but is easy to program the kinds of things that run fast on modern amd64 CPUs, I think that might be an unsolved problem.
  - IshKebab 7y ago
    > Saying that killing a process from command line in Linux is bad because MacOS has GUI for that? He wasn't saying that `ps | grep | kill` is bad because MacOS has a GUI. He was saying that it is bad (it is, that's pretty undeniable), and that dogmatically following the "Unix philosophy" would lead you to conclude that it is great, even though there's an obviously better way to do it (through a nice GUI). To put it another way, "do one thing and do it well" is fine, if all of your users are happy to write all the glue to stick tiny programs together, and the only glue you have is completely unstructured text streams. But most people are not willing to do that. To take it to an obvious extreme: what is the Unix version of Excel? `set_cell A4 'hello' && recalculate_formula B5 && cat A1:D2 | pi_chart`??
    - h1x 7y ago
      > To take it to an obvious extreme: what is the Unix version of Excel? `set_cell A4 'hello' && recalculate_formula B5 && cat A1:D2 | pi_chart`?? That's a nice question. It's not that obvious extreme for me though. You could use sc and GNU plot (I know... GNU's not Unix). Simple example: - first you configure your plot $ echo "set terminal png\nset output 'plot.png'\nplot '/dev/stdin' with lines" > plotcfg.plg - then you create a spreadsheet and pass it to gnuplot. It's easier to edit the spreadsheet with sc's TUI $ echo 'let A0 = 1\nlet B0 = 8\nlet A1 = 2\nlet B1 = 16\nlet A2 = 3\nlet B2 = 32\n' | sc -W '%' - | gnuplot plotcfg.plg
      - IshKebab 7y ago
        So easy!
  - fetbaffe 7y ago
    Here is an article explaining Spectre & Meltdown with regards to C. https://queue.acm.org/detail.cfm?id=3212479 https://queue.acm.org/detail.cfm?id=3212479 It has been discussed previously multiple times here at HN, the latest from three weeks ago. https://news.ycombinator.com/item?id=21888096 https://news.ycombinator.com/item?id=21888096
- hinkley 7y ago
  There’s a phrase I haven’t heard in about forever unless I’m the one saying it: “passing a giggle test”. If you can’t describe the thing you did with a straight face, you should take it back and keep working in it until you can. It’s so easy for things to get away from us by increments, and the longer we sit with something the harder a time we have predicting the reaction of people who have just seen it for the first time.
- qznc 7y ago
  My usual counter argument for "everything a file" is real time constraints. Playing audio or video (includes all games) is a soft real time task. The plan9 file API does not support that.
robert_tweed 7y ago
I'd like to know, from anyone with Plan 9 experience, does its approach to "everything is a file" solve these problems?
- pjmlp 7y ago
  No, everything is a file doesn't work at all if you care about high performance graphics and real time audio.
  - kragen 7y ago
    qznc made the same claim. Why would "everything is a file" be incompatible with high-performance graphics and real-time audio? I'm particularly interested in this question because my knowledge of those areas is not very deep, so I'm wondering what unknown unknowns I'm missing. Here's my naïve thinking: · Real-time audio requires low-latency scheduling, which is unrelated to the API for audio itself, and FIFO depth monitoring so that you don't get either buffer overruns or buffers that are so full that they induce latency in excess of what your scheduling and processing latency unavoidably adds. For output, you could provide this with a /dev/audio for writing data and a /dev/audiooutputbuffersize for reading the FIFO buffer depth. For input, you could provide this by making reads from /dev/audio nonblocking, by providing a /dev/audioinputbuffersize you would read before risking a blocking read on /dev/audio, or by reading in a separate thread. · High-performance graphics covers a lot of ground, but one of the big pain points is unnecessary memcpying on the paper path from your CPU graphics program into the GPU, whether that's textures, vertex buffers, or prerendered pixel data (though maybe we could argue that that last case is never going to be high performance.) One way that unnecessary memcpying arises is from buffers for write() that are not correctly aligned with respect to the start of a page, which unavoidably means that they will have to get copied to a place with the correct alignment. This is a big problem if you're streaming out gigabytes per second of graphics data. In particular you don't want the data that goes to the GPU concatenated with data that tells the kernel graphics driver what to do with it. Solution: open a new file in a /dev/gfx directory for the data to be sent to the GPU, then write the data to it from a properly aligned buffer. Fork a separate thread to write this data, which is done by marking the pages COW rather than copying them. (A more radical and less sneaky solution I'm exploring for Wercam and BubbleOS is to transfer ownership of pixel data buffers from the application to the window system, causing them to disappear from the application's memory map, avoiding the need to either copy or COW them. But this definitely departs from the Unix read()/write() model.) · A different aspect of high-performance graphics, other than raw throughput, is that you might need to synchronize your drawing with monitor refreshes to avoid an extra half-frame of latency, particularly at low refresh rates like 60 Hz. This seems straightforward to solve in a filesystem interface: reading from /dev/vbi produces a byte just before each VBI, giving you time to send any necessary new data to the GPU. Certainly it's true that Plan 9 does not provide high-performance graphics or low-latency audio. But I don't think that proves that filesystem-based interfaces can't provide them, just that the Plan 9 designers had spent the 1980s researching DSLs, fault-tolerance, concurrency, and typesetting, not rasterization, real-time control, and animation. In general, system-call or IPC interfaces of the form "invoke function foobar with a baz handle and a quux struct or memory buffer" can be cleanly replaced with "open /bazzes/$i/foobar and write a quux struct or memory buffer to it", can't they? I mean, that's three system calls instead of one, so about 1 μs of system-call overhead instead of 300 ns, but in the cases where that's the performance bottleneck you can usually solve it by writing N quuxes instead of one. Can't you? A potential problem arises when you need to combine multiple handles to extra-process resources in a single system call, like SCM_RIGHTS or rename(), but I don't know where those would come up in the particular application domains you identify as problematic. But maybe my proposed solutions above won't solve the problems I think they will, or maybe I'm missing the biggest issues entirely. What am I missing? (I'm not claiming that this approach would be easier, safer, or more discoverable than the approach of adding a bunch of system calls; Benno's talk discusses the many ingenious ways a filesystem-based API can be hard to use, and no doubt some of the same criticisms can be leveled at the above strawman proposals. I'm just saying I don't see where it inherently fails to meet performance requirements.)
pjmlp 7y ago
Very entertaining talk, specially regarding the cargo cult that gets carried around.
thunderbong 7y ago
Wonderful talk. Lot's of interesting points. Much of the technical went over my head though. But from what I understood, the main point he's trying to make was - we are stuck with a way of thinking in technology which was right when it was thought up, but has run out of it's usefulness. I found his point, with respect to *nix, very insightful - > Unix suited it's time. I worry it has ended up straight-jacketing the way that we think because that was quite a while ago. It still works, which is amazing. But that doesn't mean that it's tenets and it's way of work should be sacrosanct. We should feel free to examine every idea and throw them out if we feel they no longer have value for what we're doing. The analogies from history and communities were also very interesting (e.g. meritocracy). Other points that I liked - - Complex problems have simple, easy to understand, wrong answers. - Understand the past, but don't let it bind the future
linguae 7y ago
I thought this talk was an interesting overview of some of the limitations of Unix (particularly Linux) and C, although I wish the speaker had discussed more details as to why C is not a match for modern processor architectures and how modern languages such as Rust address these issues. Another thing that I'm curious about is how much familiarity the speaker has with Plan 9? Plan 9 from Bell Labs is the spiritual successor to Unix, created by many of the Bell Labs researchers who originally worked on Unix. The creators of Plan 9 pushed "everything is a file" to its limits. Sadly, although I have read some Plan 9 papers, I haven't actually used the operating system. I'm really curious about what the USB example in this talk would look like in Plan 9? I do wish there was more development in alternative operating systems. I like Unix, but there were many ideas from VMS, Smalltalk, the Genera operating system for Symbolics LISP machines, the classic Mac OS, BeOS, Plan 9, and other operating systems that we can learn from and that would be very useful to have today. Also, today's hardware is quite different from the hardware of 50 years ago. We have multi-core CPUs, GPUs with hundreds of cores, very fast NVMe storage devices, and other amazing technologies. There has also been much advancements in programming languages, with more people willing to explore alternatives to C and C++ such as Python, Rust, Go, Clojure, Haskell, Swift, and many more, each providing different abstractions for dealing with diverse programming tasks. I wonder what an operating system for the 2020s would look like given the advances in technology and the lessons learned in the past decade.
- jammygit 7y ago
  He gave a good talk about SystemD once too: https://www.youtube.com/watch?v=o_AIw9bGogo https://www.youtube.com/watch?v=o_AIw9bGogo edit: found a version with better camera angles
  - sprash 7y ago
    I didn't like that talk at all. I disagree with almost everything he says[1]. This talk is much better but some of his criticism are rather pointless because what he wants is a complete new OS design vastly different from UNIX. If he likes Windows/MacOS so much he should just use that. 1.: https://news.ycombinator.com/item?id=19966321 https://news.ycombinator.com/item?id=19966321
dscpls 7y ago
Awesome talk. Especially connecting how leaving an entrenched mindset behind applies both to architecture and to diversity in our communities. And that our meritocracies are not as pure as we'd like to think.
- dscpls 7y ago
  Lol - pointing out the elephant in the room really bugged someone
  - rumanator 7y ago
    Your post has nothing to do with pointing elephants. You've just threw a bunch of baseless assertions mixed with gratuitous blanket personal attacks. Ignorance mixed with resentment is never helpful.
    - dscpls 7y ago
      Please point out my - baseless assertions - gratuitous blanket personal attacks Right now it looks like that's what you're doing, because I don't see that in my comment. Did you watch the talk to the end? Edit: line breaks
      - wizzwizz4 7y ago
        I cannot refute your comment, because it is not actually clear what you are referring to. Your comment is too vague to be useful; it barely conveys information above a sentiment. An unfalsifiable theory is useless. An unfalsifiable comment equally so.
        dscpls 7y ago
        I'm literally repeating words and phrases from the talk.
        lightedman 7y ago
        You just clearly demonstrated that you did NOT watch the entire talk through to the end. What OP stated is in fact present in the talk itself.
incompatible 7y ago
I suppose on Linux, libusb itself is supposed to be the general programming interface, not the poking around with /dev or ioctl. The Windows and Mac interfaces he was using are quite possibly implemented as libraries over a more primitive layer. Speculation, knowing nothing about USB specifically.
- pjmlp 7y ago
  On Windows and Mac, the more primitive layer are kernel private APIs, which you aren't supposed to call directly.
  - magicalhippo 7y ago
    Linux is the kernel, so really the equivalent on Windows is the Windows kernel API, not the Win32 (user) API. However, the kernel API seems to be somewhat similar to the Win32 version presented in the talk and not nearly as gnarly as the Linux equivalent: https://github.com/microsoft/Windows-driver-samples/blob/master/hid/hidusbfx2/sys/usb.c https://github.com/microsoft/Windows-driver-samples/blob/mas... https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/wdfusb/ https://docs.microsoft.com/en-us/windows-hardware/drivers/dd... That said I'm just an application programmer so I've only ever used the Win32 API.
clktmr 7y ago
"Everything is a file" is one of the most misunderstood concepts I encountered, to the point where people think that disk IO is involved. A file doesn't even need a filename (eg sockets). The main point is to have a common interface for system resources, which happens to be the file api in Linux. Think of it as the base class of everything. There is a bunch of tools/code that operates on file descriptors, which you can reuse now. Build better interfaces on top of that if you want to.
- discreteevent 7y ago
  Everything is a bytestream.
- wbl 7y ago
  Sockets depart badly from that model with special purpose setup syscalls.
  - clktmr 7y ago
    Yeah but that's rather because the socket api is older than Linux. Underneath there is the sockfs implementation.
soraminazuki 7y ago
Not everything may be a file indeed, but I didn't get what he was trying to point out with the specific example he gave. To make his point, he compared the USB APIs provided by Windows, macOS, and Linux. Then he went ranting about how Linux looks terrible because you had to manipulate files and use functions such as snprintf, ioctl, and fnctl. However, this is not at all a fair comparison because he was comparing the Linux kernel interface against much higher-level userspace library APIs in Windows and macOS. So what are the problems he encountered with this specific example, that had actually harmful consequences? I honestly can't tell.
- pjmlp 7y ago
  It was completely fair, because other OSes don't expect users to directly access kernel interfaces, most of them even forbid it, other than for writing device drivers purposes.
  - clktmr 7y ago
    Linux doesn't expect you to either. You are free to choose.
  - soraminazuki 7y ago
    How can it be "completely fair" when you're not comparing the same thing? Linux applications can and should use higher-level libraries too.
    - pjmlp 7y ago
      That is the low level version available to applications on Mac and Windows. The high level ones are Mac Frameworks in Objective-C/Swift and .NET/COM libraries. Direct kernel access is out of bounds.
    - blablabla123 7y ago
      On the other hand it's possible to use Syscalls on macOS or Linux to try out new low level functions, since these tend to be documented quite well unlike on Windows. For instance copy-on-write or ptrace to alter/inspect low-level calls.
swiley 7y ago
Unix at least picked one abstraction and stuck with it, contrast windows which is a dizzying mess of incompatible ideas. It’s so bad that more paranoid people may think Microsoft disorients people on purpose. There are other operating systems that also chose singular abstractions like the cannon cat and smalltalk.
- teddyh 7y ago
  Reference: https://en.wikipedia.org/wiki/Canon_Cat https://en.wikipedia.org/wiki/Canon_Cat
BlackLotus89 7y ago
This talk took some weird directions. It's called "What Unix Cost Us", but to be honest it isn't. Starts with USB-driver coding in Win, Mac, Linux. Goes over to colonialism. Made some good points about Computer-architecture and how they shape programming languages and then some bad/wrong points. Ends on community cultures and some controversial thoughts about gender equality and then he doesn't take any questions. Talks off a shitstorm and then isn't ready to face it. Don't know have seen better videos about the same topic.
- magicalhippo 7y ago
  It had some interesting points, but also some weird detours. One of his core points revolve around hanging on to the way things were, and how that often ends up poorly. Though I got this feeling that really, it's that often it's just really difficult to come up with good abstractions. Abstractions is after all what allows us to be so productive with our hardware. For example, Electron and similar is abstracting away the operating system, which makes sense because for a lot of programs the specifics of the operating system doesn't really matter. However JavaScript and HTML can hardly be described as the best way of obtaining that abstraction.
layoutIfNeeded 7y ago
I’m 29 now. I wonder if I’ll live long enough to be able to use cancellable non-blocking mkdir.
kragen 7y ago
I'm skeptical of Benno's claim that we should consider meritocracy "a dirty word" because "it's a lie". Certainly it is true that no community of practice achieves meritocracy, just as no polity achieves democracy — there are always some citizens with more influence than others, so it is always possible for the government to act against the interests and values of the majority of its population. Should we therefore consider democracy "a dirty word" because democracy "is a lie"? Perhaps it is better to consider it an ideal to which we aspire, without feigning to have achieved it. Let's consider what alternative ideals are available in place of meritocracy for governing a community of intellectual practice. We could strive for a democracy, in which the decisions are made by the majority — but the majority of whom? For this to meaningfully distinguish a community of intellectual practice from the surrounding community from which it arose, as a lotus blossom arises spotless from the swamp, a distinction must be made between voting members and outsiders. (Can you imagine a Linux User's Group where all the presentations are about Microsoft Windows, or a Python conference where all the talks are about Java?) But that is just a way of postponing the question of who the voters are. We could strive for consensus, like the Quakers, in which any collective decision is postponed until every member agrees; but, like democracy, that demands gatekeeping that draws an ingroup/outgroup distinction, so it is not really an answer to the question of who governs, just how they govern. We could strive for anarchy, in which all decisions are made individually, and there are thus no collective decisions to be made, whether by the meritorious or by anyone else. A variant of anarchy is "do-ocracy", where decisions are made by whoever shows up and makes the effort required to implement them. We could strive for a gerontocracy, in which the oldest members — perhaps by length of membership rather than by physical age — make the collective decisions. We could strive for a high-school clique, where the decisions are made by whoever is most popular. We could strive for a plutocracy, where the decisions are made by whoever is wealthiest, or who donates the most. Given these alternatives, it seems to me that when anarchy and consensus demand unacceptable tradeoffs, the least undesirable alternative is meritocracy. In meritocracy, the decisions are made by the best members of the group, according to some measure of merit that seems worthwhile to the group; in a community of intellectual practice, this usually amounts to some kind of knowledge and skill, seasoned with judgment and perhaps a guess about aptitude. If they are the wisest members, then they will make the best decisions. The greatest foolishness is to subject the wise to the government of the foolish. If some of the foolish and ignorant currently are so simply because they have not had the opportunity to learn, we can best remedy that by guiding them to learn from the wise, not by putting the foolish and ignorant in charge. Of course, meritocracy as an ideal cannot be reached, only striven for; but it is a better ideal to strive for than a high-school clique, a gerontocracy, or a plutocracy. Which of these are the campaigners against meritocracy hoping for?
gdm85 7y ago
Some parts of the talk did not really stitch together; if we consider the main arguments against Unix (and the funny dab at Plan9) I felt them a bit lacking in the sense that they do not propose alternative philosophies, but only a liberating "mixing things together is cool". I can totally understand that as a form of catharsis, and a mindset totally apt for hobby projects, but if we are playing in the realm of OS design...I would think we can do better in terms of architecture and philosophy? "anything else is better" does not really convince me, some structure is usually better than none at all.
rs23296008n1 7y ago
I think I prefer the plan9 approach of "everything is a file/filesystem" rather than the unix "everything is a file" as typically seen. Actual usage of files, filesystems and namespaces under plan9 was more consistent than the file approach under linux. It was a lot of fun playing in plan9. Regarding the video, of the three examples the windows approach was shown as more succinct. I thought the linux version was messy. The mac version seemed overly complex. Whether these were actually representative is another matter. I can't say how but I'm not convinced - there seemed to be some artistic license / suspension of belief required. The remainder seemed to be meandering but I sense you had to be there.
Aqueous 7y ago
am i the only one who thinks it’s a good thing that his Linux example was shorter than the other two and required less calling into hyper-specific API functionality?

152 points by stargrave 7y ago

I watched the video. A few good points: should everything really be a file? He used the example of simply talking to a USB device on Linux without libusb involving a bunch of snprintf to construct a file path, and then a big bunch of ioctl. Setting up a device apparently requires creating a bunch of magic directories and magic files and magic symlinks and mounting magic filesystems... I feel that he has a good point about this. Not sure how Plan 9 does this better though. Besides, he also talked about how I/O was blocking, and even when non-blocking I/O was available it's still synchronous until the recent introduction of io_uring. Windows appears to do this "right" with its completion ports. This part of the video was much less convincing, IMO.

h1x 7y ago
Also thought that some of the technical points were good, but... Mixing together Linux and Unix? Saying that killing a process from command line in Linux is bad because MacOS has GUI for that? Not explaining in more detail a claim that C is responsible for Spectre? Topping a technical presentation with political statements? That's too much for 30 minutes I think.
- catalogia 7y ago
  The Activity Monitor is hardly even the fastest way to kill a process in macOS. Usually when I'm working on something that goes haywire needs killing, I already know the process name and a simple `killall example` can be typed out faster than he can ⌘-space his way into the Activity Monitor and navigate that GUI. I don't think that proves anything profound though, just that his example was weak. I went into this video agreeing with the stated premise (and still do), but this presentation was a bit underwhelming.
  - celticmusic 7y ago
    linux has pkill as well.
- H_Pylori 7y ago
  >Mixing together Linux and Unix? Oh, the horror. For all intent and purposes they are exactly the same and they share the same philosophy.
- celticmusic 7y ago
  yeah, the whole conduct thing was weird and felt like part of a different talk. I kept expecting him to try and bring it back to the rest of his talk and he never really did outside of it being somewhat related to the idea of "lets do things differently in the future".
- kragen 7y ago
  > Not explaining in more detail a claim that C is responsible for Spectre? He said trying to paper over the concurrency of the hardware to provide the simple in-order model C expects is responsible for Spectre (and he did explain how Spectre works). This is true but I don't think C is really the reason for this; designing a language that productively exposes parallelism but is easy to program the kinds of things that run fast on modern amd64 CPUs, I think that might be an unsolved problem.
- IshKebab 7y ago
  > Saying that killing a process from command line in Linux is bad because MacOS has GUI for that? He wasn't saying that `ps | grep | kill` is bad because MacOS has a GUI. He was saying that it is bad (it is, that's pretty undeniable), and that dogmatically following the "Unix philosophy" would lead you to conclude that it is great, even though there's an obviously better way to do it (through a nice GUI). To put it another way, "do one thing and do it well" is fine, if all of your users are happy to write all the glue to stick tiny programs together, and the only glue you have is completely unstructured text streams. But most people are not willing to do that. To take it to an obvious extreme: what is the Unix version of Excel? `set_cell A4 'hello' && recalculate_formula B5 && cat A1:D2 | pi_chart`??
  - h1x 7y ago
    > To take it to an obvious extreme: what is the Unix version of Excel? `set_cell A4 'hello' && recalculate_formula B5 && cat A1:D2 | pi_chart`?? That's a nice question. It's not that obvious extreme for me though. You could use sc and GNU plot (I know... GNU's not Unix). Simple example: - first you configure your plot $ echo "set terminal png\nset output 'plot.png'\nplot '/dev/stdin' with lines" > plotcfg.plg - then you create a spreadsheet and pass it to gnuplot. It's easier to edit the spreadsheet with sc's TUI $ echo 'let A0 = 1\nlet B0 = 8\nlet A1 = 2\nlet B1 = 16\nlet A2 = 3\nlet B2 = 32\n' | sc -W '%' - | gnuplot plotcfg.plg
    - IshKebab 7y ago
      So easy!
- fetbaffe 7y ago
  Here is an article explaining Spectre & Meltdown with regards to C. https://queue.acm.org/detail.cfm?id=3212479 https://queue.acm.org/detail.cfm?id=3212479 It has been discussed previously multiple times here at HN, the latest from three weeks ago. https://news.ycombinator.com/item?id=21888096 https://news.ycombinator.com/item?id=21888096
hinkley 7y ago
There’s a phrase I haven’t heard in about forever unless I’m the one saying it: “passing a giggle test”. If you can’t describe the thing you did with a straight face, you should take it back and keep working in it until you can. It’s so easy for things to get away from us by increments, and the longer we sit with something the harder a time we have predicting the reaction of people who have just seen it for the first time.
qznc 7y ago
My usual counter argument for "everything a file" is real time constraints. Playing audio or video (includes all games) is a soft real time task. The plan9 file API does not support that.

robert_tweed 7y ago

I'd like to know, from anyone with Plan 9 experience, does its approach to "everything is a file" solve these problems?

pjmlp 7y ago
No, everything is a file doesn't work at all if you care about high performance graphics and real time audio.
- kragen 7y ago
  qznc made the same claim. Why would "everything is a file" be incompatible with high-performance graphics and real-time audio? I'm particularly interested in this question because my knowledge of those areas is not very deep, so I'm wondering what unknown unknowns I'm missing. Here's my naïve thinking: · Real-time audio requires low-latency scheduling, which is unrelated to the API for audio itself, and FIFO depth monitoring so that you don't get either buffer overruns or buffers that are so full that they induce latency in excess of what your scheduling and processing latency unavoidably adds. For output, you could provide this with a /dev/audio for writing data and a /dev/audiooutputbuffersize for reading the FIFO buffer depth. For input, you could provide this by making reads from /dev/audio nonblocking, by providing a /dev/audioinputbuffersize you would read before risking a blocking read on /dev/audio, or by reading in a separate thread. · High-performance graphics covers a lot of ground, but one of the big pain points is unnecessary memcpying on the paper path from your CPU graphics program into the GPU, whether that's textures, vertex buffers, or prerendered pixel data (though maybe we could argue that that last case is never going to be high performance.) One way that unnecessary memcpying arises is from buffers for write() that are not correctly aligned with respect to the start of a page, which unavoidably means that they will have to get copied to a place with the correct alignment. This is a big problem if you're streaming out gigabytes per second of graphics data. In particular you don't want the data that goes to the GPU concatenated with data that tells the kernel graphics driver what to do with it. Solution: open a new file in a /dev/gfx directory for the data to be sent to the GPU, then write the data to it from a properly aligned buffer. Fork a separate thread to write this data, which is done by marking the pages COW rather than copying them. (A more radical and less sneaky solution I'm exploring for Wercam and BubbleOS is to transfer ownership of pixel data buffers from the application to the window system, causing them to disappear from the application's memory map, avoiding the need to either copy or COW them. But this definitely departs from the Unix read()/write() model.) · A different aspect of high-performance graphics, other than raw throughput, is that you might need to synchronize your drawing with monitor refreshes to avoid an extra half-frame of latency, particularly at low refresh rates like 60 Hz. This seems straightforward to solve in a filesystem interface: reading from /dev/vbi produces a byte just before each VBI, giving you time to send any necessary new data to the GPU. Certainly it's true that Plan 9 does not provide high-performance graphics or low-latency audio. But I don't think that proves that filesystem-based interfaces can't provide them, just that the Plan 9 designers had spent the 1980s researching DSLs, fault-tolerance, concurrency, and typesetting, not rasterization, real-time control, and animation. In general, system-call or IPC interfaces of the form "invoke function foobar with a baz handle and a quux struct or memory buffer" can be cleanly replaced with "open /bazzes/$i/foobar and write a quux struct or memory buffer to it", can't they? I mean, that's three system calls instead of one, so about 1 μs of system-call overhead instead of 300 ns, but in the cases where that's the performance bottleneck you can usually solve it by writing N quuxes instead of one. Can't you? A potential problem arises when you need to combine multiple handles to extra-process resources in a single system call, like SCM_RIGHTS or rename(), but I don't know where those would come up in the particular application domains you identify as problematic. But maybe my proposed solutions above won't solve the problems I think they will, or maybe I'm missing the biggest issues entirely. What am I missing? (I'm not claiming that this approach would be easier, safer, or more discoverable than the approach of adding a bunch of system calls; Benno's talk discusses the many ingenious ways a filesystem-based API can be hard to use, and no doubt some of the same criticisms can be leveled at the above strawman proposals. I'm just saying I don't see where it inherently fails to meet performance requirements.)

pjmlp 7y ago

Very entertaining talk, specially regarding the cargo cult that gets carried around.

thunderbong 7y ago

Wonderful talk. Lot's of interesting points. Much of the technical went over my head though. But from what I understood, the main point he's trying to make was - we are stuck with a way of thinking in technology which was right when it was thought up, but has run out of it's usefulness. I found his point, with respect to *nix, very insightful - > Unix suited it's time. I worry it has ended up straight-jacketing the way that we think because that was quite a while ago. It still works, which is amazing. But that doesn't mean that it's tenets and it's way of work should be sacrosanct. We should feel free to examine every idea and throw them out if we feel they no longer have value for what we're doing. The analogies from history and communities were also very interesting (e.g. meritocracy). Other points that I liked - - Complex problems have simple, easy to understand, wrong answers. - Understand the past, but don't let it bind the future

linguae 7y ago

I thought this talk was an interesting overview of some of the limitations of Unix (particularly Linux) and C, although I wish the speaker had discussed more details as to why C is not a match for modern processor architectures and how modern languages such as Rust address these issues. Another thing that I'm curious about is how much familiarity the speaker has with Plan 9? Plan 9 from Bell Labs is the spiritual successor to Unix, created by many of the Bell Labs researchers who originally worked on Unix. The creators of Plan 9 pushed "everything is a file" to its limits. Sadly, although I have read some Plan 9 papers, I haven't actually used the operating system. I'm really curious about what the USB example in this talk would look like in Plan 9? I do wish there was more development in alternative operating systems. I like Unix, but there were many ideas from VMS, Smalltalk, the Genera operating system for Symbolics LISP machines, the classic Mac OS, BeOS, Plan 9, and other operating systems that we can learn from and that would be very useful to have today. Also, today's hardware is quite different from the hardware of 50 years ago. We have multi-core CPUs, GPUs with hundreds of cores, very fast NVMe storage devices, and other amazing technologies. There has also been much advancements in programming languages, with more people willing to explore alternatives to C and C++ such as Python, Rust, Go, Clojure, Haskell, Swift, and many more, each providing different abstractions for dealing with diverse programming tasks. I wonder what an operating system for the 2020s would look like given the advances in technology and the lessons learned in the past decade.

jammygit 7y ago
He gave a good talk about SystemD once too: https://www.youtube.com/watch?v=o_AIw9bGogo https://www.youtube.com/watch?v=o_AIw9bGogo edit: found a version with better camera angles
- sprash 7y ago
  I didn't like that talk at all. I disagree with almost everything he says[1]. This talk is much better but some of his criticism are rather pointless because what he wants is a complete new OS design vastly different from UNIX. If he likes Windows/MacOS so much he should just use that. 1.: https://news.ycombinator.com/item?id=19966321 https://news.ycombinator.com/item?id=19966321

dscpls 7y ago

Awesome talk. Especially connecting how leaving an entrenched mindset behind applies both to architecture and to diversity in our communities. And that our meritocracies are not as pure as we'd like to think.

dscpls 7y ago
Lol - pointing out the elephant in the room really bugged someone
- rumanator 7y ago
  Your post has nothing to do with pointing elephants. You've just threw a bunch of baseless assertions mixed with gratuitous blanket personal attacks. Ignorance mixed with resentment is never helpful.
  - dscpls 7y ago
    Please point out my - baseless assertions - gratuitous blanket personal attacks Right now it looks like that's what you're doing, because I don't see that in my comment. Did you watch the talk to the end? Edit: line breaks
    - wizzwizz4 7y ago
      I cannot refute your comment, because it is not actually clear what you are referring to. Your comment is too vague to be useful; it barely conveys information above a sentiment. An unfalsifiable theory is useless. An unfalsifiable comment equally so.
      - dscpls 7y ago
        I'm literally repeating words and phrases from the talk.
      - lightedman 7y ago
        You just clearly demonstrated that you did NOT watch the entire talk through to the end. What OP stated is in fact present in the talk itself.

incompatible 7y ago

I suppose on Linux, libusb itself is supposed to be the general programming interface, not the poking around with /dev or ioctl. The Windows and Mac interfaces he was using are quite possibly implemented as libraries over a more primitive layer. Speculation, knowing nothing about USB specifically.

pjmlp 7y ago
On Windows and Mac, the more primitive layer are kernel private APIs, which you aren't supposed to call directly.
- magicalhippo 7y ago
  Linux is the kernel, so really the equivalent on Windows is the Windows kernel API, not the Win32 (user) API. However, the kernel API seems to be somewhat similar to the Win32 version presented in the talk and not nearly as gnarly as the Linux equivalent: https://github.com/microsoft/Windows-driver-samples/blob/master/hid/hidusbfx2/sys/usb.c https://github.com/microsoft/Windows-driver-samples/blob/mas... https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/wdfusb/ https://docs.microsoft.com/en-us/windows-hardware/drivers/dd... That said I'm just an application programmer so I've only ever used the Win32 API.

clktmr 7y ago

"Everything is a file" is one of the most misunderstood concepts I encountered, to the point where people think that disk IO is involved. A file doesn't even need a filename (eg sockets). The main point is to have a common interface for system resources, which happens to be the file api in Linux. Think of it as the base class of everything. There is a bunch of tools/code that operates on file descriptors, which you can reuse now. Build better interfaces on top of that if you want to.

discreteevent 7y ago
Everything is a bytestream.
wbl 7y ago
Sockets depart badly from that model with special purpose setup syscalls.
- clktmr 7y ago
  Yeah but that's rather because the socket api is older than Linux. Underneath there is the sockfs implementation.

soraminazuki 7y ago

Not everything may be a file indeed, but I didn't get what he was trying to point out with the specific example he gave. To make his point, he compared the USB APIs provided by Windows, macOS, and Linux. Then he went ranting about how Linux looks terrible because you had to manipulate files and use functions such as snprintf, ioctl, and fnctl. However, this is not at all a fair comparison because he was comparing the Linux kernel interface against much higher-level userspace library APIs in Windows and macOS. So what are the problems he encountered with this specific example, that had actually harmful consequences? I honestly can't tell.

pjmlp 7y ago
It was completely fair, because other OSes don't expect users to directly access kernel interfaces, most of them even forbid it, other than for writing device drivers purposes.
- clktmr 7y ago
  Linux doesn't expect you to either. You are free to choose.
- soraminazuki 7y ago
  How can it be "completely fair" when you're not comparing the same thing? Linux applications can and should use higher-level libraries too.
  - pjmlp 7y ago
    That is the low level version available to applications on Mac and Windows. The high level ones are Mac Frameworks in Objective-C/Swift and .NET/COM libraries. Direct kernel access is out of bounds.
  - blablabla123 7y ago
    On the other hand it's possible to use Syscalls on macOS or Linux to try out new low level functions, since these tend to be documented quite well unlike on Windows. For instance copy-on-write or ptrace to alter/inspect low-level calls.

swiley 7y ago

Unix at least picked one abstraction and stuck with it, contrast windows which is a dizzying mess of incompatible ideas. It’s so bad that more paranoid people may think Microsoft disorients people on purpose. There are other operating systems that also chose singular abstractions like the cannon cat and smalltalk.

teddyh 7y ago
Reference: https://en.wikipedia.org/wiki/Canon_Cat https://en.wikipedia.org/wiki/Canon_Cat

BlackLotus89 7y ago

This talk took some weird directions. It's called "What Unix Cost Us", but to be honest it isn't. Starts with USB-driver coding in Win, Mac, Linux. Goes over to colonialism. Made some good points about Computer-architecture and how they shape programming languages and then some bad/wrong points. Ends on community cultures and some controversial thoughts about gender equality and then he doesn't take any questions. Talks off a shitstorm and then isn't ready to face it. Don't know have seen better videos about the same topic.

magicalhippo 7y ago
It had some interesting points, but also some weird detours. One of his core points revolve around hanging on to the way things were, and how that often ends up poorly. Though I got this feeling that really, it's that often it's just really difficult to come up with good abstractions. Abstractions is after all what allows us to be so productive with our hardware. For example, Electron and similar is abstracting away the operating system, which makes sense because for a lot of programs the specifics of the operating system doesn't really matter. However JavaScript and HTML can hardly be described as the best way of obtaining that abstraction.

layoutIfNeeded 7y ago

I’m 29 now. I wonder if I’ll live long enough to be able to use cancellable non-blocking mkdir.

kragen 7y ago

I'm skeptical of Benno's claim that we should consider meritocracy "a dirty word" because "it's a lie". Certainly it is true that no community of practice achieves meritocracy, just as no polity achieves democracy — there are always some citizens with more influence than others, so it is always possible for the government to act against the interests and values of the majority of its population. Should we therefore consider democracy "a dirty word" because democracy "is a lie"? Perhaps it is better to consider it an ideal to which we aspire, without feigning to have achieved it. Let's consider what alternative ideals are available in place of meritocracy for governing a community of intellectual practice. We could strive for a democracy, in which the decisions are made by the majority — but the majority of whom? For this to meaningfully distinguish a community of intellectual practice from the surrounding community from which it arose, as a lotus blossom arises spotless from the swamp, a distinction must be made between voting members and outsiders. (Can you imagine a Linux User's Group where all the presentations are about Microsoft Windows, or a Python conference where all the talks are about Java?) But that is just a way of postponing the question of who the voters are. We could strive for consensus, like the Quakers, in which any collective decision is postponed until every member agrees; but, like democracy, that demands gatekeeping that draws an ingroup/outgroup distinction, so it is not really an answer to the question of who governs, just how they govern. We could strive for anarchy, in which all decisions are made individually, and there are thus no collective decisions to be made, whether by the meritorious or by anyone else. A variant of anarchy is "do-ocracy", where decisions are made by whoever shows up and makes the effort required to implement them. We could strive for a gerontocracy, in which the oldest members — perhaps by length of membership rather than by physical age — make the collective decisions. We could strive for a high-school clique, where the decisions are made by whoever is most popular. We could strive for a plutocracy, where the decisions are made by whoever is wealthiest, or who donates the most. Given these alternatives, it seems to me that when anarchy and consensus demand unacceptable tradeoffs, the least undesirable alternative is meritocracy. In meritocracy, the decisions are made by the best members of the group, according to some measure of merit that seems worthwhile to the group; in a community of intellectual practice, this usually amounts to some kind of knowledge and skill, seasoned with judgment and perhaps a guess about aptitude. If they are the wisest members, then they will make the best decisions. The greatest foolishness is to subject the wise to the government of the foolish. If some of the foolish and ignorant currently are so simply because they have not had the opportunity to learn, we can best remedy that by guiding them to learn from the wise, not by putting the foolish and ignorant in charge. Of course, meritocracy as an ideal cannot be reached, only striven for; but it is a better ideal to strive for than a high-school clique, a gerontocracy, or a plutocracy. Which of these are the campaigners against meritocracy hoping for?

gdm85 7y ago

Some parts of the talk did not really stitch together; if we consider the main arguments against Unix (and the funny dab at Plan9) I felt them a bit lacking in the sense that they do not propose alternative philosophies, but only a liberating "mixing things together is cool". I can totally understand that as a form of catharsis, and a mindset totally apt for hobby projects, but if we are playing in the realm of OS design...I would think we can do better in terms of architecture and philosophy? "anything else is better" does not really convince me, some structure is usually better than none at all.

rs23296008n1 7y ago

I think I prefer the plan9 approach of "everything is a file/filesystem" rather than the unix "everything is a file" as typically seen. Actual usage of files, filesystems and namespaces under plan9 was more consistent than the file approach under linux. It was a lot of fun playing in plan9. Regarding the video, of the three examples the windows approach was shown as more succinct. I thought the linux version was messy. The mac version seemed overly complex. Whether these were actually representative is another matter. I can't say how but I'm not convinced - there seemed to be some artistic license / suspension of belief required. The remainder seemed to be meandering but I sense you had to be there.

Aqueous 7y ago

am i the only one who thinks it’s a good thing that his Linux example was shorter than the other two and required less calling into hyper-specific API functionality?