7 ms·
Do Files want to be Actors?
- crabmusket 2y agoIs this Greenspun's tenth rule but for the BEAM? Is everything a distributed system? (Trick question! A computer has for a long time been a distributed system, we just don't like to worry about that usually.)
- toast0 2y agoI may be too far into the BEAM, but it's definitely the case that a modern computer is a distributed system at so many levels.
- LAC-Tech 2y agoThe fact I even got into this is Joe Armstrong's fault. I was listening to his talk about programming multicores and thought "you know I'm on-board with the actor model across the network, but what's the point of having multiple actors on one machine?" Which lead me down the async rabbithole, which through io_uring ends up reminding me of actors!
- throwawaymaths 2y agoyeah. but remrmber that the BEAM is more than just a message queue, its stuff like monitors and links, which are not part of the actor model and confers benefits like why you dont typically have to manually close a file in the beam, or how you can write to a (non-raw) file over the network transparently
- pjmlp 2y agoThe Network is the Computer. :)
- bobnamob 2y agoIs apparently (re)trademarked... https://blog.cloudflare.com/the-network-is-the-computer/ https://blog.cloudflare.com/the-network-is-the-computer/
- pjmlp 2y agoYep. I still have some of the Sun manuals with that sentence. It is kind of ironic how many decades we have been doing distributed systems, and now everyone talks about microservices as if rediscovered powder.
- 01HNNWZ0MV43FF 2y agoOh, file descriptors, yeah. Not files themselves, that would be weird.
- rednafi 2y agoI wonder how the stdlibs of existing languages will take advantage of io_uring and async syscalls. Does this mean we won’t have to spin up a subprocess for the most trivial things?
- LAC-Tech 2y agoThe node.js std lib has had async versions of some of their modules for a while, eg fs/promise[1] I know node.js uses epoll but I am pretty sure bun uses io_uring. Also worth mentioning Zig has a realy great iouring library[2] right in the stdlib, which without looking up the source code I am guessing is what bun uses under the hood: [1] https://nodejs.org/api/fs.html#promises-api https://nodejs.org/api/fs.html#promises-api [2] https://ziglang.org/documentation/0.13.0/std/#std.os.linux.IoUring https://ziglang.org/documentation/0.13.0/std/#std.os.linux.I... - (give it a minute to load, the zig people are great system programmers but webdevs they are not lol)
- rednafi 2y agoWow, the Zig lib is as close to raw io_uring as it gets. It directly accepts an fd, a submission, and a result queue.
- nicoburns 2y agoNode.js has had async version of their modules from the very first version (or at least, long before it got popular). The "/promises" modules just wrap that in promises so you can use it with a nicer interface and async/await syntax (the original versions were callback based).
- accrual 2y agoMaybe it's a bad analogy, but this reminded me of how the "actors" in Sims 1 were implemented. Rather than having a massive Sim class that knows how to interact with every object, the Sims ask each object what it can do, and it's up to the object (kernel) to handle the implementation details (IO in this case), do its work, and return later when it's done.
- manmal 2y agoIsn’t that just OOO? Or was there actual concurrency involved?
- dsp_person 2y agoWouldn't we only have 2 actors, the kernel and our program / event loop? But "actors" plural makes it sound like a system of many independent nodes "sends new messages to other actors and computes a new local state for itself."
- ilaksh 2y agoThere are a lot of ways to skin a cat. I doubt that most of how Linux works is really ideal at this point. (Which is not to say it's easy to replace or impractical). But that structure/API is very similar to many similar patterns in computing. Look at Smalltalk or OOP to some degree. Maybe extend it with operational transforms. I think there are a lot of interesting alternative ways to look at operating systems and many developments over the years. Such as Plan 9, MirageOS, and several other projects. Also just to clarify I love cats and need a better metaphor.
- deepsun 2y agoThis reminds me of modern mathematics -- a lot of interesting and useful stuff comes from looking at old concepts from another viewpoint (different way to skin a cat). E.g. "what if we look at our stuffs in hand as elements of group XXX -- we get the whole group theory applicable now" etc. The trick is to note similarities. So now we can study Actor problems and apply it to io_uring to avoid pitfalls earlier.
- BJones12 2y agoThere's more than one way to skin a carrot
- deleted 2y ago[deleted]
- deepsun 2y agoThe main benefit with Actors for me was that each Actor is single-threaded and stateful. That simplifies a lot (but moves the complexity to cross-actor communication). Not sure each file IO and state is guaranteed to be single-threaded.
- RossBencina 2y agoAnother characteristic of Actors is that Actors can spawn other Actors. Also, "the mail system" (queues, message dispatch) is not part of the definition of the Actor Model. It's an implementation detail.
- louthy 2y ago> Another characteristic of Actors is that Actors can spawn other Actors. But it's entirely up to the actor if it wants to or not. I could imagine an actor supervision-hierarchy with directory-actors that have child file-actors and other child directory-actors. The file-actors could just be leaves in the hierarchy. Each write operation on a file would be guaranteed to be single threaded via the file-actor. But a file-actor could also launch ephemeral child-actors that do read-only processing on a snapshot of the file. So, parallel processing would be possible for read-only operations. A file-system does fit quite elegantly into the actor model imho. Whether it would be efficient or not, who knows, but at least on the surface it fits.
- RossBencina 2y agoFor what it's worth, I have developed an actor-based async file i/o framework that does multiplexing, caching and format translation. Files are actors and pages (which serve fixed size blocks out of a particular file) are actors. The main motivation was to support caching and prioritised i/o. It doesn't target io_uring yet though.
- gigatexal 2y agoSounds interesting. When can you share it with us on HN?
- RossBencina 2y agoI'm not really sure what the point of this post is. As soon as you introduce asynchronous queueing of operation descriptors and completions the analogy to Actors is present. IO completion ports are not a new concept in operating systems, nor is inter-process messaging. Hell classic MacOS has async IO command objects, with an API to queue them, most likely you're going to put them in a queue when you get the completion callback too.
- Guthur 2y agoInteresting bit of synchronicity. I was recently looking into Erlang in the context of large scale data distribution and was wondering if instead of everything is a file everything is a process would be more appropriate. The rationale was that it would better aggregate behaviour and lend itself to more reasonable scaling methodologies.
- kragen 2y agoAsynchronous I/O completion notification was a huge innovation, allowing computation to proceed concurrently with I/O and thus dramatically increasing the throughput of the computer. Unfortunately it was also a huge source of bugs. The following innovation that made it usable, by making it less bug-prone, was called a "multitasking operating system". The so-called "OS" allowed you to write simple sequential code, but used the computer efficiently by switching back and forth between multiple tasks as their respective I/Os completed. We're talking about the introduction of the Univac 1103A in 01953, 72 years ago, and the following 20 years of innovations, including things like Dijkstra's THE operating system. That is, asynchronous I/O is 20 years older than the Unix system call interface this article speculates it should replace. Of course, context switching between different tasks is not free, and event loops have frequently been able to provide higher efficiency. The equilibrium has rocked back and forth as I/O has gotten faster and slower relative to task context switching. CICS, select(), poll(), Oberon, the Macintosh system, Win16, the JavaScript event loop, Tcl/Tk, Win32 IOCP, Symbian active objects, kqueue, epoll, and io_uring are some of the results. But don't try to sell asynchronous I/O as a "game-changing" paradigm shift. It's a different programming model that's harder to program but can provide higher performance, just like it has been for 70 years. If you're shopping for a paradigm shift that can improve this tradeoff, there are several candidates. Erlang-style lightweight processes, software transactional memory, and JS-style promises (originally from E, which was inspired by KeyKOS, which had event loops but no promises or asynchronous I/O) come to mind. The hardware development that might be actually new has arguably already failed in the market: Intel/Micron's "Optane" memory and Flash-based NVDIMMs. Flash has big disk energy, but is fast enough that copying it to RAM one word at a time like disk will probably bottleneck your performance by an order of magnitude. io_uring doesn't fix this. We need interfaces designed for zero-copy access to bulk persistent data. Maybe something like Multics or mmap(). LMDB and FlatBuffers suggest that the potential for improved performance is significant. Could such high-bandwidth, low-cost memory keep up with LLM inference, allowing you to do inference on a 256-gigabyte model with 256 gigabytes of Flash but a much smaller amount of RAM?
- baruch 2y agoAsynchronous IO with user space threads works wonders to get both the performance of async IO and the convenience of sequential programming.
- netbsdusers 2y ago>"Our operating systems want to do things asynchronously, on their own terms. They'll tell you when they're done. Maybe this is a new era. Maybe making syscalls from 1970s Unix directly is like a remote procedure call to another machine - a leaky abstraction, a feeble attempt to impose your old mental model onto a new reality." I think the article fell off here. Linux has a profoundly synchronous I/O system. The I/O U-Ring works there by dispatching requests to a kernel-maintained pool of worker threads. There the submitted request are run synchronously. So the I/O U-Ring is in fact the abstraction, "a feeble attempt to impose your [new] mental model onto an [old] reality", as the author might put it. The actual Linux I/O system is most trivially exposed to userland by the same old "1970s Unix" means of the traditional system calls. The real change with the I/O U-Ring is that it offers a way to submit work in bulk and finally introduces to Linux a form of I/O that appears to the program to be truly asynchronous. (And it's not without some fair cause that file I/O is implemented as synchronous code. It's hard to do it asynchronously when it's a complex operation with many steps, where you are interacting with a page cache, not only for file contents but even caching the very metadata that describes how to get from an offset into a file to the block numbers where the data is stored. Windows has a famously profoundly asynchronous I/O system and even there actual file I/O is done just the same with synchronous logic ran in a worker thread, or copied directly from the cache if the requested data is already there.)
- anarazel 2y agoFwiw, a good portion of IO via io_uring is not executed via the threaded work queue anymore. Even with buffered file reads it's avoided for some common filesystems. You're right of course that there are lots of cases (missing metadata, synchronous operation like extending files, ...) where it's all offloaded to the wq.
- LAC-Tech 2y agoI suppose by "new reality" I meant that asynchronicity seems to be happening from both above (programming lagnauges) and below (hardware and I suppose..physics), and then in the middle we still have units of execution pausing on syscalls. I get that io_uring is an abstraction built on top of classic unix syscalls, but I still find it interesting that even linux people are starting need an async model at the OS level.
- davexunit 2y agoWrapping file i/o in actors is exactly what we do in the Goblins distributed programming system. It ensures that only one operation is happening against the file descriptor at any given time while allowing other actors to use it asynchronously.
- chubot 2y agoI think what's missing is that this an extremely low level programming model: Upon receipt of this message in the event E, the target consults its script (the actor analogue of program text), and using its current local state and the message as parameters, sends new messages to other actors and computes a new local state for itself. It doesn't say anything about whether you do: - nginx-style state machines in C - callbacks in C++, or C++ 20 coroutines - async/await in Rust - Goroutines in Go - async/await in Python or JS, with garbage collection etc. I don't think the "actor model" really means that much these days. What's a "canonical" and successful actor model program? What can we learn from such programs? I think if you ask 5 people you'll get 5 different answers. --- Also, with __u8 opcode; /* type of operation for this sqe */ __s32 fd; /* file descriptor to do IO on */ then you have lost all static typing. It is too low level, so the analogy doesn't really hold up IMO. Also, I don't understand why it's "do files want to be actors?", not "do Unix PROCESSES want to be actors?" (copy of lobste.rs comment)
- worik 2y ago> then you have lost all static typing Do you need it at this level? At some point everything is a bit-field. We impose typing to aid our mental models and build useful abstractions. When interacting with the kernel we can let go, then reclaim, our types
- pjmlp 2y agoWhich is why when you have programming languages with rich runtimes and ecosystem, the OS kind of becomes irrelevant. "an operating system is a collection of things that don't fit inside a language; there shouldn't be one" -- Dan Ingalls So what happens is that those runtimes built on top of whatever low level primitives are available, and that is about it. Even considering UNIX alone, many ways to do asynchronous IO aren't even part of POSIX, it has remained specific to each UNIX flavour. To some extent, UNIX/POSIX API surface has been the C and C++ standard library that WG14 and WG21 didn't want to take over into ISO, but almost every C and C++ developer expects to exist anyway.
- PaulDavisThe1st 2y ago> Is it just me, or are these two seemingly unrelated schools of computing converging on the exact same idea? You send messages to some target (ie, the file descriptor). Seems to me that it's a matter of perspective, specifically about who you're sending messages to. You can consider the message as being sent to the file (descriptor). You can also consider it a message sent to the kernel, in which case the kernel is an actor and the file is a passive data abstraction. Both are accurate and useful; which one makes more sense will depend on context and your own background.
- btown 2y agoI do think there's something to be said for files being seen not as things that can be "owned" and "manipulated" and "locked" by a main process, but rather as their own independent services, in a unified registry, who can only return results asynchronously, and only over an external API - a la the Amazon Memo (https://chrislaing.net/blog/the-memo/ https://chrislaing.net/blog/the-memo/). Whether there's an API broker (in this case, the kernel) is just an implementation detail of the "network" layer.
- pjmlp 2y agoThis change of mentality is similar to how monolithic kernels get praised, only to have daemons, microservices and containers all over the place, using a much slower OS IPC communication channel than if it had been a microkernel from the start.
- manmal 2y agoTo me, it’s not a change of mentality, but a change of POV on how to think about concurrency. If everything is an actor, and you recognize it early on, you can use tech (elephant in the room is Erlang/Elixir) that might fare better overall. There’s no silver bullet ofc.
- pjmlp 2y agoOne thing common to that tech, including stuff on JVM and .NET, is that then you are at a point where UNIX/POSIX becomes irrelevant, basically my distributed systems story, since the early days of Java and .NET. Naturally considering Erlang, and any other language with rich runtime and ecosystem as well.
- remram 2y agoLike everything else, the sweet spot is probably in between.
- LAC-Tech 2y agoYes you're actually not the first person to mention microkernels to me, when I've ranted about this stuff. How do they do syscalls? Is every one basically an IPC?
- pjmlp 2y agoYes, but a special kind of IPC, where ownership of message block is exchanged. QNX and se4L are two of the fastest ones, as general purposes OSes. Then you have embedded ones for high integrity computing like INTEGRITY RTOS. Here some info, https://swd.de/Support/Documents/Manuals/Neutrino-Microkernel-System-Architecture/Chapter-2-The-QNX-Neutrino-Microkernel-Part-2/ https://swd.de/Support/Documents/Manuals/Neutrino-Microkerne... https://docs.sel4.systems/Tutorials/#sel4-mechanisms-tutorials https://docs.sel4.systems/Tutorials/#sel4-mechanisms-tutoria... https://www.researchgate.net/publication/386549964_An_Overview_of_Operating_Systems_Based_onMicrokernel_Technology_and_their_EssentialComponents https://www.researchgate.net/publication/386549964_An_Overvi...
- btbuildem 2y agoI've said this before, and I'll say it again: On a long enough timescale, everything converges to Erlang.
- jiehong 2y agoAnybody knows if MacOS / iOS uses something similar to io_uring?
- p_ing 2y agoIt does not. https://en.wikipedia.org/wiki/Grand_Central_Dispatch https://en.wikipedia.org/wiki/Grand_Central_Dispatch (which uses pthreads)
- pjmlp 2y agoNot really, it depends on the OS X version, and the one used by Swift has multiple implementations. Also it isn't regular pthreads on OS X, rather Apple own's flavour. Described on the Internals section.
- Quekid5 2y agoNo. The only thing needed is Capabilities and we're done.
- erichocean 2y agoFiles actually want to be microservices.