3 ms·
Keep in mind that (conventional) micro-kernels are not the same as exokernels. FUSE is fun, I've written my own filesystems with it, but it's basically a micro
by eru 4y ago
Keep in mind that (conventional) micro-kernels are not the same as exokernels.
FUSE is fun, I've written my own filesystems with it, but it's basically a micro-kernel idea, not an exokernel one. (L4 is also great! But I don't think it qualifies as an exokernel?)
https://pdos.csail.mit.edu/6.828/2019/lec/faq-exokernel.txt https://pdos.csail.mit.edu/6.828/2019/lec/faq-exokernel.txt explains a lot about exokernels that's not mentioned in the papers.
Exokernels never caught on, at least not under that name. The closest equivalent in widespread use today are actually hypervisors for running virtual machines. (Especially if you are running a so called 'unikernel' on them.)
About filesystems: if you just want the kinds of abstractions that conventional filesystems already give you, you won't get too much out of using an exokernel. (As you mention, perhaps you can get a bit of extra performance?) From the FAQ I linked above:
> Q: In what kind of applications is an exokernel operating system preferable?
There are naturally tradeoffs with the extra flexibility provided e.g. it is
easier to make a mistake in user code.
> A: An exokernel is most attractive to an application that needs to do
something that is possible with an exokernel, but not possible with
other kernels. The main area in which the 1995 exokernel paper increased
flexibility was virtual memory. It turns out there are a bunch of neat
techniques applications can use if they have low-level access to virtual
memory mappings; the Appel and Li paper (citation [5]) discusses some of
them. Examples include distributed shared memory and certain garbage
collection tricks. Many operating systems in 1995 didn't give enough
low-level access to virtual memory mappings to implement such
techniques, but the exokernel did. The exokernel authors wrote a later
paper (in SOSP 1997) that describes some examples in much more depth,
including a web server that uses a customized file system layout to
provide very high performance.
The HN submission we are nominally discussing here is also about memory, so that might be applicable.
An example for filesystems I could envision: direct low-level hardware access to an SSD's internals for a database. Databases don't really care about files, and might also want to deal with SSD's peculiar writing processes in a way that's different from the abstractions typical file systems give you.
> Perhaps this is stating the obvious, but I don't think you can get much fault isolation with a pure library filesystem; if all the processes participating in the filesystem are faulty then there's no way to protect the filesystem from fatal corruption from faults. You might be able to reduce the presumed-correct filesystem core process to something like a simplified Kafka: a process that grants other processes read-only access to an append-only log and accepts properly delimited and identified blocks of data from them to append to it.
That might be possible, but wouldn't really be faster than letting a kernel handle it, I'd guess? (But it would perhaps be more flexible to develop, since it's all userland.) You can also take inspiration from how eBPF allows you to upload user level logic into the Linux kernel and run them securely. Instead of uploading them into the kernel, you could also upload them into your filesystem service, I guess?
Some of the original exokernel papers had some more interesting ideas sketched out.
> I know you have a very substantial amount of experience with this as a result of Varnish and your involvement with Fastly. What do you think?
I'm afraid you are mixing me up with someone else?
- SSLy 4y ago>I'm afraid you are mixing me up with someone else? To clear this up – they were addressing phk, their parent comment.
- kragen 4y agoI agree that L4 is not an exokernel, though it does go a little further in the exokernel direction than conventional microkernels. I agree that FUSE is microkernelish rather than exokernelish, though there's nothing in the exokernel concept as I understand it that excludes the possibility of having servers for things like some or all of your filesystem functionality. Databases are indeed an application that commonly suffers from having to run on top of a filesystem. > That might be possible, but wouldn't really be faster than letting a kernel handle it, I'd guess? I think reading files by invoking library calls that follow pointers around a memory-mapped filesystem might well be faster than reading files by repeatedly context-switching back and forth into even a supervisor-mode kernel, much less IPC rendezvous via a kernel with a filesystem server. This is particularly true in the brave new SSD world where context switch time is comparable to block device I/O latency, rather than being orders of magnitude smaller. Writes to Kafka are very cheap and support extreme fan-in because the Kafka design pushes almost all the work out to the clients; the Kafka server does very little more than appending chunks of bytes, containing potentially many separate operations, to a log. It seems very plausible to me that this could be faster than handling a series of individual filesystem operations (whether in a kernel or in a microkernel-style server), at least for some applications; particularly with orders of magnitude lower penalties for nonlocality of reference than for traditional filesystems, and for applications where many writes are never read. Running logic in the kernel or in a server using a restrictive interpreter is indeed an interesting architectural possibility, but from a certain point of view it's the opposite extreme from the Kafka approach. > > I know you have a very substantial amount of experience with this as a result of Varnish and your involvement with Fastly. What do you think? > I'm afraid you are mixing me up with someone else? I hope this isn't rude, but I wrote that in response to phk's comment, so I was addressing him in it, not you, eru, although I did enjoy your comment very much as well.
- eru 4y ago> Running logic in the kernel or in a server using a restrictive interpreter is indeed an interesting architectural possibility, but from a certain point of view it's the opposite extreme from the Kafka approach. In general, a restricted language. You interpret or compile that language, and still have similar security guarantees. > I hope this isn't rude, but I wrote that in response to phk's comment, so I was addressing him in it, not you, eru, although I did enjoy your comment very much as well. Oh, that's fine. I was just confused because that came in a reply to my comment.