4 ms·

Because it's not obvious without glancing at the code, this relies on 9P-over-virtio to implement the filesystem, which means its pretty much always going to be

by _wmd 12y ago

Because it's not obvious without glancing at the code, this relies on 9P-over-virtio to implement the filesystem, which means its pretty much always going to be limited to running Linux images. This sounds like a nice and clean solution, but at least it forces the guest to do basically no useful caching of its file system, in order to remain coherent with the host fs. The remaining alternative is on-demand synthesis of a block image given a directory tree, which IIRC Qemu supports. As always, making a couple of syscalls isn't a huge amount of work, and this project punts on anything difficult (e.g. supporting traditional boot environment & emulated devices) needed for handling basically any other OS. It can't even boot a standard vmlinuz. Not sure what gap this is supposed to fill. If you're running a Linux host, and you want ultra-quick dev machine booted from your filesystem, User Mode Linux already exists and has way more support and flexibility. If you need support, performance, and code that's been security audited, you probably want Qemu/Xen/libvirt edit: just because it may not be clear to a lot of people.. Literally KVM setup is a handful of ioctl() system calls, almost everything non-device-emulation related is handled by Linux for you. So when you say you're implementing a KVM-based hypervisor that doesn't implement any device emulation, that is to say there's very little real work being done. It's easy to claim to be fast when your code does nothing. The only special sauce left is a fundamentally slow method for handling the root filesystem which is worth avoiding for the reasons I mentioned.

peterwaller 12y ago
Is it interesting if only because it's written in a "memory-safe"* language? * Terms and conditions apply.
derefr 12y ago
One could write a Windows IFS driver for 9P if that had some benefit. As it is, there's a Windows IFS FUSE driver that has a 9P server written for it.
- _wmd 12y ago
  You could, but you almost certainly don't want to.. Network filesystems that try to emulate regular filesystem semantics fundamentally suck, not least because you basically can't do any useful caching, and the caching that is done (because it must be done) can introduce very hard to detect bugs (e.g. coherence between a file and its in-memory mapping. If you want to execute a program from the filesystem you need such a mapping).
- 4ad 12y ago
  Ignoring details like the impossibility of loading the kernel (because the Windows kernel requires to be loaded by its own bootloader, which does a lot of setup), you will not be able to boot from 9P.
philip1209 12y ago
Could this in any way explain why Google Cloud does not support Ubuntu?
- deleted 12y ago
  [deleted]
- morpher 12y ago
  GCE added Ubuntu support recently:https://cloud.google.com/compute/docs/operating-systems/linux-os#ubuntu https://cloud.google.com/compute/docs/operating-systems/linu...
onedognight 12y ago
> it forces the guest to do basically no useful caching of its file system Of a filesystem was the impression I got. The author implied that you could, in addition, mount any block device provided to you in any way you wanted. > It can't even boot a standard vmlinuz. It was mentioned at the end that while it won't run the real-mode code therein, it will attempt extract the elf binary and run that.
shillster 12y ago
If you take a glance at Paragraph 1 of the Readme (its the main description for the project), it says, "Its goal is to provide an alternate, high-performance Linux hypervisor for cloud workloads." So you can probably answer your own question about the intentions of this project.
JoshTriplett 12y ago
9p works from OSes other than Linux; there's a driver for Windows as well. That said, it will likely always work best in Linux. Unfortunately, even over virtio, 9p has awful performance. > The remaining alternative is on-demand synthesis of a block image given a directory tree, which IIRC Qemu supports. If you mean vvfat, that hasn't worked in quite a while, and it never supported writes. > If you're running a Linux host, and you want ultra-quick dev machine booted from your filesystem, User Mode Linux already exists and has way more support and flexibility. UML tends to bitrot; it's hit or miss whether it even compiles in each new release, and it doesn't get any significant new development. It also relies on ptracing the target process, which is rather awful; unfortunately there's no better way to implement userspace syscalls. I wonder if UML could be accelerated by adding some new features to the BPF-based seccomp to support syscall emulation? BPF could translate each syscall into some efficient communication to the UML kernel process. Alternatively, with a 64-bit address space available, UML could learn to do its own memory and process management, and not rely on host Linux processes at all.
- _wmd 12y ago
  I imagine seccomp and ptrace have pretty similar performance, since they both wake the 'host' using a signal, and stay out of the way until their trigger is met. As for UML stability, Debian have been providing packages of it for years, I can't remember the last time I built it directly.
- kijiki 12y ago
  Could you elaborate on 9p/virtio performance? I've had pretty good results booting qemu/kvm VMs with 9p root filesystems... UML used to have some host kernel patches that were a lot faster for intercepting syscalls. There was never enough interest (even before Vanderpool/Pacifica) to merge it though.
  - JoshTriplett 12y ago
    > Could you elaborate on 9p/virtio performance? I've had pretty good results booting qemu/kvm VMs with 9p root filesystems... The throughput is passable, if you do large enough reads; however, the latency for individual operations is terrible. Try booting via virtio-9p and building a kernel inside the virtual machine, as compared to the host system. Last time I tried it, it took several orders of magnitude longer, due in large part to the latency of simple operations like stat or open/read/close.
voidlogic 12y ago
>but at least it forces the guest to do basically no useful caching of its file system, in order to remain coherent with the host fs. Sure, but I assume the host/hypervisor is doing the caching on its side, so this may not be a huge loss for the gain.
amscanne 12y ago
There are several emulated devices simply because there's no way around it (the rtc, PCI bus, uart, etc.). Obviously KVM handles a fair amount of emulation (PIT, IOAPIC) which is very nice. Although the Linux implementation for 9p isn't great, there's nothing "fundamental" that limits the caching in the guest. For many use cases, you will have no shared mutable state in the host. Think docker: you have some shared read-only state, but the write bits are yours alone. You may cache anything freely. This project was simply about playing with these ideas (and implementing a VMM using Go). You should check the slides out to understand the gap that I was looking at, but suffice to say I think there's a wide opportunity for something that gives you a process-like model but whose interface is that of a virtual machine. The key point is that you don't really want an ultra-quick "machine", you probably want to run a process ala Docker. I still want it to be contained as a VM however, for security and compatibility reasons.

by _wmd 12y ago

peterwaller 12y ago

Is it interesting if only because it's written in a "memory-safe"* language? * Terms and conditions apply.

derefr 12y ago

One could write a Windows IFS driver for 9P if that had some benefit. As it is, there's a Windows IFS FUSE driver that has a 9P server written for it.

_wmd 12y ago
You could, but you almost certainly don't want to.. Network filesystems that try to emulate regular filesystem semantics fundamentally suck, not least because you basically can't do any useful caching, and the caching that is done (because it must be done) can introduce very hard to detect bugs (e.g. coherence between a file and its in-memory mapping. If you want to execute a program from the filesystem you need such a mapping).
4ad 12y ago
Ignoring details like the impossibility of loading the kernel (because the Windows kernel requires to be loaded by its own bootloader, which does a lot of setup), you will not be able to boot from 9P.

philip1209 12y ago

Could this in any way explain why Google Cloud does not support Ubuntu?

deleted 12y ago
[deleted]
morpher 12y ago
GCE added Ubuntu support recently:https://cloud.google.com/compute/docs/operating-systems/linux-os#ubuntu https://cloud.google.com/compute/docs/operating-systems/linu...

onedognight 12y ago

> it forces the guest to do basically no useful caching of its file system Of a filesystem was the impression I got. The author implied that you could, in addition, mount any block device provided to you in any way you wanted. > It can't even boot a standard vmlinuz. It was mentioned at the end that while it won't run the real-mode code therein, it will attempt extract the elf binary and run that.

shillster 12y ago

If you take a glance at Paragraph 1 of the Readme (its the main description for the project), it says, "Its goal is to provide an alternate, high-performance Linux hypervisor for cloud workloads." So you can probably answer your own question about the intentions of this project.

JoshTriplett 12y ago

9p works from OSes other than Linux; there's a driver for Windows as well. That said, it will likely always work best in Linux. Unfortunately, even over virtio, 9p has awful performance. > The remaining alternative is on-demand synthesis of a block image given a directory tree, which IIRC Qemu supports. If you mean vvfat, that hasn't worked in quite a while, and it never supported writes. > If you're running a Linux host, and you want ultra-quick dev machine booted from your filesystem, User Mode Linux already exists and has way more support and flexibility. UML tends to bitrot; it's hit or miss whether it even compiles in each new release, and it doesn't get any significant new development. It also relies on ptracing the target process, which is rather awful; unfortunately there's no better way to implement userspace syscalls. I wonder if UML could be accelerated by adding some new features to the BPF-based seccomp to support syscall emulation? BPF could translate each syscall into some efficient communication to the UML kernel process. Alternatively, with a 64-bit address space available, UML could learn to do its own memory and process management, and not rely on host Linux processes at all.

_wmd 12y ago
I imagine seccomp and ptrace have pretty similar performance, since they both wake the 'host' using a signal, and stay out of the way until their trigger is met. As for UML stability, Debian have been providing packages of it for years, I can't remember the last time I built it directly.
kijiki 12y ago
Could you elaborate on 9p/virtio performance? I've had pretty good results booting qemu/kvm VMs with 9p root filesystems... UML used to have some host kernel patches that were a lot faster for intercepting syscalls. There was never enough interest (even before Vanderpool/Pacifica) to merge it though.
- JoshTriplett 12y ago
  > Could you elaborate on 9p/virtio performance? I've had pretty good results booting qemu/kvm VMs with 9p root filesystems... The throughput is passable, if you do large enough reads; however, the latency for individual operations is terrible. Try booting via virtio-9p and building a kernel inside the virtual machine, as compared to the host system. Last time I tried it, it took several orders of magnitude longer, due in large part to the latency of simple operations like stat or open/read/close.

voidlogic 12y ago

>but at least it forces the guest to do basically no useful caching of its file system, in order to remain coherent with the host fs. Sure, but I assume the host/hypervisor is doing the caching on its side, so this may not be a huge loss for the gain.

amscanne 12y ago

There are several emulated devices simply because there's no way around it (the rtc, PCI bus, uart, etc.). Obviously KVM handles a fair amount of emulation (PIT, IOAPIC) which is very nice. Although the Linux implementation for 9p isn't great, there's nothing "fundamental" that limits the caching in the guest. For many use cases, you will have no shared mutable state in the host. Think docker: you have some shared read-only state, but the write bits are yours alone. You may cache anything freely. This project was simply about playing with these ideas (and implementing a VMM using Go). You should check the slides out to understand the gap that I was looking at, but suffice to say I think there's a wide opportunity for something that gives you a process-like model but whose interface is that of a virtual machine. The key point is that you don't really want an ultra-quick "machine", you probably want to run a process ala Docker. I still want it to be contained as a VM however, for security and compatibility reasons.