4 ms·

The reason why metal doesn't exist now is because you can't turn the memory protection stuff off in modern CPU's. For some weird reason (I'm not an OS/CPU deve

by tinco 13y ago

The reason why metal doesn't exist now is because you can't turn the memory protection stuff off in modern CPU's. For some weird reason (I'm not an OS/CPU developer) switching to long mode on an x86 cpu also turns on the mmu stuff. You just can't have one without the other. There's a whole bunch of research done on VM software managed operating systems, back when the VM's started becoming really good. Microsoft's Singularity OS was the hippest I think.[0] Perhaps that ARM cpu's don't have this restriction, and we will benefit from ARM's upmarch sometime? [0] http://research.microsoft.com/en-us/projects/singularity/ http://research.microsoft.com/en-us/projects/singularity/

gary_bernhardt 13y ago
I didn't want to go into this level of detail in the talk, but... I think you still want the MMU enabled, just not used for process isolation. With virtual memory totally disabled, a 1 GB malloc takes 1 GB physical memory even if it's not touched, you can't have swap at all, memory fragmentation kills you dead, etc. It still has a lot of utility outside of isolation. I don't have a good sense of how the performance cost of hardware isolation breaks down into {virtual memory enabled,TLB thrashing,protection ring switching}. That's one of the reasons that I reduced the speed-up from "25-33%" in the MSR paper down to 20% in METAL. Maybe the speed-up would be less than that if virtual memory were still enabled. Unfortunately, that distinction may have been blurred in the talk. That is, I may have implied that METAL would turn the MMU off entirely. If so, it was an oversight. I've done the talk end-to-end at least fifty times, which is how I smooth my execution out. Occasionally it can "smooth" the ideas out a bit too, leading to small inaccuracies. It's sort of like playing the telephone game with yourself (which is a very strange experience). The MSR paper that I quote came from the Singularity team, so your reference is right on. Reading "Deconstructing Process Isolation" in fall of 2012 was probably the germ of the core narrative of the talk.
- nteon 13y ago
  On linux, system calls don't result in a TLB flush - kernel data structures and code are in a different portion of the virtual address space (starting from the top of VM memory, if I remember right) that is tagged as not being available from ring 3. So system calls are quite fast. EDIT: Kernel memory begins at PAGE_OFFSET, see here: https://www.kernel.org/doc/gorman/html/understand/understand007.html https://www.kernel.org/doc/gorman/html/understand/understand... Kernel memory lacks the flag _PAGE_USER so that it isn't accessible from userspace: https://www.kernel.org/doc/gorman/html/understand/understand006.html#toc19 https://www.kernel.org/doc/gorman/html/understand/understand...
  - gary_bernhardt 13y ago
    I didn't know that! It certainly makes sense. Context switches still thrash the TLB, though. The performance cost of that has gotten better as time has gone on, but I wonder how many transistors (and how much power) CPUs are burning for that mitigation. The "how computers actually work" digression originally had a section on context switches, but I removed it early on because I felt like that section was dragging. To try to paint a very rough picture of the larger thoughts from which this talk was taken: I think that microkernels and the actor model are both the right thing (most of the time). When implemented naively, they both happen to take a big penalty from context switch cost. But Erlang can host a million processes in its VM, and we're using VMs for almost everything now anyway. The obvious (to me) solution is to move both the VM and an Erlang-style, single-address-space scheduler into the kernel. Then you can have a microkernel and a million native processes without the huge overhead of naive implementations. There are surely many huge practical hurdles to overcome with that, and maybe some that can't be overcome at all, but it sure sounds right when written in two paragraphs. ;)
    - deleted 13y ago
      [deleted]
    - nly 13y ago
      What you seem to be missing with re: asm.js is that, while the JIT to native code gets you your super-fast integer operations, it's still critically incomplete with regard to memory access. Every single individual memory access has to be bounds checked or pushed through some other indirection inside the runtime. Google demonstrated similar ideas with NaCl, which achieved safety with a similarly restricted native code and a just-in-time verification step. Even if these memory accesses could be made as efficient as those performed by the CPUs access protection, you're still not gaining anything you don't already have. Regarding context switches: A full CPU context switch on x86 (not to ring1 but between two arbitrary points within a single userland address space) takes a few dozen instructions and about 40-80 cycles. A single cache-line miss resulting in a load from main memory on the other hand takes at least twice that (~200 cycles). Again, hits from jumping around in memory will dominate. How significant is a 20% overhead from virtual memory? Probably about the same as getting 1% more of your memory accesses back in to high level caches.
- microcolonel 13y ago
  There's a new CPU architecture in the pipe(should have good silicon within five years, if their projections can be trusted), and it has a very good design around system calls, and also in terms of virtual memory. It has a single 64-bit address space and only protection contexts, and due to the general design of the system it doesn't require any register push(or registers at all, at least in the traditional sense). In addition, it has primitives which would allow programs to call directly into drivers and kernel services without a context switch. Anyway, I don't mean to sound like an advertisement, and we've yet to see any silicon, so the jury's out. Aside: Starting process address spaces at 0 is not really a convenience as far as I know(other than offering consistent addresses for jumping to static symbols), it's a way to enable PAE on 32-bit machines so that single contexts(typically processes) can use the whole address space.
  - wolf550e 13y ago
    You forgot to provide a link: http://millcomputing.com/docs/ http://millcomputing.com/docs/
  - sparkie 13y ago
    There's an even bigger revolution in CPU design, with Ivan Sutherland's Fleet - which does away with the clock and sequential execution of instructions - instead, the programming model is based on messaging and traffic control - you direct signals to the units in the chip which perform the computations you want, asynchronously by default - if you need synchronicity, you need to program your own models for it. While these probably won't be available in the next 5 years, and probably won't be acknowledged by existing programmers for decades - I think these ideas will take over. http://arc.cecs.pdx.edu/publications http://arc.cecs.pdx.edu/publications https://www.youtube.com/watch?v=jR9pAaQlVRc https://www.youtube.com/watch?v=jR9pAaQlVRc
- tinco 13y ago
  After the Operating Systems and Computer Organization courses in my first year of universisty I became a little obsessed with the idea of software managed operating systems. Cool to hear Singularity inspired you as well. Now I didn't read the paper, but I think the 20% is purely the MMU, I think the protection ring switching thing is much less significant, so I think if you leave the MMU still on that 20% profit is still very optimistic. Now, if you forget about compiling C (which defeats the purpose of your talk) and just compile managed languages like regular JS, the garbage collector can build a great model of memory usage. Therefore I think it could be much better to let the garbage collector manage both the isolation, and the swapping. So everything in software. The swapping process would suffer some performance, but that's just CPU cycles, as everyone knows persistent data access isn't even in the same league as CPU memory access. So yeah, that would mean that you would have to run all untrusted code in managed mode. And with untrusted I would mean code you can't trust with full physical memory access.
pjmlp 13y ago
In a way that is no different from the older Xerox PARC systems or the Oberon based ones at ETHZ. All of them are based on the concept of using memory safe languages for coding while leaving the runtime the OS role. Except for C and C++, language standard libraries tend to be very rich and to certain extent also offer the same features one would expect from OS services. As such, bypassing what we know as standard OS with direct hardware integration, coupled with language memory safety, could be an interesting design as well. That is why I follow the Mirage, Erlang on Xen and HaLVM research.

by tinco 13y ago

gary_bernhardt 13y ago

I didn't want to go into this level of detail in the talk, but... I think you still want the MMU enabled, just not used for process isolation. With virtual memory totally disabled, a 1 GB malloc takes 1 GB physical memory even if it's not touched, you can't have swap at all, memory fragmentation kills you dead, etc. It still has a lot of utility outside of isolation. I don't have a good sense of how the performance cost of hardware isolation breaks down into {virtual memory enabled,TLB thrashing,protection ring switching}. That's one of the reasons that I reduced the speed-up from "25-33%" in the MSR paper down to 20% in METAL. Maybe the speed-up would be less than that if virtual memory were still enabled. Unfortunately, that distinction may have been blurred in the talk. That is, I may have implied that METAL would turn the MMU off entirely. If so, it was an oversight. I've done the talk end-to-end at least fifty times, which is how I smooth my execution out. Occasionally it can "smooth" the ideas out a bit too, leading to small inaccuracies. It's sort of like playing the telephone game with yourself (which is a very strange experience). The MSR paper that I quote came from the Singularity team, so your reference is right on. Reading "Deconstructing Process Isolation" in fall of 2012 was probably the germ of the core narrative of the talk.

nteon 13y ago
On linux, system calls don't result in a TLB flush - kernel data structures and code are in a different portion of the virtual address space (starting from the top of VM memory, if I remember right) that is tagged as not being available from ring 3. So system calls are quite fast. EDIT: Kernel memory begins at PAGE_OFFSET, see here: https://www.kernel.org/doc/gorman/html/understand/understand007.html https://www.kernel.org/doc/gorman/html/understand/understand... Kernel memory lacks the flag _PAGE_USER so that it isn't accessible from userspace: https://www.kernel.org/doc/gorman/html/understand/understand006.html#toc19 https://www.kernel.org/doc/gorman/html/understand/understand...
- gary_bernhardt 13y ago
  I didn't know that! It certainly makes sense. Context switches still thrash the TLB, though. The performance cost of that has gotten better as time has gone on, but I wonder how many transistors (and how much power) CPUs are burning for that mitigation. The "how computers actually work" digression originally had a section on context switches, but I removed it early on because I felt like that section was dragging. To try to paint a very rough picture of the larger thoughts from which this talk was taken: I think that microkernels and the actor model are both the right thing (most of the time). When implemented naively, they both happen to take a big penalty from context switch cost. But Erlang can host a million processes in its VM, and we're using VMs for almost everything now anyway. The obvious (to me) solution is to move both the VM and an Erlang-style, single-address-space scheduler into the kernel. Then you can have a microkernel and a million native processes without the huge overhead of naive implementations. There are surely many huge practical hurdles to overcome with that, and maybe some that can't be overcome at all, but it sure sounds right when written in two paragraphs. ;)
  - deleted 13y ago
    [deleted]
  - nly 13y ago
    What you seem to be missing with re: asm.js is that, while the JIT to native code gets you your super-fast integer operations, it's still critically incomplete with regard to memory access. Every single individual memory access has to be bounds checked or pushed through some other indirection inside the runtime. Google demonstrated similar ideas with NaCl, which achieved safety with a similarly restricted native code and a just-in-time verification step. Even if these memory accesses could be made as efficient as those performed by the CPUs access protection, you're still not gaining anything you don't already have. Regarding context switches: A full CPU context switch on x86 (not to ring1 but between two arbitrary points within a single userland address space) takes a few dozen instructions and about 40-80 cycles. A single cache-line miss resulting in a load from main memory on the other hand takes at least twice that (~200 cycles). Again, hits from jumping around in memory will dominate. How significant is a 20% overhead from virtual memory? Probably about the same as getting 1% more of your memory accesses back in to high level caches.
microcolonel 13y ago
There's a new CPU architecture in the pipe(should have good silicon within five years, if their projections can be trusted), and it has a very good design around system calls, and also in terms of virtual memory. It has a single 64-bit address space and only protection contexts, and due to the general design of the system it doesn't require any register push(or registers at all, at least in the traditional sense). In addition, it has primitives which would allow programs to call directly into drivers and kernel services without a context switch. Anyway, I don't mean to sound like an advertisement, and we've yet to see any silicon, so the jury's out. Aside: Starting process address spaces at 0 is not really a convenience as far as I know(other than offering consistent addresses for jumping to static symbols), it's a way to enable PAE on 32-bit machines so that single contexts(typically processes) can use the whole address space.
- wolf550e 13y ago
  You forgot to provide a link: http://millcomputing.com/docs/ http://millcomputing.com/docs/
- sparkie 13y ago
  There's an even bigger revolution in CPU design, with Ivan Sutherland's Fleet - which does away with the clock and sequential execution of instructions - instead, the programming model is based on messaging and traffic control - you direct signals to the units in the chip which perform the computations you want, asynchronously by default - if you need synchronicity, you need to program your own models for it. While these probably won't be available in the next 5 years, and probably won't be acknowledged by existing programmers for decades - I think these ideas will take over. http://arc.cecs.pdx.edu/publications http://arc.cecs.pdx.edu/publications https://www.youtube.com/watch?v=jR9pAaQlVRc https://www.youtube.com/watch?v=jR9pAaQlVRc
tinco 13y ago
After the Operating Systems and Computer Organization courses in my first year of universisty I became a little obsessed with the idea of software managed operating systems. Cool to hear Singularity inspired you as well. Now I didn't read the paper, but I think the 20% is purely the MMU, I think the protection ring switching thing is much less significant, so I think if you leave the MMU still on that 20% profit is still very optimistic. Now, if you forget about compiling C (which defeats the purpose of your talk) and just compile managed languages like regular JS, the garbage collector can build a great model of memory usage. Therefore I think it could be much better to let the garbage collector manage both the isolation, and the swapping. So everything in software. The swapping process would suffer some performance, but that's just CPU cycles, as everyone knows persistent data access isn't even in the same league as CPU memory access. So yeah, that would mean that you would have to run all untrusted code in managed mode. And with untrusted I would mean code you can't trust with full physical memory access.

pjmlp 13y ago

In a way that is no different from the older Xerox PARC systems or the Oberon based ones at ETHZ. All of them are based on the concept of using memory safe languages for coding while leaving the runtime the OS role. Except for C and C++, language standard libraries tend to be very rich and to certain extent also offer the same features one would expect from OS services. As such, bypassing what we know as standard OS with direct hardware integration, coupled with language memory safety, could be an interesting design as well. That is why I follow the Mirage, Erlang on Xen and HaLVM research.