5 ms·

I didn't want to go into this level of detail in the talk, but... I think you still want the MMU enabled, just not used for process isolation. With virtual memo

I didn't want to go into this level of detail in the talk, but... I think you still want the MMU enabled, just not used for process isolation. With virtual memory totally disabled, a 1 GB malloc takes 1 GB physical memory even if it's not touched, you can't have swap at all, memory fragmentation kills you dead, etc. It still has a lot of utility outside of isolation. I don't have a good sense of how the performance cost of hardware isolation breaks down into {virtual memory enabled,TLB thrashing,protection ring switching}. That's one of the reasons that I reduced the speed-up from "25-33%" in the MSR paper down to 20% in METAL. Maybe the speed-up would be less than that if virtual memory were still enabled. Unfortunately, that distinction may have been blurred in the talk. That is, I may have implied that METAL would turn the MMU off entirely. If so, it was an oversight. I've done the talk end-to-end at least fifty times, which is how I smooth my execution out. Occasionally it can "smooth" the ideas out a bit too, leading to small inaccuracies. It's sort of like playing the telephone game with yourself (which is a very strange experience). The MSR paper that I quote came from the Singularity team, so your reference is right on. Reading "Deconstructing Process Isolation" in fall of 2012 was probably the germ of the core narrative of the talk.

nteon 12y ago
On linux, system calls don't result in a TLB flush - kernel data structures and code are in a different portion of the virtual address space (starting from the top of VM memory, if I remember right) that is tagged as not being available from ring 3. So system calls are quite fast. EDIT: Kernel memory begins at PAGE_OFFSET, see here: https://www.kernel.org/doc/gorman/html/understand/understand007.html https://www.kernel.org/doc/gorman/html/understand/understand... Kernel memory lacks the flag _PAGE_USER so that it isn't accessible from userspace: https://www.kernel.org/doc/gorman/html/understand/understand006.html#toc19 https://www.kernel.org/doc/gorman/html/understand/understand...
- gary_bernhardt 12y ago
  I didn't know that! It certainly makes sense. Context switches still thrash the TLB, though. The performance cost of that has gotten better as time has gone on, but I wonder how many transistors (and how much power) CPUs are burning for that mitigation. The "how computers actually work" digression originally had a section on context switches, but I removed it early on because I felt like that section was dragging. To try to paint a very rough picture of the larger thoughts from which this talk was taken: I think that microkernels and the actor model are both the right thing (most of the time). When implemented naively, they both happen to take a big penalty from context switch cost. But Erlang can host a million processes in its VM, and we're using VMs for almost everything now anyway. The obvious (to me) solution is to move both the VM and an Erlang-style, single-address-space scheduler into the kernel. Then you can have a microkernel and a million native processes without the huge overhead of naive implementations. There are surely many huge practical hurdles to overcome with that, and maybe some that can't be overcome at all, but it sure sounds right when written in two paragraphs. ;)
  - deleted 12y ago
    [deleted]
  - nly 12y ago
    What you seem to be missing with re: asm.js is that, while the JIT to native code gets you your super-fast integer operations, it's still critically incomplete with regard to memory access. Every single individual memory access has to be bounds checked or pushed through some other indirection inside the runtime. Google demonstrated similar ideas with NaCl, which achieved safety with a similarly restricted native code and a just-in-time verification step. Even if these memory accesses could be made as efficient as those performed by the CPUs access protection, you're still not gaining anything you don't already have. Regarding context switches: A full CPU context switch on x86 (not to ring1 but between two arbitrary points within a single userland address space) takes a few dozen instructions and about 40-80 cycles. A single cache-line miss resulting in a load from main memory on the other hand takes at least twice that (~200 cycles). Again, hits from jumping around in memory will dominate. How significant is a 20% overhead from virtual memory? Probably about the same as getting 1% more of your memory accesses back in to high level caches.
    - andhow 12y ago
      On x64 in Firefox, at least, there are no bounds checks; the index is a uint32; the entire accessible 4GB range is mapped PROT_NONE with only the accessible region mapped PROT_READ|PROT_WRITE; out-of-bounds accesses thus reliably turn into SIGSEGVs which are handled safely after which execution resumes. Thus, bounds checking is effectively performed by the MMU.
      - nly 12y ago
        Interesting approach, might have to look at the code. Nonetheless it highlights how useful the MMU is and how none of this is free.
    - cornholio 12y ago
      I agree, and I think the whole premise of the performance gain is based on a "have the cake and eat it too" fallacy. Sure, the virtualized syscall to the virtualized OS will be free, but the painting of the font on the screen or the reading of the socket data will be done by the actual bare metal OS which the VM will invoke to get the actual job done. So as long as we are talking about interprocess communication there will be a gain, but not for the actual hardware facing operation. Then again, you are trading a hardware enforced isolation which is simple and proven for a isolation enforced by a complex and fragile VM.
microcolonel 12y ago
There's a new CPU architecture in the pipe(should have good silicon within five years, if their projections can be trusted), and it has a very good design around system calls, and also in terms of virtual memory. It has a single 64-bit address space and only protection contexts, and due to the general design of the system it doesn't require any register push(or registers at all, at least in the traditional sense). In addition, it has primitives which would allow programs to call directly into drivers and kernel services without a context switch. Anyway, I don't mean to sound like an advertisement, and we've yet to see any silicon, so the jury's out. Aside: Starting process address spaces at 0 is not really a convenience as far as I know(other than offering consistent addresses for jumping to static symbols), it's a way to enable PAE on 32-bit machines so that single contexts(typically processes) can use the whole address space.
- wolf550e 12y ago
  You forgot to provide a link: http://millcomputing.com/docs/ http://millcomputing.com/docs/
- sparkie 12y ago
  There's an even bigger revolution in CPU design, with Ivan Sutherland's Fleet - which does away with the clock and sequential execution of instructions - instead, the programming model is based on messaging and traffic control - you direct signals to the units in the chip which perform the computations you want, asynchronously by default - if you need synchronicity, you need to program your own models for it. While these probably won't be available in the next 5 years, and probably won't be acknowledged by existing programmers for decades - I think these ideas will take over. http://arc.cecs.pdx.edu/publications http://arc.cecs.pdx.edu/publications https://www.youtube.com/watch?v=jR9pAaQlVRc https://www.youtube.com/watch?v=jR9pAaQlVRc
tinco 12y ago
After the Operating Systems and Computer Organization courses in my first year of universisty I became a little obsessed with the idea of software managed operating systems. Cool to hear Singularity inspired you as well. Now I didn't read the paper, but I think the 20% is purely the MMU, I think the protection ring switching thing is much less significant, so I think if you leave the MMU still on that 20% profit is still very optimistic. Now, if you forget about compiling C (which defeats the purpose of your talk) and just compile managed languages like regular JS, the garbage collector can build a great model of memory usage. Therefore I think it could be much better to let the garbage collector manage both the isolation, and the swapping. So everything in software. The swapping process would suffer some performance, but that's just CPU cycles, as everyone knows persistent data access isn't even in the same league as CPU memory access. So yeah, that would mean that you would have to run all untrusted code in managed mode. And with untrusted I would mean code you can't trust with full physical memory access.

nteon 12y ago

On linux, system calls don't result in a TLB flush - kernel data structures and code are in a different portion of the virtual address space (starting from the top of VM memory, if I remember right) that is tagged as not being available from ring 3. So system calls are quite fast. EDIT: Kernel memory begins at PAGE_OFFSET, see here: https://www.kernel.org/doc/gorman/html/understand/understand007.html https://www.kernel.org/doc/gorman/html/understand/understand... Kernel memory lacks the flag _PAGE_USER so that it isn't accessible from userspace: https://www.kernel.org/doc/gorman/html/understand/understand006.html#toc19 https://www.kernel.org/doc/gorman/html/understand/understand...

gary_bernhardt 12y ago
I didn't know that! It certainly makes sense. Context switches still thrash the TLB, though. The performance cost of that has gotten better as time has gone on, but I wonder how many transistors (and how much power) CPUs are burning for that mitigation. The "how computers actually work" digression originally had a section on context switches, but I removed it early on because I felt like that section was dragging. To try to paint a very rough picture of the larger thoughts from which this talk was taken: I think that microkernels and the actor model are both the right thing (most of the time). When implemented naively, they both happen to take a big penalty from context switch cost. But Erlang can host a million processes in its VM, and we're using VMs for almost everything now anyway. The obvious (to me) solution is to move both the VM and an Erlang-style, single-address-space scheduler into the kernel. Then you can have a microkernel and a million native processes without the huge overhead of naive implementations. There are surely many huge practical hurdles to overcome with that, and maybe some that can't be overcome at all, but it sure sounds right when written in two paragraphs. ;)
- deleted 12y ago
  [deleted]
- nly 12y ago
  What you seem to be missing with re: asm.js is that, while the JIT to native code gets you your super-fast integer operations, it's still critically incomplete with regard to memory access. Every single individual memory access has to be bounds checked or pushed through some other indirection inside the runtime. Google demonstrated similar ideas with NaCl, which achieved safety with a similarly restricted native code and a just-in-time verification step. Even if these memory accesses could be made as efficient as those performed by the CPUs access protection, you're still not gaining anything you don't already have. Regarding context switches: A full CPU context switch on x86 (not to ring1 but between two arbitrary points within a single userland address space) takes a few dozen instructions and about 40-80 cycles. A single cache-line miss resulting in a load from main memory on the other hand takes at least twice that (~200 cycles). Again, hits from jumping around in memory will dominate. How significant is a 20% overhead from virtual memory? Probably about the same as getting 1% more of your memory accesses back in to high level caches.
  - andhow 12y ago
    On x64 in Firefox, at least, there are no bounds checks; the index is a uint32; the entire accessible 4GB range is mapped PROT_NONE with only the accessible region mapped PROT_READ|PROT_WRITE; out-of-bounds accesses thus reliably turn into SIGSEGVs which are handled safely after which execution resumes. Thus, bounds checking is effectively performed by the MMU.
    - nly 12y ago
      Interesting approach, might have to look at the code. Nonetheless it highlights how useful the MMU is and how none of this is free.
  - cornholio 12y ago
    I agree, and I think the whole premise of the performance gain is based on a "have the cake and eat it too" fallacy. Sure, the virtualized syscall to the virtualized OS will be free, but the painting of the font on the screen or the reading of the socket data will be done by the actual bare metal OS which the VM will invoke to get the actual job done. So as long as we are talking about interprocess communication there will be a gain, but not for the actual hardware facing operation. Then again, you are trading a hardware enforced isolation which is simple and proven for a isolation enforced by a complex and fragile VM.

microcolonel 12y ago

There's a new CPU architecture in the pipe(should have good silicon within five years, if their projections can be trusted), and it has a very good design around system calls, and also in terms of virtual memory. It has a single 64-bit address space and only protection contexts, and due to the general design of the system it doesn't require any register push(or registers at all, at least in the traditional sense). In addition, it has primitives which would allow programs to call directly into drivers and kernel services without a context switch. Anyway, I don't mean to sound like an advertisement, and we've yet to see any silicon, so the jury's out. Aside: Starting process address spaces at 0 is not really a convenience as far as I know(other than offering consistent addresses for jumping to static symbols), it's a way to enable PAE on 32-bit machines so that single contexts(typically processes) can use the whole address space.

wolf550e 12y ago
You forgot to provide a link: http://millcomputing.com/docs/ http://millcomputing.com/docs/
sparkie 12y ago
There's an even bigger revolution in CPU design, with Ivan Sutherland's Fleet - which does away with the clock and sequential execution of instructions - instead, the programming model is based on messaging and traffic control - you direct signals to the units in the chip which perform the computations you want, asynchronously by default - if you need synchronicity, you need to program your own models for it. While these probably won't be available in the next 5 years, and probably won't be acknowledged by existing programmers for decades - I think these ideas will take over. http://arc.cecs.pdx.edu/publications http://arc.cecs.pdx.edu/publications https://www.youtube.com/watch?v=jR9pAaQlVRc https://www.youtube.com/watch?v=jR9pAaQlVRc

tinco 12y ago

After the Operating Systems and Computer Organization courses in my first year of universisty I became a little obsessed with the idea of software managed operating systems. Cool to hear Singularity inspired you as well. Now I didn't read the paper, but I think the 20% is purely the MMU, I think the protection ring switching thing is much less significant, so I think if you leave the MMU still on that 20% profit is still very optimistic. Now, if you forget about compiling C (which defeats the purpose of your talk) and just compile managed languages like regular JS, the garbage collector can build a great model of memory usage. Therefore I think it could be much better to let the garbage collector manage both the isolation, and the swapping. So everything in software. The swapping process would suffer some performance, but that's just CPU cycles, as everyone knows persistent data access isn't even in the same league as CPU memory access. So yeah, that would mean that you would have to run all untrusted code in managed mode. And with untrusted I would mean code you can't trust with full physical memory access.