5 ms·
The Mill's memory model is one of its most interesting features IMO [1] and solves some of the same problems, but by going the other way. On the Mill the whole
by infogulch 4y ago
The Mill's memory model is one of its most interesting features IMO [1] and solves some of the same problems, but by going the other way.
On the Mill the whole processor bank uses a global virtual address space. TLB and mapping to physical memory happens at the memory controller. Everything above the memory controller is in the same virtual address space, including L1-L3+ caches. This solves a lot of problems, for example: If you go out to main memory you're already paying ~300 cycles of latency, so having a large silicon area / data structure for translation is no longer a 1-cycle latency problem. Writes to main memory are flushed down the same memory hierarchy that reads come from and succeed as soon as they hit L1. Since all cache lines are in the same virtual address space you don't have to track and synchronize reads and writes across translation zones within the cache hierarchy. When you request an unallocated page you get the whole pre-zeroed page back instantly, since it doesn't need to be mapped to physical pages until writes are flushed out of L3. This means its possible for a page to be allocated, written to, read, and deallocated which never actually touches physical memory throughout the whole sequence and the whole workload is served purely within the cache hierarchy.
Protection is a separate system ("PLB") and can be much smaller and more streamlined since it's not trying to do two jobs at once. The PLB allows processes to give fine-grained temporary access of a portion of its memory to another process; RW, Ro, Wo, byte-addressed ranges, for one call or longer etc. Processes get allocated available address space on start, they can't just assume they own the whole address space or start at some specific address (you should be using ASLR anyways so this should have no effect on well-formed programs, though there is a legacy fallback).
[1]: My previous comment: https://news.ycombinator.com/item?id=27952660 https://news.ycombinator.com/item?id=27952660
- pclmulqdq 4y agoThe Mill model is kind of cool, but today, many peripherals (including GPUs and NICs) have the ability to dump bytes straight into L3 cache. This improves latency in a lot of tasks, including the server-side ones that the Mill is designed for. This is possible due to the fact that MMUs are above the L3 cache. Honestly, I'm happy waiting for 4k pages to die and be replaced by huge pages. Page tables were added to the x86 architecture in 1985, when 1MB of memory was a ton of memory to have. Having 256 pages worth of memory in your computer was weird and exotic. Fast forward to today, and the average user has several GB of memory - mainstream computers can be expanded to over 128 GB today - and we still mainly use 4k pages. That is the problem here. If we could swap to 2M pages in most applications, we would be able to reduce page table sizes by a factor of 512, and they would still be a lot larger than page tables when virtual memory was invented. And we wouldn't waste much memory! But no, 4k pages for backwards compatibility. 4k pages forever. And while we're at it, let's add features to Linux (like TCP zero copy) that rely on having 4k pages.
- rsaxvc 4y agoOnce on an embedded system, we found 30% of our processing time was spent handling TLB misses. We also didn't have access to the code that set up the page tables. Somehow I got the linker to emit a function hook which I used to defrag the page tables after they were written but before the MMU was enabled. I remember thinking, "This is not what I learned about in school."
- pclmulqdq 4y agoIn a server system I worked on recently, 10% of its time was spent on page table walks of a multi-GB structure that it allocates flat. That was nuts to me. It could have been 0.
- StillBored 4y agoI can probably guess the OS you were running, but lets just say, some OS's put a lot more effort into assure that virtually continuous mappings tend to also be physically continuous because it then becomes possible to map larger blocks of ram continuously. AKA, both X86 and ARM (as do various other processors) have intermediate bits in their page directory structures that flag 'instead of using another level of PTE/etc below this level assume we are just pointing at an equal size physical range" which on x86-64 can be 2M or 1G. Given the latest Intel x86's have a multilevel TLB, with 8 1G L1 entries, and 1K of L2 TLB entries it should be fairly straightforward to fix with appropriate huge page tweaks unless your data structure exceeds a TB. And even if it does, keeping everything in 1G pages means that the data caches should be able to keep the top level global page directory/etc cached avoiding main ram hits, and resulting in fairly quick TLB refills.
- Symmetry 4y agoThe obvious thing to do is put the address translation between the CPU cache and system level cache.
- ansible 4y agoAgree completely. If you have any interest at all in the design of CPUs and instruction sets, I recommend viewing the entire series of video lectures by Ivan Godard: https://www.youtube.com/playlist?list=PLFls3Q5bBInj_FfNLrV7gGdVtikeGoUc9 https://www.youtube.com/playlist?list=PLFls3Q5bBInj_FfNLrV7g... Warning: Viewing these is likely to make you at least a little bit upset with the current state-of-the-art architectures.
- infogulch 4y agoReading over the article and this thread again, I realize that this isn't the 'opposite direction', this is exactly what the article is aiming at.