9 ms·
Linux Memory Management FAQ
- deleted 6y ago[deleted]
- blt 6y agoMuch of this applies to other OSes with virtual memory also.
- dbattaglia 6y ago"Virtual addresses are the size of a CPU register. On 32 bit systems each process has 4 gigabytes of virtual address space all to itself, which is often more memory than the system actually has." I guess this is not the most up-to-date document?
- sigjuice 6y agoI think there might be some more hardware-specific nuance here. e.g. /proc/cpuinfo says this on a couple of different x86_64 systems that I checked. address sizes : 36 bits physical, 48 bits virtual address sizes : 40 bits physical, 48 bits virtual PS: I don't understand what this means, btw.
- db48x 6y agoYour CPU can handle 39-bit physical memory addresses (up to 512 GB of physical memory), and 48-bit virtual addresses (256 TB). Your operating system maintains a mapping from virtual to physical addresses, usually arranging the map so that every process has a separate memory space. Pointers are all still 64 bits long though.
- barnacled 6y agoIn practice the actual available usable address space for userland is 64 TiB due to user/kernel split and the kernel maintaining a virtual mapping of the entire physical address space (minus I/O ranges) [0]. However newer incoming 5-level page intel chips [1] will allow up to 57 bits of address space, 128 PiB in theory though in practice 32 PiB of userland memory. See also [0] for discussion on practical limit for 5-page too! [0]:https://github.com/lorenzo-stoakes/linux-mm-notes/blob/master/virt_layout.md#available-address-space https://github.com/lorenzo-stoakes/linux-mm-notes/blob/maste... [1]:https://en.wikipedia.org/wiki/Intel_5-level_paging https://en.wikipedia.org/wiki/Intel_5-level_paging
- db48x 6y agoTrue, though /proc/cpuinfo only reports the size, which is ultimately what the CPU cares about. Plus the most relevant limit is what your motherboard and wallet supports, which is often far lower.
- barnacled 6y agoIndeed, and as you say, sensibly speaking you are hardly likely to hit those limits in any likely (esp. home) setup. The actual meaningful limit is usually the CPU physical one as home CPUs very often have stringent memory limits (often 32 GiB or so) and of course you rely on the motherboard's limitations also. Having said that I did write a patch to ensure that the system would boot correctly with 256 TiB of RAM [0] so perhaps I am not always a realist... or dream of the day I can own that system ;) [0]:https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/?id=167dcfc08b0b1f964ea95d410aa496fd78adf475 https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-n...
- db48x 6y agoYou're not the only one dreaming; I had to use >200GB of swap on my home system last year.
- sigjuice 6y agoSo are the 16 leftmost bits of a virtual address always 0?
- barnacled 6y agoThey have to be same as the maximum addressable bit, i.e. in the case of 48 bit virtual address size the 48th bit. This is actually kind of a cute way of dividing kernel and userland space as you just set the upper bit to 1 for kernel addresses and 0 for userland. EDIT: Specifically talking about x86-64 here. https://github.com/lorenzo-stoakes/linux-mm-notes/blob/master/virt_layout.md#virtual-address-layout https://github.com/lorenzo-stoakes/linux-mm-notes/blob/maste...
- amscanne 6y agoNo, it must be sign-extended from the top bit of the valid set. Otherwise the address is non-canonical.
- xxpor 6y agoThis is true for x86-64, not true for other architectures such as arm64. Apple uses the high bits to cryptographicly sign the pointer value.
- everybodyknows 6y agoFascinating. Does this confer some of the benefits of ECC RAM, for pointer data only — without the hardware cost?
- danielheath 6y agoSome (but I believe the advantage is that it’s much harder to inject valid code from a buffer overflow).
- saagarjha 6y agoPAC is more about CFI than preventing shellcode injection (which is done through codesigning and memory protection, mostly).
- throwaway8581 6y agoIt means that it can address 40-bits of address space worth of physical memory, but that virtual memory addresses can use 48 bits. Physical addresses are just your RAM bytes numbered 1 through whatever. Virtual address space is the address space of a process, which includes mapped physical memory, unmapped pages, guard pages, and other virtual memory tricks.
- hansendc 6y ago> Physical addresses are just your RAM bytes numbered 1 through whatever. Not really. There are lots of holes in the physical address map. Look at /proc/iomem. Look at all of the gunk in there at addresses lower than the amount of RAM you have. Look at the highest “System RAM” address. It will be higher than the amount of actual physical RAM that you have.
- littlestymaar 6y agoIf you are thinking about the “which is often more memory than the system actually has" part, I don't know if it's outdated even today: the vast majority of Linux systems these days are Android phones, and I wouldn't be surprised at all if a good proportion of those didn't have more than 4GB of RAM.
- bonzini 6y agoOn 32-bit systems, 4 GiB is indeed often more memory than the system has (think 512 MiB for some Raspberry Pis). And on 64-bit x86 systems each process has 256 PiB, which is also more memory than the system has.
- kelnos 6y agoI think that's probably still true for what 32-bit systems are still out there today. And regardless, I think the majority of systems running Linux today are phones, which usually have 4GB or less of RAM. But I expect the FAQ was probably originally thinking about desktop or server systems, so, yeah, the intent there is probably out of date. Those types of systems are rarely 32-bit these days, and usually have a bit more than 4GB of RAM.
- spijdar 6y ago> I think the majority of systems running Linux today are phones, which usually have 4GB or less of RAM. Even this is quickly becoming less and less true (for new phones). Even the Pinephone comes with 3 GB of RAM at a $200 price point, and that's inflated because of the niche, low volume nature of its production. Samsung's "mid range" A series smartphones, for instance, start at 3GB at the absolute lowest end, with most models coming with 6 GB of memory. I expect this will be even more common in a year or two.
- mhh__ 6y agoMy OnePlus 3T is nearly 4 years old now and has 6GB (and is really showing its age...)
- forty 6y agoWhat's the use case of having so much ram on a smartphone ? Gaming?
- herpderperator 6y agoApp-switching (multitasking) without LRU apps getting force-closed to make room for active apps. In other words, if you like to keep apps open, more RAM will reduce the chances of an app opened a while ago having to "start fresh" when you switch back to it, losing whatever state it had when you last used it.
- ta988 6y agoAllowing app developers to not worry about optimization, put more trackers and more annoying ads...
- gruez 6y ago>I guess this is not the most up-to-date document? it's also not correct. It doesn't have all 4GB "all to itself", because a portion of that (usually 1 or 2 GB) is mapped to the kernel.
- Out_of_Characte 6y ago"each process has 4 gigabytes of virtual address space all to itself" A process does indeed have all 4GB of VIRTUAL adress space to itself. unless I'm misunderstanding you.
- dezgeg 6y agoYes, VIRTUAL memory. Most operating systems (Windows, Linux) leave 2 or 3 GB for the user process and reserve the of the address space for themselves. That way userspace-to-kernel switch does not require changing active page table (and also avoids switcharoo each time kernel needs to access userspace memory).
- deleted 6y ago[deleted]
- barnacled 6y agoFor anybody who's interested I also wrote up a whole bunch of notes on this at https://github.com/lorenzo-stoakes/linux-vm-notes https://github.com/lorenzo-stoakes/linux-vm-notes and superceded by far more recent https://github.com/lorenzo-stoakes/linux-mm-notes https://github.com/lorenzo-stoakes/linux-mm-notes I have made a few patches into the mm subsystem some simply inspired by researching for the articles.
- aduitsis 6y agoThank you, this is great!
- barnacled 6y agoThank you! I wanted to reply last night but got rate limited. I've been questioning my side project recently (hence why no updates for a month) so hearing positive feedback from people does help motivate.
- thricegr8 6y agoIf someone wants to begin their journey in understanding and level of aptitude you've displayed on your sites, where should they begin?
- barnacled 6y agoThanks, I am reading the kernel source as I go and using that to answer questions as I figure things out.
- phtrivier 6y agoThe Drepper series of article dates from 2007. Is it still relevant or has anything fundamental changed in memory handling in the last 13 years ?
- Agingcoder 6y agoIt's still relevant. Other than that, I also think that even when outdated, computing history is worth reading anyway, since it gives you a natural understanding of _why_ we do what we do these days. In your day job, it also gives you a different appreciation for what people did and why they did it, and why 'this horrible code' may have made sense at the time. Furthermore, performance engineering is fundamentally about opposing code and hardware limitations. If hardware limitations are different, you'll get different code, but the principles remain the same. If you're curious, write a basic emulator for older hardware (the NES is a great choice) , it's both fun and eye-opening! Edit: the NES emulator will answer 'how do you fit super mario bros in 32k, and how can it run on such limited hardware?'
- einpoklum 6y ago> computing history is worth reading anyway Sometimes, but a description of the state of the art in the past does not become a historical tract with the passage of time. The better ones do; others just become outdated.
- Agingcoder 6y agoWell, the ones which fail (and which become outdated) can also teach us valuable lessons : looking at the current state of the art doesn't necessarily tells you what happens if you do things differently. In other words, we tend to focus on positive results, but negative ones ('don't do this or.. !') can be equally interesting and useful.
- brandmeyer 6y agoHighly relevant. The only part that I would discount is that he was pretty bullish on the prospects for hardware transactional memory, and his forward-looking statements about it didn't pan out. In fairness, much of the industry was bullish about HTM at that time.
- mlaretallack 6y agoThe times I have had to explain how mm works is draining. yes you can malloc 2M, no that does not mean you have 2M to use.
- dataflow 6y agoWell, it does mean that in C. But some folks prefer to play by their own rules.
- barnacled 6y agoActually no, the malloc doesn't allocate any memory it just updates the process's VMA to say that the allocated virtual range is valid. The pages are then faulted in on write. This is where things like OOM killer become very confusing for people. In linux (in sane configurations) allocations are just preorders. EDIT: I can't reply below due to rate limiting: I'd argue that overcommit just makes the difference between allocation and backing very stark. Your memory IS in fact allocated in the process VMA, it's just the anonymous pages cannot necessarily be backed. This differs, obviously, in other OSes as pointed out. Also differs if you turn overcommit off but since so much in linux assumes it your system will soon break if you try it.
- dataflow 6y agoI said "in C". You're talking "in Linux" (or glibc/whatever). Which, as I already said, plays by its own rules and defies C. It's broken by design.
- AnimalMuppet 6y agoSo if I malloc 2 MB or 2 GB or whatever in a C program running on Linux, but I have not yet either read from or written to that memory, then what's the state? Has the C library forced Linux to actually allocate it, or has it not? Or does it depend, and if so, on what?
- 6y ago