3 ms·

> what does locking memory mean in this context? I think the person is referring to the mlock() and mlockall() functions (or equivalents on other OS), which ke

> what does locking memory mean in this context? I think the person is referring to the mlock() and mlockall() functions (or equivalents on other OS), which keep pages resident / prevents pages from being paged out. Forces them to remain in RAM. It can be used to, e.g., prevent an encryption key or password from being swapped out to disk, where it might then be recoverable. (Personally, this is why I encrypt swap.) > what is the performance degradation that happens with low-spec computers? Locking a larger portion of RAM means less room for the OS to page out unused pages and free up the space for other programs. While one can try to selectively lock buffers with sensitive data with mlock(), you have to be sure they aren't copied into other buffers that aren't locked (and could thus be subsequently paged out). If you're writing a UI program that displays or receives those in a widget, this might be harder (you might not have access to the internal buffer of the widget, as it is an "implementation detail" of your library), and locking the entire process might be a simpler solution (albeit being a bigger hammer).

rkeene2 8y ago
Linux cgroups and Solaris containers atleast also provide a way to avoid paging to disk, without modifying the program.
scott_s 8y ago
Good explanations of what and why one might lock memory. But, in the context of this overall discussion, I think it's important to keep in mind that when a process allocates a large amount of virtual memory, it does not automatically allocate any physical memory. So a process allocating a lot of virtual memory up front should not impact other processes which have locked some of their memory into physical memory.
- deathanatos 8y ago
  > I think it's important to keep in mind that when a process allocates a large amount of virtual memory, it does not automatically allocate any physical memory. It will, if you mlock() it, I believe. The manual page notes "real-time processes" as a main user of mlock() (the other being the cryptographic uses I hinted at); it cites their use case as locking the page to avoid delays due to paging during critical sections. In order for that to work, the OS would need to bring the pages in, at the time of locking; so at that point, a large virtual allocation becomes equivalent to a physical one.
  - scott_s 8y ago
    Correct, but my point was that a process with allocates a lot of physical memory will not interfere with the other processes that have locked pages.
monocasa 8y ago
My understanding was that mlock doesn't really keep the page from being paged out, since it can't even begin to do that in the hibernation case, or if you're running as a VM who's guest RAM wasn't mlocked. What it does is keep a canonical version of the page in memory. That's useful being able to deterministically touch a piece of memory, but it doesn't really help you as far as making sure the page never touches disk.
- deathanatos 8y ago
  I hadn't considered hibernation, and indeed, a deeper reading of the manual confirms that hibernation doesn't count, which is rather interesting (given the implications of hitting disk), but I don't really see a good way around it, short of aborting the hibernation, or providing a mechanism to inform the program that those pages were lost. The man page (later, annoyingly, after its initial description) notes this: > Memory locking has two main applications: real-time algorithms and high-security data processing. Real-time applications require deterministic timing, and, like scheduling, paging is one major cause of unexpected program execution delays. Real-time applications will usually also switch to a real-time scheduler with sched_setscheduler(2). Cryptographic security software often handles critical bytes like passwords or secret keys as data structures. As a result of paging, these secrets could be transferred onto a persistent swap store medium, where they might be accessible to the enemy long after the security software has erased the secrets in RAM and terminated. (But be aware that the suspend mode on laptops and some desktop computers will save a copy of the system's RAM to disk, regardless of memory locks.) I would be rather disappointed if a hypervisor swapped out my guest (at least, in a context like AWS; I suppose if you're just running qemu on your laptop, that's a different matter), but I hadn't considered that either, and it is certainly possible.
  - monocasa 8y ago
    IMO, the right answer is to better define your threat model. Are you concerned about someone pulling the HDD and reading the swap? Use an FDE scheme that covers your swap too. Are you concerned about someone getting access to the swap file programmatically? At that point they have so many other ways of slurping the memory out of your process that it's a lost cause.

rkeene2 8y ago

Linux cgroups and Solaris containers atleast also provide a way to avoid paging to disk, without modifying the program.

scott_s 8y ago

Good explanations of what and why one might lock memory. But, in the context of this overall discussion, I think it's important to keep in mind that when a process allocates a large amount of virtual memory, it does not automatically allocate any physical memory. So a process allocating a lot of virtual memory up front should not impact other processes which have locked some of their memory into physical memory.

deathanatos 8y ago
> I think it's important to keep in mind that when a process allocates a large amount of virtual memory, it does not automatically allocate any physical memory. It will, if you mlock() it, I believe. The manual page notes "real-time processes" as a main user of mlock() (the other being the cryptographic uses I hinted at); it cites their use case as locking the page to avoid delays due to paging during critical sections. In order for that to work, the OS would need to bring the pages in, at the time of locking; so at that point, a large virtual allocation becomes equivalent to a physical one.
- scott_s 8y ago
  Correct, but my point was that a process with allocates a lot of physical memory will not interfere with the other processes that have locked pages.

monocasa 8y ago

My understanding was that mlock doesn't really keep the page from being paged out, since it can't even begin to do that in the hibernation case, or if you're running as a VM who's guest RAM wasn't mlocked. What it does is keep a canonical version of the page in memory. That's useful being able to deterministically touch a piece of memory, but it doesn't really help you as far as making sure the page never touches disk.

deathanatos 8y ago
I hadn't considered hibernation, and indeed, a deeper reading of the manual confirms that hibernation doesn't count, which is rather interesting (given the implications of hitting disk), but I don't really see a good way around it, short of aborting the hibernation, or providing a mechanism to inform the program that those pages were lost. The man page (later, annoyingly, after its initial description) notes this: > Memory locking has two main applications: real-time algorithms and high-security data processing. Real-time applications require deterministic timing, and, like scheduling, paging is one major cause of unexpected program execution delays. Real-time applications will usually also switch to a real-time scheduler with sched_setscheduler(2). Cryptographic security software often handles critical bytes like passwords or secret keys as data structures. As a result of paging, these secrets could be transferred onto a persistent swap store medium, where they might be accessible to the enemy long after the security software has erased the secrets in RAM and terminated. (But be aware that the suspend mode on laptops and some desktop computers will save a copy of the system's RAM to disk, regardless of memory locks.) I would be rather disappointed if a hypervisor swapped out my guest (at least, in a context like AWS; I suppose if you're just running qemu on your laptop, that's a different matter), but I hadn't considered that either, and it is certainly possible.
- monocasa 8y ago
  IMO, the right answer is to better define your threat model. Are you concerned about someone pulling the HDD and reading the swap? Use an FDE scheme that covers your swap too. Are you concerned about someone getting access to the swap file programmatically? At that point they have so many other ways of slurping the memory out of your process that it's a lost cause.