5 ms·

I do this in some programs, it makes many pieces of coding much easier if I know I will never have to move some objects as they grow. The ideal for me would be

I do this in some programs, it makes many pieces of coding much easier if I know I will never have to move some objects as they grow. The ideal for me would be a function that marked some memory as "these addresses are taken, do not give them out to malloc, or anything else", but which still required me to actually "ask" for the memory before using it, so it didn't look like I was using 64gb of memory at startup. Is that possible?

sjmulder 8y ago
On Windows that's VirtualAlloc with MEM_RESERVE: Reserves a range of the process's virtual address space without allocating any actual physical storage in memory or in the paging file on disk. https://msdn.microsoft.com/en-us/library/Aa366887(v=VS.85).aspx https://msdn.microsoft.com/en-us/library/Aa366887(v=VS.85).a... Perhaps mmap() can achieve the same on Unices?
- cyphar 8y ago
  This is what Linux does by default for anonymous memory (it's called overcommitment). Go makes very liberal use of this, but it has other issues (namely Linux will kill a process if it goes over a certain overcommitment threshold).
  - scott_s 8y ago
    I'm not sure overcommitment is the same thing: https://www.kernel.org/doc/Documentation/vm/overcommit-accounting https://www.kernel.org/doc/Documentation/vm/overcommit-accou... Reading that, I think overcommitment is to determine the kernel's behavior when you try to allocate more virtual memory than physical memory that is present on the system. That is a different (but related) concern from the fact that mmap() will allocate virtual memory but not physical memory. (That is, mmap() reserves locations in the address space, but you don't have any physical memory backing it until you use that memory.)
    - cyphar 8y ago
      Yeah, sorry -- mmap()'s lazy allocation of pages is a separate but related concept to overcommit (I was writing a tirade about that in a separate thread and my wires got crossed).
- scott_s 8y ago
  My first thought on unix was mmap() as well, but one of the requirements was that it doesn't "look like" the process is using that memory, which I interpret as "I don't want to allocate this as virtual memory." I don't think mmap() allows that, as its entire purpose is to allocate virtual memory and return a pointer to that memory space. Personally, my feeling is: why does it matter if it "looks like" a process is using a lot of memory? That is, why does it matter if a process allocates a lot of virtual memory up front? It's not consuming physical memory, it's just updating some bookkeeping in the kernel. I know people feel uneasy about seeing large values for virtual memory, but... they shouldn't.
boomlinde 8y ago
> The ideal for me would be a function that marked some memory as "these addresses are taken, do not give them out to malloc, or anything else", but which still required me to actually "ask" for the memory before using it, so it didn't look like I was using 64gb of memory at startup. Is that possible? So you want to reserve memory, basically the way malloc does, depriving any other process of that memory, but you then want to reserve it again, and also somehow hide that the memory has been reserved? What exactly is the benefit except making it not look like you're using the memory? Actually, what's the benefit of that in itself?
- twtw 8y ago
  No, he wants to separate virtual address space allocation from physical memory backing allocation, with both still happening manually. It's a very reasonable request, and the benefit is that you can be guaranteed to have the address space while still only using a smaller max amount of physical memory.
  - scott_s 8y ago
    I don't interpret that as the request. I interpret it as wanting virtual memory "allocated" but not counted against the processes' virtual memory allocation, and then getting a secondary mechanism to allocate the sorta-but-not-quite-allocated memory. I also don't see the value in that. (I don't see any request for different physical memory allocation, so I assume that would still be handled by page faulting in the kernel.)
    - koala_man 8y ago
      The value would be to be able to grow arrays and similar without relocation. People do it all the time in real life, with area codes, zip codes, case numbers, etc. For example, a long street full of strip malls in California will often have street numbers 50 apart to allow them to remain sequential even after new developments. No one would complain that this is a wasteful use of precious street numbers, or that it deprives other streets of those numbers. Imagine the nightmare if you periodically had to renumber all the buildings instead, like computers routinely have to.
kccqzy 8y ago
On FreeBSD there's MAP_GUARD. From the documentation: > Instead of a mapping, create a guard of the specified size. Guards allow a process to create reservations in its address space, which can later be replaced by actual mappings. mmap will not create mappings in the address range of a guard unless the request specifies MAP_FIXED. Guards can be destroyed with munmap(2). Any memory access by a thread to the guarded range results in the delivery of a SIGSEGV signal to that thread. So it seems like a good fit for your use case, but I've never used this.
- CJefferson 8y ago
  That is exactly what I want! I wonder if it will get added to Linux.

sjmulder 8y ago

On Windows that's VirtualAlloc with MEM_RESERVE: Reserves a range of the process's virtual address space without allocating any actual physical storage in memory or in the paging file on disk. https://msdn.microsoft.com/en-us/library/Aa366887(v=VS.85).aspx https://msdn.microsoft.com/en-us/library/Aa366887(v=VS.85).a... Perhaps mmap() can achieve the same on Unices?

cyphar 8y ago
This is what Linux does by default for anonymous memory (it's called overcommitment). Go makes very liberal use of this, but it has other issues (namely Linux will kill a process if it goes over a certain overcommitment threshold).
- scott_s 8y ago
  I'm not sure overcommitment is the same thing: https://www.kernel.org/doc/Documentation/vm/overcommit-accounting https://www.kernel.org/doc/Documentation/vm/overcommit-accou... Reading that, I think overcommitment is to determine the kernel's behavior when you try to allocate more virtual memory than physical memory that is present on the system. That is a different (but related) concern from the fact that mmap() will allocate virtual memory but not physical memory. (That is, mmap() reserves locations in the address space, but you don't have any physical memory backing it until you use that memory.)
  - cyphar 8y ago
    Yeah, sorry -- mmap()'s lazy allocation of pages is a separate but related concept to overcommit (I was writing a tirade about that in a separate thread and my wires got crossed).
scott_s 8y ago
My first thought on unix was mmap() as well, but one of the requirements was that it doesn't "look like" the process is using that memory, which I interpret as "I don't want to allocate this as virtual memory." I don't think mmap() allows that, as its entire purpose is to allocate virtual memory and return a pointer to that memory space. Personally, my feeling is: why does it matter if it "looks like" a process is using a lot of memory? That is, why does it matter if a process allocates a lot of virtual memory up front? It's not consuming physical memory, it's just updating some bookkeeping in the kernel. I know people feel uneasy about seeing large values for virtual memory, but... they shouldn't.

boomlinde 8y ago

> The ideal for me would be a function that marked some memory as "these addresses are taken, do not give them out to malloc, or anything else", but which still required me to actually "ask" for the memory before using it, so it didn't look like I was using 64gb of memory at startup. Is that possible? So you want to reserve memory, basically the way malloc does, depriving any other process of that memory, but you then want to reserve it again, and also somehow hide that the memory has been reserved? What exactly is the benefit except making it not look like you're using the memory? Actually, what's the benefit of that in itself?

twtw 8y ago
No, he wants to separate virtual address space allocation from physical memory backing allocation, with both still happening manually. It's a very reasonable request, and the benefit is that you can be guaranteed to have the address space while still only using a smaller max amount of physical memory.
- scott_s 8y ago
  I don't interpret that as the request. I interpret it as wanting virtual memory "allocated" but not counted against the processes' virtual memory allocation, and then getting a secondary mechanism to allocate the sorta-but-not-quite-allocated memory. I also don't see the value in that. (I don't see any request for different physical memory allocation, so I assume that would still be handled by page faulting in the kernel.)
  - koala_man 8y ago
    The value would be to be able to grow arrays and similar without relocation. People do it all the time in real life, with area codes, zip codes, case numbers, etc. For example, a long street full of strip malls in California will often have street numbers 50 apart to allow them to remain sequential even after new developments. No one would complain that this is a wasteful use of precious street numbers, or that it deprives other streets of those numbers. Imagine the nightmare if you periodically had to renumber all the buildings instead, like computers routinely have to.

kccqzy 8y ago

On FreeBSD there's MAP_GUARD. From the documentation: > Instead of a mapping, create a guard of the specified size. Guards allow a process to create reservations in its address space, which can later be replaced by actual mappings. mmap will not create mappings in the address range of a guard unless the request specifies MAP_FIXED. Guards can be destroyed with munmap(2). Any memory access by a thread to the guarded range results in the delivery of a SIGSEGV signal to that thread. So it seems like a good fit for your use case, but I've never used this.

CJefferson 8y ago
That is exactly what I want! I wonder if it will get added to Linux.