4 ms·

> The ideal for me would be a function that marked some memory as "these addresses are taken, do not give them out to malloc, or anything else", but which still

> The ideal for me would be a function that marked some memory as "these addresses are taken, do not give them out to malloc, or anything else", but which still required me to actually "ask" for the memory before using it, so it didn't look like I was using 64gb of memory at startup. Is that possible? So you want to reserve memory, basically the way malloc does, depriving any other process of that memory, but you then want to reserve it again, and also somehow hide that the memory has been reserved? What exactly is the benefit except making it not look like you're using the memory? Actually, what's the benefit of that in itself?

twtw 8y ago
No, he wants to separate virtual address space allocation from physical memory backing allocation, with both still happening manually. It's a very reasonable request, and the benefit is that you can be guaranteed to have the address space while still only using a smaller max amount of physical memory.
- scott_s 8y ago
  I don't interpret that as the request. I interpret it as wanting virtual memory "allocated" but not counted against the processes' virtual memory allocation, and then getting a secondary mechanism to allocate the sorta-but-not-quite-allocated memory. I also don't see the value in that. (I don't see any request for different physical memory allocation, so I assume that would still be handled by page faulting in the kernel.)
  - koala_man 8y ago
    The value would be to be able to grow arrays and similar without relocation. People do it all the time in real life, with area codes, zip codes, case numbers, etc. For example, a long street full of strip malls in California will often have street numbers 50 apart to allow them to remain sequential even after new developments. No one would complain that this is a wasteful use of precious street numbers, or that it deprives other streets of those numbers. Imagine the nightmare if you periodically had to renumber all the buildings instead, like computers routinely have to.
    - deleted 8y ago
      [deleted]
    - scott_s 8y ago
      How would this proposed mechanism allow that, and what about the current workings of virtual memory disallow it? One can already do this with realloc() (https://linux.die.net/man/3/realloc https://linux.die.net/man/3/realloc). And if you don't want to use the malloc family, but instead want to manage your virtual memory yourself, you can easily allocate large sections of virtual memory and then manage it yourself - which would include enabling behavior such as growing arrays. (Even though you're really just re-implementing realloc-like behavior yourself.) And to make sure we're on the same page (ha), I want to reiterate that there is a big difference between virtual and physical memory. You can allocate virtual memory without allocating physical memory.
      - CJefferson 8y ago
        This is kind of the point of this whole post :) reserving lots of virtual memory causes posts like this, where people complain about how much memory you are "using".
        scott_s 8y ago
        I think that just kicks the ball further down the court, as it's yet another thing to track. It think it's better to just explain that (for the most part), high virtual memory usage isn't a problem. But as someone pointed out above, FreeBSD does have a facility for it, which is interesting.
      - koala_man 8y ago
        I misunderstood what it was that you didn't see the value of. The original point was entirely to avoid scaring newbies who can't tell virtual from physical, while still being liberal with your virtual usage. Haskell with GHC has a great solution: just allocate 1TB up front. A newbie does not need improved tooling, a reworked memory system or any education to realize that this can't possibly be RAM.
CJefferson 8y ago
No, I want, in my address space, to say to the kernel "when you decide what bit of my memory map to return from mmap and friends, don't use these addresses.". That shouldn't cost anything, except storing the blocks I want reserving. This is surprisingly hard to do in user space. You can tell mmap where to put allocations, but not where not to put them. Also it's hard to control what your Libc's malloc will do.
- klodolph 8y ago
  You can do this with mmap(). Just use PROT_NONE, you'll get a mapping but you can't read, write, or execute it. Sure, it will show up in the "virtual memory" usage of your process. That's just how the virtual memory accounting works.
hinkley 8y ago
What operating system are you working on that has a shared address space between multiple unrelated processes? Nobody that I know of works that way besides some L4 kernels and they only use it as an IPC optimization strategy.
- boomlinde 8y ago
  What magic computer are you using where reserving a chunk of memory doesn't necessarily mean that no other process can reserve that chunk of memory?
  - foldr 8y ago
    The process isn't reserving memory, it's reserving ranges in its virtual address space: https://en.wikipedia.org/wiki/Virtual_address_space https://en.wikipedia.org/wiki/Virtual_address_space
    - boomlinde 8y ago
      So like allocating a large virtual address space and managing that on your own? That already exists, then, and the only remaining issue is making it look like you haven't, for some unclear reason.
      - CJefferson 8y ago
        The reason is the thread you are currently in :) it is common for people to complain you are "using the memory".
  - hinkley 8y ago
    Each process has its own virtual address space. That’s half the fun of virtual addressing. (and unfortunately, at least according to the L4 people, almost half the cost of a context switch - fixing the TLB)

twtw 8y ago

No, he wants to separate virtual address space allocation from physical memory backing allocation, with both still happening manually. It's a very reasonable request, and the benefit is that you can be guaranteed to have the address space while still only using a smaller max amount of physical memory.

scott_s 8y ago
I don't interpret that as the request. I interpret it as wanting virtual memory "allocated" but not counted against the processes' virtual memory allocation, and then getting a secondary mechanism to allocate the sorta-but-not-quite-allocated memory. I also don't see the value in that. (I don't see any request for different physical memory allocation, so I assume that would still be handled by page faulting in the kernel.)
- koala_man 8y ago
  The value would be to be able to grow arrays and similar without relocation. People do it all the time in real life, with area codes, zip codes, case numbers, etc. For example, a long street full of strip malls in California will often have street numbers 50 apart to allow them to remain sequential even after new developments. No one would complain that this is a wasteful use of precious street numbers, or that it deprives other streets of those numbers. Imagine the nightmare if you periodically had to renumber all the buildings instead, like computers routinely have to.
  - deleted 8y ago
    [deleted]
  - scott_s 8y ago
    How would this proposed mechanism allow that, and what about the current workings of virtual memory disallow it? One can already do this with realloc() (https://linux.die.net/man/3/realloc https://linux.die.net/man/3/realloc). And if you don't want to use the malloc family, but instead want to manage your virtual memory yourself, you can easily allocate large sections of virtual memory and then manage it yourself - which would include enabling behavior such as growing arrays. (Even though you're really just re-implementing realloc-like behavior yourself.) And to make sure we're on the same page (ha), I want to reiterate that there is a big difference between virtual and physical memory. You can allocate virtual memory without allocating physical memory.
    - CJefferson 8y ago
      This is kind of the point of this whole post :) reserving lots of virtual memory causes posts like this, where people complain about how much memory you are "using".
      - scott_s 8y ago
        I think that just kicks the ball further down the court, as it's yet another thing to track. It think it's better to just explain that (for the most part), high virtual memory usage isn't a problem. But as someone pointed out above, FreeBSD does have a facility for it, which is interesting.
    - koala_man 8y ago
      I misunderstood what it was that you didn't see the value of. The original point was entirely to avoid scaring newbies who can't tell virtual from physical, while still being liberal with your virtual usage. Haskell with GHC has a great solution: just allocate 1TB up front. A newbie does not need improved tooling, a reworked memory system or any education to realize that this can't possibly be RAM.

CJefferson 8y ago

No, I want, in my address space, to say to the kernel "when you decide what bit of my memory map to return from mmap and friends, don't use these addresses.". That shouldn't cost anything, except storing the blocks I want reserving. This is surprisingly hard to do in user space. You can tell mmap where to put allocations, but not where not to put them. Also it's hard to control what your Libc's malloc will do.

klodolph 8y ago
You can do this with mmap(). Just use PROT_NONE, you'll get a mapping but you can't read, write, or execute it. Sure, it will show up in the "virtual memory" usage of your process. That's just how the virtual memory accounting works.

hinkley 8y ago

What operating system are you working on that has a shared address space between multiple unrelated processes? Nobody that I know of works that way besides some L4 kernels and they only use it as an IPC optimization strategy.

boomlinde 8y ago
What magic computer are you using where reserving a chunk of memory doesn't necessarily mean that no other process can reserve that chunk of memory?
- foldr 8y ago
  The process isn't reserving memory, it's reserving ranges in its virtual address space: https://en.wikipedia.org/wiki/Virtual_address_space https://en.wikipedia.org/wiki/Virtual_address_space
  - boomlinde 8y ago
    So like allocating a large virtual address space and managing that on your own? That already exists, then, and the only remaining issue is making it look like you haven't, for some unclear reason.
    - CJefferson 8y ago
      The reason is the thread you are currently in :) it is common for people to complain you are "using the memory".
- hinkley 8y ago
  Each process has its own virtual address space. That’s half the fun of virtual addressing. (and unfortunately, at least according to the L4 people, almost half the cost of a context switch - fixing the TLB)