3 ms·

That will actually allocate 4GB to your process, EDIT: OS dependent. This just reserves 4GB of virtual memory addresses.

by _4r6j 5y ago

judofyr 5y ago
I’ll have to admit that I don’t fully know how these things work, but my impression was that a malloc(4GB) on most systems will not actually allocate 4GB, but only acquire virtual memory and the OS will give it memory when it’s being used. Is this too simplistic and this package is doing something else?
- woodruffw 5y ago
  I haven't looked at this implementation yet, but your understanding is correct: reserving 4GB up-front with `std::vector` should only acquire virtual memory and not require an actual commitment, unless `std::vector` is doing something nuts internally like default-constructing each element that's been reserved. But I don't believe it does that.
  - tom_ 5y ago
    I think you've got two sensible options in this case: commit, or do nothing. The standard's wording accommodates both: it's valid for a reserve to be a no-op, a situation the caller can detect (if they want to - assuming they even imagined anybody would ever do this! - I certainly never did, until now) by checking the new capacity. It's not obvious to me that it buys you much to reserve without committing. You're taking the hint, then ignoring it. Why bother? Why not just do nothing?r
jeffbee 5y ago
That is incorrect. If I call std::vector<int>::reserve(1<<30) on Linux w/ GNU standard library, using the GNU default allocator, memory is lazily allocated: $ ./1 allocating 4294967296 bytes ^Z [1]+ Stopped ./1 $ grep -E VmSize\|RssAnon /proc/$!/status VmSize: 4199804 kB RssAnon: 144 kB
- _4r6j 5y ago
  Yeah i verified the same thing on Linux. I will update the README with that.
- edflsafoiewq 5y ago
  Whether this works depends on several things, like /proc/sys/vm/overcommit_memory and how much RAM+swap you actually have available.
  - jeffbee 5y ago
    No, this is the behavior for any value of vm.overcommit_memory.
    - edflsafoiewq 5y ago
      ~ $ cat a.cpp #include <vector> int main() { std::vector<int> v; v.reserve(1ULL<<32); } ~ $ g++ a.cpp ~ $ sudo sh -c 'echo 0 > /proc/sys/vm/overcommit_memory' ~ $ ./a.out terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc Aborted (core dumped) ~ $ sudo sh -c 'echo 1 > /proc/sys/vm/overcommit_memory' ~ $ ./a.out ~ $
      - jeffbee 5y ago
        I see, thanks. It didn't seem to matter on my box, but perhaps that's what "heuristic overcommit" means: outcomes may vary?
blt 5y ago
On systems with overcommit, how would the resulting sequence of MMU states be different?

That will actually allocate 4GB to your process, EDIT: OS dependent. This just reserves 4GB of virtual memory addresses.

by _4r6j 5y ago

That will actually allocate 4GB to your process, EDIT: OS dependent. This just reserves 4GB of virtual memory addresses.

judofyr 5y ago

I’ll have to admit that I don’t fully know how these things work, but my impression was that a malloc(4GB) on most systems will not actually allocate 4GB, but only acquire virtual memory and the OS will give it memory when it’s being used. Is this too simplistic and this package is doing something else?

woodruffw 5y ago
I haven't looked at this implementation yet, but your understanding is correct: reserving 4GB up-front with `std::vector` should only acquire virtual memory and not require an actual commitment, unless `std::vector` is doing something nuts internally like default-constructing each element that's been reserved. But I don't believe it does that.
- tom_ 5y ago
  I think you've got two sensible options in this case: commit, or do nothing. The standard's wording accommodates both: it's valid for a reserve to be a no-op, a situation the caller can detect (if they want to - assuming they even imagined anybody would ever do this! - I certainly never did, until now) by checking the new capacity. It's not obvious to me that it buys you much to reserve without committing. You're taking the hint, then ignoring it. Why bother? Why not just do nothing?r

jeffbee 5y ago

That is incorrect. If I call std::vector<int>::reserve(1<<30) on Linux w/ GNU standard library, using the GNU default allocator, memory is lazily allocated: $ ./1 allocating 4294967296 bytes ^Z [1]+ Stopped ./1 $ grep -E VmSize\|RssAnon /proc/$!/status VmSize: 4199804 kB RssAnon: 144 kB

_4r6j 5y ago
Yeah i verified the same thing on Linux. I will update the README with that.
edflsafoiewq 5y ago
Whether this works depends on several things, like /proc/sys/vm/overcommit_memory and how much RAM+swap you actually have available.
- jeffbee 5y ago
  No, this is the behavior for any value of vm.overcommit_memory.
  - edflsafoiewq 5y ago
    ~ $ cat a.cpp #include <vector> int main() { std::vector<int> v; v.reserve(1ULL<<32); } ~ $ g++ a.cpp ~ $ sudo sh -c 'echo 0 > /proc/sys/vm/overcommit_memory' ~ $ ./a.out terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc Aborted (core dumped) ~ $ sudo sh -c 'echo 1 > /proc/sys/vm/overcommit_memory' ~ $ ./a.out ~ $
    - jeffbee 5y ago
      I see, thanks. It didn't seem to matter on my box, but perhaps that's what "heuristic overcommit" means: outcomes may vary?

blt 5y ago

On systems with overcommit, how would the resulting sequence of MMU states be different?