16 ms·

I see, I rarely use lower level stuff like this, so don't encounter it much, but, when I have to, I would get deallocation errors when dealing with things like

I see, I rarely use lower level stuff like this, so don't encounter it much, but, when I have to, I would get deallocation errors when dealing with things like metal buffers and just assumed the system always took care to protect the memory.

klodolph 4y ago
It can be a bit mysterious until you look at how it works. Most CPUs provide memory permissions with granularity of pages, which are usually 4 KB in size. The OS manages these by modifying a "page table" which the CPU uses. Changing the page table is not free, so the userland allocator (malloc/free) will request large blocks of memory from the OS (with mmap) (or sbrk, if you're a dinosaur) and then divide them up into smaller blocks for your program. They typically remain accessible until the entire original block is freed. If you allocate a large enough block of memory, it will typically just be backed by one region of pages made with mmap. Then, when you free it, the allocator will munmap it. You can test this out by allocating a large block of memory, freeing it, and then dereferencing it. Your system might do this for 1 MiB allocations, for example... change the allocation size and see whether it still crashes. #include <stdio.h> #include <stdlib.h> int main(int argc, char **argv) { volatile int *ptr = malloc(1024 * 1024); puts("*ptr = 1"); *ptr = 1; free(ptr); puts("*ptr = 2"); *ptr = 2; }
- astrange 4y ago
  Pages are 16KB on iOS (and AS Macs). They're 4KB everywhere else on the world, which is too small these days, although there's things like "superpages" to help with that.
  - cesarb 4y ago
    > They're 4KB everywhere else on the world They're 64KB on RHEL/CentOS 8 on 64-bit ARM, see for instance https://git.centos.org/rpms/kernel/blob/c8/f/SOURCES/kernel-aarch64.config https://git.centos.org/rpms/kernel/blob/c8/f/SOURCES/kernel-... which is AFAIK the kernel configuration for CentOS 8 (search for CONFIG_ARM64_64K_PAGES=y).
- ncmncm 4y ago
  Malloc will not unmap memory. Period. That would have huge performance cost in multi-thread programs.
  - klodolph 4y ago
    This is factually false, and I'm wondering why you'd say that. Some of the more common malloc implementations out there are dlmalloc, jemalloc, and tcmalloc. Here is where they call munmap. https://github.com/ennorehling/dlmalloc/blob/71296436f979a350870e10b869ccfd28bfcc17e4/malloc.c#L3859 https://github.com/ennorehling/dlmalloc/blob/71296436f979a35... https://github.com/jemalloc/jemalloc/blob/c0c9783ec9289e6d1de749ff20081af65bdd78b8/src/pages.c#L233 https://github.com/jemalloc/jemalloc/blob/c0c9783ec9289e6d1d... Note that not all allocators work this way, you can see that tcmalloc will instead MADV_DONTNEED. (You can find munmap in tcmalloc, but it's not being used to free memory.) https://github.com/google/tcmalloc/blob/48808681a9f38e974b49619a126f6f949ff59520/tcmalloc/system-alloc.cc#L411 https://github.com/google/tcmalloc/blob/48808681a9f38e974b49...
    - ncmncm 4y ago
      I am corrected. Now I need to start turning on nonstandard flags to disable it.

klodolph 4y ago

It can be a bit mysterious until you look at how it works. Most CPUs provide memory permissions with granularity of pages, which are usually 4 KB in size. The OS manages these by modifying a "page table" which the CPU uses. Changing the page table is not free, so the userland allocator (malloc/free) will request large blocks of memory from the OS (with mmap) (or sbrk, if you're a dinosaur) and then divide them up into smaller blocks for your program. They typically remain accessible until the entire original block is freed. If you allocate a large enough block of memory, it will typically just be backed by one region of pages made with mmap. Then, when you free it, the allocator will munmap it. You can test this out by allocating a large block of memory, freeing it, and then dereferencing it. Your system might do this for 1 MiB allocations, for example... change the allocation size and see whether it still crashes. #include <stdio.h> #include <stdlib.h> int main(int argc, char **argv) { volatile int *ptr = malloc(1024 * 1024); puts("*ptr = 1"); *ptr = 1; free(ptr); puts("*ptr = 2"); *ptr = 2; }

astrange 4y ago
Pages are 16KB on iOS (and AS Macs). They're 4KB everywhere else on the world, which is too small these days, although there's things like "superpages" to help with that.
- cesarb 4y ago
  > They're 4KB everywhere else on the world They're 64KB on RHEL/CentOS 8 on 64-bit ARM, see for instance https://git.centos.org/rpms/kernel/blob/c8/f/SOURCES/kernel-aarch64.config https://git.centos.org/rpms/kernel/blob/c8/f/SOURCES/kernel-... which is AFAIK the kernel configuration for CentOS 8 (search for CONFIG_ARM64_64K_PAGES=y).
ncmncm 4y ago
Malloc will not unmap memory. Period. That would have huge performance cost in multi-thread programs.
- klodolph 4y ago
  This is factually false, and I'm wondering why you'd say that. Some of the more common malloc implementations out there are dlmalloc, jemalloc, and tcmalloc. Here is where they call munmap. https://github.com/ennorehling/dlmalloc/blob/71296436f979a350870e10b869ccfd28bfcc17e4/malloc.c#L3859 https://github.com/ennorehling/dlmalloc/blob/71296436f979a35... https://github.com/jemalloc/jemalloc/blob/c0c9783ec9289e6d1de749ff20081af65bdd78b8/src/pages.c#L233 https://github.com/jemalloc/jemalloc/blob/c0c9783ec9289e6d1d... Note that not all allocators work this way, you can see that tcmalloc will instead MADV_DONTNEED. (You can find munmap in tcmalloc, but it's not being used to free memory.) https://github.com/google/tcmalloc/blob/48808681a9f38e974b49619a126f6f949ff59520/tcmalloc/system-alloc.cc#L411 https://github.com/google/tcmalloc/blob/48808681a9f38e974b49...
  - ncmncm 4y ago
    I am corrected. Now I need to start turning on nonstandard flags to disable it.