4 ms·

I thought accessing deallocated memory would crash an app anyway?

323 4y ago
You can think of two levels of deallocated memory: deallocated just inside the app, and fully deallocated towards the OS. Only the fully deallocated towards the OS will crash. When you deallocate small amounts of memory, the underlying C library allocator will keep the deallocated memory around and not return it to the OS in case you need a similar sized piece of memory soon.
- ncmncm 4y ago
  Not correct. It is very rare for memory ever to be freed to the OS before process exit. Malloc never does. And, zeroing the memory, if code later uses it like a pointer, that will be null and the use will trap.
  - saagarjha 4y ago
    You’re the one who’s not correct. malloc is more than willing to hand memory back to the OS when feasible, most implementations will unmap the memory or at least mark it as unneeded.
    - ncmncm 4y ago
      I am corrected. Eek.
yakubin 4y ago
That's true for memory which has been unmapped from the process (as long as something else isn't mapped to the same virtual address range). But different implementations of C malloc()+calloc()+free() don't necessarily unmap memory when you call free(). Instead they mark it as something that may be returned from subsequent calls to malloc()/calloc(), i.e. the memory isn't returned to the OS.
- ncmncm 4y ago
  None of them unmap freed memory.
  - yakubin 4y ago
    That depends on the specific implementation. glibc does unmap memory[1]. Not in every free() call, but still. There are also implementations which unmap in every free(), which are used to quickly detect bugs. Honestly, I wouldn't want to use an implementation which never unmaps memory. I've already been irritated by Go's reluctance to return memory to the OS in the past. I'm certainly glad it's less of a problem in C. (Although I've read some reports of glibc being a bit suboptimal here in multithreaded programs, compared to allocators like tcmalloc or jemalloc.) [1]: <https://github.com/bminor/glibc/blob/a364a3a7090b82ddd30e9209df2af56e781d51e4/malloc/malloc.c#L3355 https://github.com/bminor/glibc/blob/a364a3a7090b82ddd30e920...>
    - ncmncm 4y ago
      It won't do that by default. You need to set some non-portable flags to get it to munmap things. Unmapping trashes page-map caches of other threads in the process. That involves inter-core interrupts. Not pretty.
      - yakubin 4y ago
        Example program: #include <stdlib.h> int main() { for (size_t i = 0; i < 4096*4096; i++) { void* p = malloc(i * 4096); free(p); } } Running strace on it gives me a whole spam of logs like: mmap(NULL, 139264, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f0e7d46f000 munmap(0x7f0e7d46f000, 139264) = 0 mmap(NULL, 143360, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f0e7d46e000 munmap(0x7f0e7d46e000, 143360) = 0 mmap(NULL, 147456, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f0e7d46d000 munmap(0x7f0e7d46d000, 147456) = 0 mmap(NULL, 151552, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f0e7d46c000 munmap(0x7f0e7d46c000, 151552) = 0 mmap(NULL, 155648, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f0e7d46b000 munmap(0x7f0e7d46b000, 155648) = 0 mmap(NULL, 159744, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f0e7d46a000 munmap(0x7f0e7d46a000, 159744) = 0 That was run under a standard Debian 11 installation.
secretsatan 4y ago
I see, I rarely use lower level stuff like this, so don't encounter it much, but, when I have to, I would get deallocation errors when dealing with things like metal buffers and just assumed the system always took care to protect the memory.
- klodolph 4y ago
  It can be a bit mysterious until you look at how it works. Most CPUs provide memory permissions with granularity of pages, which are usually 4 KB in size. The OS manages these by modifying a "page table" which the CPU uses. Changing the page table is not free, so the userland allocator (malloc/free) will request large blocks of memory from the OS (with mmap) (or sbrk, if you're a dinosaur) and then divide them up into smaller blocks for your program. They typically remain accessible until the entire original block is freed. If you allocate a large enough block of memory, it will typically just be backed by one region of pages made with mmap. Then, when you free it, the allocator will munmap it. You can test this out by allocating a large block of memory, freeing it, and then dereferencing it. Your system might do this for 1 MiB allocations, for example... change the allocation size and see whether it still crashes. #include <stdio.h> #include <stdlib.h> int main(int argc, char **argv) { volatile int *ptr = malloc(1024 * 1024); puts("*ptr = 1"); *ptr = 1; free(ptr); puts("*ptr = 2"); *ptr = 2; }
  - astrange 4y ago
    Pages are 16KB on iOS (and AS Macs). They're 4KB everywhere else on the world, which is too small these days, although there's things like "superpages" to help with that.
    - cesarb 4y ago
      > They're 4KB everywhere else on the world They're 64KB on RHEL/CentOS 8 on 64-bit ARM, see for instance https://git.centos.org/rpms/kernel/blob/c8/f/SOURCES/kernel-aarch64.config https://git.centos.org/rpms/kernel/blob/c8/f/SOURCES/kernel-... which is AFAIK the kernel configuration for CentOS 8 (search for CONFIG_ARM64_64K_PAGES=y).
sumtechguy 4y ago
That depends on the applications CRT and compiler and how they are setting up their memory regions. Some CRT's just go get their own block of memory then malloc out of that. Some use the OS extensively to minimize that sort of thing. It just depends.

323 4y ago

You can think of two levels of deallocated memory: deallocated just inside the app, and fully deallocated towards the OS. Only the fully deallocated towards the OS will crash. When you deallocate small amounts of memory, the underlying C library allocator will keep the deallocated memory around and not return it to the OS in case you need a similar sized piece of memory soon.

ncmncm 4y ago
Not correct. It is very rare for memory ever to be freed to the OS before process exit. Malloc never does. And, zeroing the memory, if code later uses it like a pointer, that will be null and the use will trap.
- saagarjha 4y ago
  You’re the one who’s not correct. malloc is more than willing to hand memory back to the OS when feasible, most implementations will unmap the memory or at least mark it as unneeded.
  - ncmncm 4y ago
    I am corrected. Eek.

yakubin 4y ago

That's true for memory which has been unmapped from the process (as long as something else isn't mapped to the same virtual address range). But different implementations of C malloc()+calloc()+free() don't necessarily unmap memory when you call free(). Instead they mark it as something that may be returned from subsequent calls to malloc()/calloc(), i.e. the memory isn't returned to the OS.

ncmncm 4y ago
None of them unmap freed memory.
- yakubin 4y ago
  That depends on the specific implementation. glibc does unmap memory[1]. Not in every free() call, but still. There are also implementations which unmap in every free(), which are used to quickly detect bugs. Honestly, I wouldn't want to use an implementation which never unmaps memory. I've already been irritated by Go's reluctance to return memory to the OS in the past. I'm certainly glad it's less of a problem in C. (Although I've read some reports of glibc being a bit suboptimal here in multithreaded programs, compared to allocators like tcmalloc or jemalloc.) [1]: <https://github.com/bminor/glibc/blob/a364a3a7090b82ddd30e9209df2af56e781d51e4/malloc/malloc.c#L3355 https://github.com/bminor/glibc/blob/a364a3a7090b82ddd30e920...>
  - ncmncm 4y ago
    It won't do that by default. You need to set some non-portable flags to get it to munmap things. Unmapping trashes page-map caches of other threads in the process. That involves inter-core interrupts. Not pretty.
    - yakubin 4y ago
      Example program: #include <stdlib.h> int main() { for (size_t i = 0; i < 4096*4096; i++) { void* p = malloc(i * 4096); free(p); } } Running strace on it gives me a whole spam of logs like: mmap(NULL, 139264, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f0e7d46f000 munmap(0x7f0e7d46f000, 139264) = 0 mmap(NULL, 143360, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f0e7d46e000 munmap(0x7f0e7d46e000, 143360) = 0 mmap(NULL, 147456, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f0e7d46d000 munmap(0x7f0e7d46d000, 147456) = 0 mmap(NULL, 151552, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f0e7d46c000 munmap(0x7f0e7d46c000, 151552) = 0 mmap(NULL, 155648, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f0e7d46b000 munmap(0x7f0e7d46b000, 155648) = 0 mmap(NULL, 159744, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f0e7d46a000 munmap(0x7f0e7d46a000, 159744) = 0 That was run under a standard Debian 11 installation.

secretsatan 4y ago

I see, I rarely use lower level stuff like this, so don't encounter it much, but, when I have to, I would get deallocation errors when dealing with things like metal buffers and just assumed the system always took care to protect the memory.

klodolph 4y ago
It can be a bit mysterious until you look at how it works. Most CPUs provide memory permissions with granularity of pages, which are usually 4 KB in size. The OS manages these by modifying a "page table" which the CPU uses. Changing the page table is not free, so the userland allocator (malloc/free) will request large blocks of memory from the OS (with mmap) (or sbrk, if you're a dinosaur) and then divide them up into smaller blocks for your program. They typically remain accessible until the entire original block is freed. If you allocate a large enough block of memory, it will typically just be backed by one region of pages made with mmap. Then, when you free it, the allocator will munmap it. You can test this out by allocating a large block of memory, freeing it, and then dereferencing it. Your system might do this for 1 MiB allocations, for example... change the allocation size and see whether it still crashes. #include <stdio.h> #include <stdlib.h> int main(int argc, char **argv) { volatile int *ptr = malloc(1024 * 1024); puts("*ptr = 1"); *ptr = 1; free(ptr); puts("*ptr = 2"); *ptr = 2; }
- astrange 4y ago
  Pages are 16KB on iOS (and AS Macs). They're 4KB everywhere else on the world, which is too small these days, although there's things like "superpages" to help with that.
  - cesarb 4y ago
    > They're 4KB everywhere else on the world They're 64KB on RHEL/CentOS 8 on 64-bit ARM, see for instance https://git.centos.org/rpms/kernel/blob/c8/f/SOURCES/kernel-aarch64.config https://git.centos.org/rpms/kernel/blob/c8/f/SOURCES/kernel-... which is AFAIK the kernel configuration for CentOS 8 (search for CONFIG_ARM64_64K_PAGES=y).

sumtechguy 4y ago

That depends on the applications CRT and compiler and how they are setting up their memory regions. Some CRT's just go get their own block of memory then malloc out of that. Some use the OS extensively to minimize that sort of thing. It just depends.