20 ms·

> * Memory allocation functions malloc, calloc, realloc, reallocarray, valloc, pvalloc, memalign, and posix_memalign fail now with total object size larger than

> * Memory allocation functions malloc, calloc, realloc, reallocarray, valloc, pvalloc, memalign, and posix_memalign fail now with total object size larger than PTRDIFF_MAX. This is to avoid potential undefined behavior with pointer subtraction within the allocated object, where results might overflow the ptrdiff_t type. I did not think they would take this decision so soon, but it is, in my opinion, the right decision to take. There will be complaints from users of memory-heavy programs running on 32-bit platforms though. For context, this blog post shows how things break when allocation functions are allowed to create blocks of more than PTRDIFF_MAX: https://trust-in-soft.com/objects-larger-than-ptrdiff_max-bytes/ https://trust-in-soft.com/objects-larger-than-ptrdiff_max-by...

jabl 7y ago
Absolutely, this is the right thing to do. If one needs to allocate more than 2 GB in a single allocation, time to move to 64-bit.
- Iwan-Zotow 7y ago
  > If one needs to allocate more than 2 GB in a single allocation, time to move to 64-bit. time to move to (anonymus) mmap - it still has size_t as type for the size argument
hoseja 7y ago
That first clang listing was hilarious.
jandrese 7y ago
> There will be complaints from users of memory-heavy programs running on 32-bit platforms though. I would think this would be a small one of many complaints such users would have on a daily basis.
ajross 7y ago
> There will be complaints from users of memory-heavy programs running on 32-bit platforms though In all of recorded history, has a malloc() call for more than 2GB ever actually succeeded anywhere? Most OSes on such platforms never supported any more than that amount of addressible memory in a user process at all. This is fine. Honestly it's seems like mostly pedantry on modern systems, but it's clearly correct.
- pkaye 7y ago
  I've done about 64GB in Go but I think it uses mmap.
  - ajross 7y ago
    On a 64 bit platform, sure, that's inside of PTRDIFF_MAX, so it's legal and supported.
- duskwuff 7y ago
  > Most OSes on such platforms never supported any more than that amount of addressible memory in a user process at all. And even when they do, it's going to be very rare to find 2GB of contiguous unused address space. With the default executable mapping low in the first GB and the stack right below 3GB, a single library mapped around the 2GB mark can fragment the address space enough to make a 2GB allocation impossible.
- pascal_cuoq 7y ago
  > In all of recorded history, has a malloc() call for more than 2GB ever actually succeeded anywhere? Yes, on OS X 10.5, and on 32-bit Linux with Glibc until two days ago. The article I linked, written before Glibc 2.30 was released, is from a period when every Unix had been allowing “malloc(0x80000001);” in 32-bit processes until recently; only OS X had had the courage to make that allocation fail. Sorry if the article doesn't make it clear enough that this is the context it is written in, but in its defense, you only needed to try it (and still need today to try it if you didn't upgrade Glibc) to see that it succeeds. Or do you think that the Glibc developers wrote a Changelog entry to explain that they changed something that didn't actually change? Linux's default limit on 32-bit has been 3GiB for a while, i think: https://stackoverflow.com/a/5080778/139746 https://stackoverflow.com/a/5080778/139746 Windows's limit is 2GiB by default, but this is only a default and 32-bit processes can be allowed access to more memory, up to IIRC nearly all of the theoretical maximum 4GiB for 32-bit processes running on 64-bit Windows.
  - ajross 7y ago
    The (sarcastic) point was about the fact that no real world code actually relied on a malloc() of half the address space. I'm sure it "worked" in some sense, though I'd be really surprised if you could make that happen with a default-linked C program on any distro that ever shipped. The holes just aren't big enough. You'd need to link the app with special care, and potentially write your own dynamic loader to keep the region you wanted free. And if you do that... you might as well just mmap() the thing. The point was that doing this with the system heap on a 32 bit system was never a serious thing. There are apps that would do management of memory spaces that large, but they didn't do it with malloc.

jabl 7y ago

Absolutely, this is the right thing to do. If one needs to allocate more than 2 GB in a single allocation, time to move to 64-bit.

Iwan-Zotow 7y ago
> If one needs to allocate more than 2 GB in a single allocation, time to move to 64-bit. time to move to (anonymus) mmap - it still has size_t as type for the size argument

hoseja 7y ago

That first clang listing was hilarious.

jandrese 7y ago

> There will be complaints from users of memory-heavy programs running on 32-bit platforms though. I would think this would be a small one of many complaints such users would have on a daily basis.

ajross 7y ago

> There will be complaints from users of memory-heavy programs running on 32-bit platforms though In all of recorded history, has a malloc() call for more than 2GB ever actually succeeded anywhere? Most OSes on such platforms never supported any more than that amount of addressible memory in a user process at all. This is fine. Honestly it's seems like mostly pedantry on modern systems, but it's clearly correct.

pkaye 7y ago
I've done about 64GB in Go but I think it uses mmap.
- ajross 7y ago
  On a 64 bit platform, sure, that's inside of PTRDIFF_MAX, so it's legal and supported.
duskwuff 7y ago
> Most OSes on such platforms never supported any more than that amount of addressible memory in a user process at all. And even when they do, it's going to be very rare to find 2GB of contiguous unused address space. With the default executable mapping low in the first GB and the stack right below 3GB, a single library mapped around the 2GB mark can fragment the address space enough to make a 2GB allocation impossible.
pascal_cuoq 7y ago
> In all of recorded history, has a malloc() call for more than 2GB ever actually succeeded anywhere? Yes, on OS X 10.5, and on 32-bit Linux with Glibc until two days ago. The article I linked, written before Glibc 2.30 was released, is from a period when every Unix had been allowing “malloc(0x80000001);” in 32-bit processes until recently; only OS X had had the courage to make that allocation fail. Sorry if the article doesn't make it clear enough that this is the context it is written in, but in its defense, you only needed to try it (and still need today to try it if you didn't upgrade Glibc) to see that it succeeds. Or do you think that the Glibc developers wrote a Changelog entry to explain that they changed something that didn't actually change? Linux's default limit on 32-bit has been 3GiB for a while, i think: https://stackoverflow.com/a/5080778/139746 https://stackoverflow.com/a/5080778/139746 Windows's limit is 2GiB by default, but this is only a default and 32-bit processes can be allowed access to more memory, up to IIRC nearly all of the theoretical maximum 4GiB for 32-bit processes running on 64-bit Windows.
- ajross 7y ago
  The (sarcastic) point was about the fact that no real world code actually relied on a malloc() of half the address space. I'm sure it "worked" in some sense, though I'd be really surprised if you could make that happen with a default-linked C program on any distro that ever shipped. The holes just aren't big enough. You'd need to link the app with special care, and potentially write your own dynamic loader to keep the region you wanted free. And if you do that... you might as well just mmap() the thing. The point was that doing this with the system heap on a 32 bit system was never a serious thing. There are apps that would do management of memory spaces that large, but they didn't do it with malloc.