4 ms·

you mean `mmap`? Also yes I'm radical enough to suggest the 50+ year old C standard library API is bad. Which isn't really a radical position >.>

shakna 8y ago
Is your irritation aimed at the implementations of malloc in the standard compilers? Or all malloc implementations? (e.g. Hoard [0]). [0] https://github.com/emeryberger/Hoard https://github.com/emeryberger/Hoard
- valarauca1 8y ago
  my irritation is C's primitive type system. And programmers willingness to conflate their standard library with system level operations.
deathanatos 8y ago
mmap and malloc do very different things. You didn't really clarify why you thought malloc() was bad, or why you think someone should call mmap(), so I'm mostly left to guess. mmap, for example, is free to require that the requested range be a multiple of a page; trying to use it as if it were malloc() would, on such systems, cause any allocations less than a page to require a whole page. And since this is a rather common case, you'd be using a ton more memory. malloc(), however, can subdivide a large allocation from the OS into small allocations for the process; it is my understanding that most modern malloc()s will do this, in some form. (It is also my understanding that some malloc() implementations will use mmap() for allocating memory from the OS…)
- valarauca1 8y ago
  `malloc` is part of the standard library. the underlying implementation is environmental specific, or implementation specific depending on the host application. Generally speaking `mmap` is the system call that * Allocates memory * Moves allocations around `malloc` is the common entry point into a library that attempts to ensure you invoke the `mmap` system call as little as possible. trying to use it as if it were malloc() would, on such systems, cause any allocations less than a page to require a whole page. And since this is a rather common case, you'd be using a ton more memory. This is a misconception [1]. Malloc is governed by hardware limitations. If you ask for 50bytes, it'll give you a full 4KiB page, because hardware can only segment memory in 4KiB chunks. Actually some allocators will attempt to _use_ those non-allocated regions of the remaining ~4046 bytes for other allocations in order to be faster (less systemcalls), this is why sometimes writing past buffer ends can give you garbage data instead of segfaults. [1] https://stackoverflow.com/questions/45458732/how-are-memory-allocations-smaller-than-page-size-handled-in-linux https://stackoverflow.com/questions/45458732/how-are-memory-...
  - deathanatos 8y ago
    The link you post is about mmap; the section you quote is about malloc. The link is correct — about mmap; but that limitation does not apply to malloc. The link precisely backs up the part you're quoting, so saying "This is a misconception" makes no sense. > If you ask for 50bytes, it'll give you a full 4KiB page, because hardware can only segment memory in 4KiB chunks. No, not really. (But you seem to realize this, as you contradict this point in the next quote I'll pull.) A malloc implementation is free to divvy up a page in order to service several allocations. Just because the pointer it returns to you is somewhere within a page that must, almost by definition, be a full page is irrelevant to the point of this discussion (that malloc() is not needed b/c mmap exists, and is somehow superior). The point is that the allocation consumes, in the grand scale of things, an amount on par with the requested size. (There is some overhead, of course, but not nearly the same are the mmap-must-return-a-full-page overhead.) > Actually some allocators will attempt to _use_ those non-allocated regions of the remaining ~4046 bytes for other allocations in order to be faster (less systemcalls), this is why sometimes writing past buffer ends can give you garbage data instead of segfaults. That was part of the point I was making. You seem to have misunderstood my post. malloc() is free to divvy up a page (potentially handed to it by mmap) into several allocations. mmap() cannot. The point here is that replacing every malloc() call with a call to mmap() results in the loss of that ability to divvy up pages, requiring each allocation to be a full page; this would significantly bloat the memory consumed by an application, as small allocations would not share pages. Hence why those two calls are not equivalent, which is what the original post: > you mean `mmap`? > Also yes I'm radical enough to suggest the 50+ year old C standard library API is bad. Very much seemed to imply. Though, again, I asked for clarification on your point, to make sure I'm understanding it, and you again failed to provide it. So, again, > You didn't really clarify why you thought malloc() was bad, or why you think someone should call mmap() [in lieu of malloc()], so I'm mostly left to guess.
    - valarauca1 8y ago
      A malloc implementation is free to divvy up a page in order to service several allocations But like the hardware won't enforce that. So yeah you can malloc 50bytes, but the hardware is just going to assume there is a 4096 based page behind it. Because what memory can/cannot be written/read too is controlled by mmap -> linux -> hardware MMU. So when ever `malloc` gives you 50bytes without allocating an entire page it's just lying to you and giving you a pointer offset from another allocation. This isn't strictly being enforced by anything. There is zero contract that enforces what memory you can/cannot use.

shakna 8y ago

Is your irritation aimed at the implementations of malloc in the standard compilers? Or all malloc implementations? (e.g. Hoard [0]). [0] https://github.com/emeryberger/Hoard https://github.com/emeryberger/Hoard

valarauca1 8y ago
my irritation is C's primitive type system. And programmers willingness to conflate their standard library with system level operations.

deathanatos 8y ago

mmap and malloc do very different things. You didn't really clarify why you thought malloc() was bad, or why you think someone should call mmap(), so I'm mostly left to guess. mmap, for example, is free to require that the requested range be a multiple of a page; trying to use it as if it were malloc() would, on such systems, cause any allocations less than a page to require a whole page. And since this is a rather common case, you'd be using a ton more memory. malloc(), however, can subdivide a large allocation from the OS into small allocations for the process; it is my understanding that most modern malloc()s will do this, in some form. (It is also my understanding that some malloc() implementations will use mmap() for allocating memory from the OS…)

valarauca1 8y ago
`malloc` is part of the standard library. the underlying implementation is environmental specific, or implementation specific depending on the host application. Generally speaking `mmap` is the system call that * Allocates memory * Moves allocations around `malloc` is the common entry point into a library that attempts to ensure you invoke the `mmap` system call as little as possible. trying to use it as if it were malloc() would, on such systems, cause any allocations less than a page to require a whole page. And since this is a rather common case, you'd be using a ton more memory. This is a misconception [1]. Malloc is governed by hardware limitations. If you ask for 50bytes, it'll give you a full 4KiB page, because hardware can only segment memory in 4KiB chunks. Actually some allocators will attempt to _use_ those non-allocated regions of the remaining ~4046 bytes for other allocations in order to be faster (less systemcalls), this is why sometimes writing past buffer ends can give you garbage data instead of segfaults. [1] https://stackoverflow.com/questions/45458732/how-are-memory-allocations-smaller-than-page-size-handled-in-linux https://stackoverflow.com/questions/45458732/how-are-memory-...
- deathanatos 8y ago
  The link you post is about mmap; the section you quote is about malloc. The link is correct — about mmap; but that limitation does not apply to malloc. The link precisely backs up the part you're quoting, so saying "This is a misconception" makes no sense. > If you ask for 50bytes, it'll give you a full 4KiB page, because hardware can only segment memory in 4KiB chunks. No, not really. (But you seem to realize this, as you contradict this point in the next quote I'll pull.) A malloc implementation is free to divvy up a page in order to service several allocations. Just because the pointer it returns to you is somewhere within a page that must, almost by definition, be a full page is irrelevant to the point of this discussion (that malloc() is not needed b/c mmap exists, and is somehow superior). The point is that the allocation consumes, in the grand scale of things, an amount on par with the requested size. (There is some overhead, of course, but not nearly the same are the mmap-must-return-a-full-page overhead.) > Actually some allocators will attempt to _use_ those non-allocated regions of the remaining ~4046 bytes for other allocations in order to be faster (less systemcalls), this is why sometimes writing past buffer ends can give you garbage data instead of segfaults. That was part of the point I was making. You seem to have misunderstood my post. malloc() is free to divvy up a page (potentially handed to it by mmap) into several allocations. mmap() cannot. The point here is that replacing every malloc() call with a call to mmap() results in the loss of that ability to divvy up pages, requiring each allocation to be a full page; this would significantly bloat the memory consumed by an application, as small allocations would not share pages. Hence why those two calls are not equivalent, which is what the original post: > you mean `mmap`? > Also yes I'm radical enough to suggest the 50+ year old C standard library API is bad. Very much seemed to imply. Though, again, I asked for clarification on your point, to make sure I'm understanding it, and you again failed to provide it. So, again, > You didn't really clarify why you thought malloc() was bad, or why you think someone should call mmap() [in lieu of malloc()], so I'm mostly left to guess.
  - valarauca1 8y ago
    A malloc implementation is free to divvy up a page in order to service several allocations But like the hardware won't enforce that. So yeah you can malloc 50bytes, but the hardware is just going to assume there is a 4096 based page behind it. Because what memory can/cannot be written/read too is controlled by mmap -> linux -> hardware MMU. So when ever `malloc` gives you 50bytes without allocating an entire page it's just lying to you and giving you a pointer offset from another allocation. This isn't strictly being enforced by anything. There is zero contract that enforces what memory you can/cannot use.