13 ms·
In C, how do you know if the dynamic allocation succeeded?
- DougN7 5y agoIs this really an issue with C? Isn’t it ultimately an OS config issue?
- mirashii 5y agoI'm a little disappointed that the article didn't answer the question, or at least try to. A discussion of using read/write vs mincore vs trying to catch a SIGSEGV would've been a nice addition.
- kibwen 5y agoThe answer is that you kind of can't. You're at the mercy of the OS to give you accurate information, and malloc as an interface isn't set up to distinguish between virtual memory and "actual" memory. We could imagine a separate interface that would allow the OS to communicate this distinction (or hacks like you allude to), but I don't know of any standard approach.
- convolvatron 5y agobefore linux - every unix OS would fail a page allocation if there wasn't a backing store. full stop. this worked really well
- ungamedplayer 5y agoUntil it didn't.
- marcosdumay 5y agoIt has always lead to all kinds of problems, like computers becoming inaccessible, and corrupted data because of segfaults between disk accesses. And also nobody ever checked the malloc result anyway. Competent programmers just ensured that the segfault wasn't a huge problem. So at the best of the days, all it did was reducing the capacity of the computer.
- rurban 5y agolibsigsegv. Used to catch out of memory errors to start a major GC.
- dezgeg 5y agoHmm, I don't think that one would help with OOM killer termination either.
- yjftsjthsd-h 5y agoTLDR: "You don't." (Because malloc hands you virtual memory and actually trying to use it might reveal that the system doesn't have the real memory to handle your request. I kept reading hoping that there was going to be a solution, but not really; there are comments discussing disabling overcommit, but even that's a tradeoff (it does fix this failure mode, but you might not want to actually run a system like that).
- one_off_comment 5y agoThere's gotta be some way to programmatically determine it, right? It may not be portable. It may require some system calls or something, but there's gotta be a way, right?
- matheusmoreira 5y agoMaybe a system call that checks whether writing to a page would result in a fault? What if we had memory file descriptors for pages? We'd be able to use them with epoll and io_uring to asynchronously check whether it's safe to write to the pages they represent.
- wmf 5y agoI guess you could mlock() your memory and see if that succeeds or fails. When overcommit is enabled no memory is really safe.
- yjftsjthsd-h 5y agoI'm not an expert enough to tell you; I just read the article and decided that it was too long so summarized for others. There's some discussion upthread about catching SIGSEGV and other methods, FWIW.
- TillE 5y agoThe question is what problem are you actually trying to solve. You can get away with just about anything if you do it in a child worker process which can safely crash.
- AshamedCaptain 5y ago
- itamarst 5y agoThis difference between "malloc() succeeded and physical/swap memory is actually available" also has somewhat corresponding impact on how you measure memory usage. One approach is RSS, the memory in physical RAM... but what if you're swapping? Then again, maybe you swapped out memory you don't actually need and ignoring swap is fine. The other approach is "how much memory you allocated", and then you hit fun issues mentioned in this article, like "the OS doesn't actually _really_ allocate until you touch the page". (Longer version: https://pythonspeed.com/articles/measuring-memory-python/ https://pythonspeed.com/articles/measuring-memory-python/)
- toast0 5y ago> how you measure memory usage. Facebook put out something about measuring memory use on Linux after I left, and I haven't looked at it... But the best way I've seen is to have swap of size min(512M, 2x RAM) and measure the usage of that. There's some cases where sometimes something bigish gets swapped and you're actually fine, but often you really want to address that anyway.
- _3u10 5y agoIt’s likely being OOM killed on Linux. Not sure what’s happening on Mac. Try allocating a terabyte of swap and it should run. Alternatively use mmap & mlock to verify the allocation succeeded, but the process can still be OOM killed at any time for any reason.
- valleyer 5y agoSimilarly on macOS. There is a limit of 64 gigs in the VM compressor (in-core and on-disk compressed "segments" combined); when this is reached, a process that owns more than 50% of the compressed memory can be killed. See no_paging_space_action() in: https://opensource.apple.com/source/xnu/xnu-7195.81.3/bsd/kern/kern_proc.c.auto.html https://opensource.apple.com/source/xnu/xnu-7195.81.3/bsd/ke... Edit: I think the 64 gigs number is out of date -- looks like it's now based in part on the amount of physical memory in the machine.
- dekhn 5y agoI once complained about malloc happily allocating memory that the physical memory system couldn't satisfy (never let your hand write a check your ass can't cash?) but the more experienced programmer asked me if I'd heard of fractional reserve banking, and if not, whether it bothered me too.
- lmilcin 5y agoNot the same thing. malloc() can tell everybody it has the memory but when push comes to shove the OS will have to admit overbooking.
- Spivak 5y agoThat's exactly what fractional reserve banking is.
- lmilcin 5y agoFunny thing, you are right. I was thinking about something else I guess.
- zeusk 5y agoWhat do you think is a bank run?
- underdeserver 5y agoTIL, unless you explicitly disable memory overcommit, it can and will overcommit. This is crazy to me.
- lmilcin 5y agoIt is not. It is actually very useful and advantageous for many reasons. The disadvantage is the pretty shitty failure mode. But in practice, I have never seen an operating system that was not a toy that was able to reliably, gracefully handle out of memory condition.
- 5y ago
- andi999 5y agoThis has nothing to do with C. The problem/feature is with the systemcall that does the allocation, and any language has to use it.
- jimbob45 5y agoIs your issue that the system call doesn't return enough diagnostic information? If so, how would you have done it differently? I'm asking out of curiosity, not out of a reflexive instinct to defend C (which has many problems).
- joosters 5y agoThe difficulty is that the lack of memory might be discovered days after it was allocated (an over-committing allocator simply doesn’t know at the time whether there will be memory or not) - how do you asynchronously tell the program that this allocation has now failed in a high-level way? Generally, UNIX will let you know about the missing memory by sending the process a signal. But by that point, there’s not much that can be done to fix things up - and remember, all the fix up code would have to run without allocating any more memory itself. That’s extremely tricky in C, and nigh-on impossible in other languages.
- astrobe_ 5y agoThe first thing that comes to mind is a "reliable" (that won't overcommit) memory allocation syscall, but one might fear that it will abused (everyone wants the best stuff even if they might not use it after all). The second thing that comes to mind is to able to probe a (virtual) address (address range) within your addressing space, to check if it's actually there without doing weird stuff, and eventually just wait until it becomes available. But at this point maybe you could just use file-backed mmap for big data (effectively managing your own swap space)?
- andi999 5y agoMy first issue is with the article, while it seems to say it is about C, it is about a different problem which is illustrated with a C program. So what I would do differently (now that memory is cheap) : do not do overcommiting, so the OOM is out of the window. Do a different approach to fork, so you can exec commands and don't need to allocate the world. Actually just using the ms windows memory model would be fine.
- not2b 5y agoOn Linux, people should be pointed to https://www.kernel.org/doc/Documentation/vm/overcommit-accounting https://www.kernel.org/doc/Documentation/vm/overcommit-accou...
- hdjjhhvvhga 5y agoThis not "In C", this is a system-related issue. I always disable overcommit as I see no benefit on my systems.
- Arnavion 5y agoI used to disable overcommit, but I ran into a bunch of compilers that would crash because of it despite having 30GiB of free (as reported by free(1) ) memory available.
- lmilcin 5y agoOne of my interview questions starts with "Can a program allocate more memory than is physically available on the server?" Everybody gets this wrong (which is funny for a binary question) but it starts an interesting discussion through which I hope to learn how much they know about OS and virtual memory.
- floxy 5y agoDon't leave us hanging. The "obvious" answer would seem to be "yes" because of swap. But if everyone gets that wrong...
- foobarian 5y agoIt's a vague question though. What kind of OS? What kind of runtime? Does "allocate" mean call malloc() and get non-null return value?
- yjftsjthsd-h 5y agoNo, it's worse than that - the answer is "yes", because virtual memory + overcommit means that most of the time the OS will happily allow you to allocate more memory than physical+swap, and essentially gamble that you won't actually need all of it (and this is implemented because apparently that's almost always a winning bet).
- lmilcin 5y ago
- davidw 5y agoThis is why some embedded systems don't do malloc (IIRC...been a while since I've read much about those).
- edwinbalani 5y agoThat's right - some "safe" coding standards like MISRA C go as far as forbidding use of malloc() without explicit justification. If you still need dynamic allocation, you might choose to have a custom allocator working from a fixed-size pool or arena created in code (possibly itself carved out of RAM with malloc(), but importantly only once at first setup, where you have a better guarantee that the allocation will succeed).
- edwinbalani 5y ago(All that said, embedded systems sometimes don't have virtual memory, so the original problem stated in the link is just not a thing...) > working from a fixed-size pool or arena created in code (possibly itself carved out of RAM with malloc(), but importantly only once at first setup, where you have a better guarantee that the allocation will succeed) And I should add to this that you probably want to access all of the pool/arena to do setup, or just ensure it's physically allocated if you are running in a virtual memory space. This is something that is reasonable at setup time, though.
- SV_BubbleTime 5y agoAll good stuff. I’m working on a project now and trying my best to make it MISRA compatible because FreeRTOS did the same. You don’t use malloc() but rather pvPortMalloc() which pulls from an already allocated pool. Their HEAP4 system tries to keep the chunks organized. So far I’ve been very happy with it.
- stefan_ 5y agoEmbedded systems (the ones where you would disallow malloc) don't generally have virtual memory by virtue of having no MMU, so on those you can't do overcommitment since there is no page fault mechanism. No, the reason is simply that by statically allocating all memory you can avoid entire classes of program faults and bugs. There are no memory leaks and you don't need to solve the NP-complete problem of "is there an execution path where a dynamic memory allocation will fail". Keep in mind that it is not just about the total amount of dynamically allocated memory, but also the order of allocations (and frees).
- pcwalton 5y agoIt's important to remember the biggest reason why overcommit exists on Linux and macOS: fork(). When a process forks, the vast majority of the child process's memory is safely shared with the parent process, due to copy-on-write. But a strict accounting would say that the total memory usage of the system has doubled, which is too conservative in most cases. Since forking is so common on Unix, overcommit ends up being necessary pretty quickly. Otherwise, fork/exec would stop working in any process using more than half of the system memory.
- secondcoming 5y agoEven the linux docs for it suggest to turn it on if you're working with sparse arrays, there's no mention of fork() https://www.kernel.org/doc/Documentation/vm/overcommit-accounting https://www.kernel.org/doc/Documentation/vm/overcommit-accou...
- convolvatron 5y agowouldn't you just account the COW pages against the parent until they are copied? kicking the can down the road means there isn't any longer a reasonable correction (failing the allocation), but instead we get to drive around randomly trying to find something to kill. this is particularly annoying if you are running a service. there is no hope for it to recover - for example by flushing a cache. instead the OS looks around - sees this fat process just sitting there, and .. good news, we have plenty of memory now.
- pjc50 5y agoBut what happens when the kernel needs to copy one and has run out of memory? You still get a random process killed. (I note that Windows has a different approach, with "reserve" vs "commit", but nobody regards that as a preferential reason for using Windows as a server OS)
- magicalhippo 5y ago> But what happens when the kernel needs to copy one and has run out of memory? Don't allow it. Fork the process with read only pages except for the ranges passed to fork(). Count read-write pages as used memory by the child process. If the forked process wants to write to a page that's read-only it'll have to do a system call to turn it read-write. That call can then fail if there's not enough free memory to copy the pages. Of course, kinda hard to fix this now...
- haxorrr666 5y agoNope. Works fine on Fedora 34:<BR> # free -h total used free shared buff/cache available Mem: 31Gi 3.1Gi 2.2Gi 27Mi 25Gi 27Gi Swap: 15Gi 62Mi 15Gi # uname -a Linux athena 5.13.19-200.fc34.x86_64 #1 SMP Sat Sep 18 16:32:24 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux # gcc -o memaloc memaloc.c # ./memaloc error!
- haxorrr666 5y agoNope. Works fine on Fedora 34: # free -h total used free shared buff/cache available Mem: 31Gi 3.1Gi 2.2Gi 27Mi 25Gi 27Gi Swap: 15Gi 62Mi 15Gi # uname -a Linux athena 5.13.19-200.fc34.x86_64 #1 SMP Sat Sep 18 16:32:24 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux # gcc -o memaloc memaloc.c # ./memaloc error!
- Const-me 5y agoThat’s one of the many reasons I think Windows NT kernel is generally better than Linux. Windows doesn’t do that. When you don’t have enough memory and not enough page file space either, these allocation functions usually do fail returning nullptr.
- milkey_mouse 5y agoI've found overcommit useful for some scientific computing applications, but you're right it's not the most intuitive default. You can disable memory overcommit on Linux for the same behavior as Windows, if you want. https://www.kernel.org/doc/Documentation/vm/overcommit-accounting https://www.kernel.org/doc/Documentation/vm/overcommit-accou... Also, I found this post that suggests Windows technically does overcommit memory, but only for stacks(‽): https://superuser.com/questions/1194263/will-microsoft-windows-10-overcommit-memory https://superuser.com/questions/1194263/will-microsoft-windo...
- toast0 5y agoDisabling overcommit on Linux doesn't tend to enable library and utility developers to have used memory responsibly. So it can be hard to run a system with common software.
- milkey_mouse 5y agoI keep it disabled on one of my systems for exactly that reason: I want to make sure my own code works with overcommit disabled, as it'll always be on Windows or OpenBSD.
- tialaramex 5y agoIf you really want that you can just tell Linux not to overcommit. I can foresee your next question though, "Wait, why did I run out of memory? Stupid Linux, I have plenty of RAM".
- dodobirdlord 5y ago
- zajio1am 5y agoIMO strict memory accounting misses the point. It can ensure that all allocated pages fits to VM (or return error during allocation), but more pragmatic memory constraint is whether working set of pages (including code/mmap-based pages) fits in physical RAM. If that is not satisfied, system/application crawls to a halt due to page thrashing and some kind of OOM killer is needed. And that may happen even if strict memory accounting is satisfied.
- a-dub 5y agoif you want to allocate physical memory, allocate address space with mmap() (malloc with options) and use mlock() to wire the physical pages to the addresses in your process. mlock() will fail if there's not enough physical ram to satisfy your request.
- bronzeage 5y agoThis is actually not the common behavior you'll see when normally exceeding memory limits. If you allocate a lot of memory with relatively small allocations, you'll actually reach a null malloc. It's only when asking for a huge allocation all at once that this behavior happens.
- steinuil 5y agoIs the title a reference to Terry Riley's In C?
- Galanwe 5y ago> If you use a system-level tool that reports the memory usage of your processes, you should look at the real memory usage. Things are a bit more complicated than that. Because RSS will contain memory mapped to your process that could also be mapped by other processes. That is the sum of RSS on your machine is also higher than your physical memory. That includes libraries dynamically linked to your executable, but more importantly shared memory mmapped to your process. A more "fair" estimate exists in the form of PSS (or USS), that will list all mapped regions from all process, and account each process a proportional share of the region. e.g. If 2 processes mmap `/dev/shm/foo` of 1GB, both will inherit 500GB by PSS computation.
- senderista 5y agoOvercommit should not be necessary in order to just reserve a range of VM without committing it. Windows gets this right (requiring explicit commit to use memory if you explicitly reserved it earlier). Demand paging on Linux is super-convenient (and my designs exploit it heavily), but you can’t really write robust programs that way.
- SavantIdiot 5y agoTry that on IAR EWARM on a Cortex-M0 target. It will absolutely fail (first because size_t is only 32b). What's behind malloc() is what matters. That being said, I never would have expected that code to ever succeed! Shows how much I take memory allocation for granted on more sophisticated systems. I can't remember the last time I malloc'd more than a few megabytes.
- aetherspawn 5y agoYou can probably mitigate this with malloc_s or similar that writes 0 bytes to the entire allocated space before returning.
- grimdata 5y agoIf allocation didn't succeed, you have bigger problems. I personally dislike articles like this one. It is apparent they have never run into the problem in a production scenario.
- DeathArrow 5y agoLife is hard for C coders. Kubernetes made writing poor code a breeze. At work we have microservices crashing 20 times a week but SLOs are not affected since traffic is routed to surviving pods. So we can concentrate on churning features fast instead of writing good code.
- carlmr 5y agoK8s might save you from crashes affecting availability but what about data corruption, logicals errors sound APIs etc.? I'm guessing if code quality is neglected all of these suffer.
- noobermin 5y agoJust enough to get a new round of funding, a bonus, then to parachute out and let someone else deal with the bills when they come due
- DeathArrow 5y agoNo founding needed, we are corporate owned. The business guys demand features like there's no tomorrow so we have to do some trade offs. If the dev team owned the thing we would have taken other decisions.
- DeathArrow 5y agoCode quality is not neglected to the point we have logical errors. Architecture is reasonable and we have an extensive suite of end to end tests and integration tests.
- carlmr 5y agoThis... sounds better than most companies.
- KronisLV 5y ago> At work we have microservices crashing 20 times a week but SLOs are not affected since traffic is routed to surviving pods. This probably makes a lot of people have strong negative emotions, but at the same time i feel that you're not wrong and it's the only way to deal with the modern web dev, where clients/business push for features instead of quality, versus something like kernel/system software development, where there is more pushback against this for historial and cultural reasons. At work, we have this one monolith system that's in the center of everything else within a particular project - it's not really scalable and it has multiple scheduled processes within it, as well as serves a lot of external API requests, oh and also has an administrative UI. So far, my attempts to warn people against having a single point of failure like this have fallen on deaf ears and we still have outages where the JVM misbehaves or scheduled processes gobble up all of the server's memory and GC slows everything down on a regular basis. Contrast this to me finally getting to implement something more like microservices in another project - the services are containerized and run on servers that have been configured with Ansible, are horizontally scalable and have proper load balancing. Furthermore, the scheduled process functionality and others can sit behind feature flags and be enabled within a particular instance, all while not having multiple separate projects and keeping things simple with a single, modular codebase. I actually dubbed this approach "moduliths", horizontally scalable and modular monoliths, since there is no way that this org can handle "proper" microservices, about which i wrote more here: https://blog.kronis.dev/articles/modulith-because-we-need-to-scale-but-we-also-cannot-afford-micro-services https://blog.kronis.dev/articles/modulith-because-we-need-to... That said, even the older monolith projects can benefit from modern approaches like Ansible for configuration management (which also prevents situations where environment configuration diverges over time and no one has any idea why) as well as being put into containers - the horrible monolith application now also lives within a container (not yet in prod, sadly) and has built in health checks. Were it ever to break and fail to recover in a set time, then it will automatically restart, making an outage that lasts an hour and possibly makes someone get paged in the middle of the night instead be a minute long interruption before everything restarts. Personally, i think that with the direction that the industry is headed all services and even servers should be restarted every now and then anyways, since with JVM/CLR you sometimes get weird things happening after a service being up for months or years. Knowing why that happens would be nice, of course, but no one actually has the time to address those.
- ncmncm 5y agoIf you care, you mmap it, and request MAP_POPULATE. It would not be very surprising if, once enough people come to depend on this, they break that, too.
- rwmj 5y agoI thought this was going to be about the more fun point that malloc is allowed to return NULL even on successful allocation (because malloc(0) is allowed to return NULL). This is a quick way to check if the authors of malloc wrappers know what they are doing.
- atfzl 5y agoThis made be curious to check the real and virtual memory of some processes on my laptop (MacBook Air M1). The real memory size of Safari is ~160MB but virtual memory size is 392GB which doesn't look right. I checked other processes and all the processes have similar virtual memory size which is around ~390GB. I wonder if this is a bug in Activity Monitor or the virtual memory allocations really are this big for each process.
- EdSchouten 5y agoThat’s likely because of the dyld shared cache.
- kccqzy 5y agoThat sounds about right. The runtime of a certain language actually allocates 1TB of virtual memory at startup, and then hands out memory from that pool. It's just reserving 1TB of virtual address space.
- _pmf_ 5y agoCould this be solved by having a process level attribute that forces for all allocations what mlock(2) does? Does mlock(2) even do this?
- globular-toast 5y agoEver since I learnt about overcommit I've enjoyed blowing C programmers' minds with it. It's really surprising how many still think malloc fails!
- errcorrectcode 5y agoI suspect calloc() doesn't have this problem as it initializes the area with zeroes.
- fellellor 5y agoI tried this program under Ubuntu 20.04 (gcc 9.3.0) and got the "Cannot allocate memory" message.
- mrlonglong 5y agoWhy use malloc() when mmap() does the job