5 ms·
it doesn't go to the OS each time you call malloc() the first time malloc() is invoked it will ask the OS for an oversized chunk of memory(probably using mmap(
by blibble 10y ago
it doesn't go to the OS each time you call malloc()
the first time malloc() is invoked it will ask the OS for an oversized chunk of memory(probably using mmap() or sbrk())
allocations consist of incrementing a pointer, or pulling a a chunk off a freelist, both of which are effectively free
mmap() is very fast to call too, so I'm not sure what you're on about
regardless, curl is a piece of software that spends 99.99% of its time waiting for network traffic
- avar 10y ago> I'm not sure what you're on about. You clearly interpreted OS to mean "the kernel" in my comment. To be clear I'm including the C library which implements malloc() in "the OS". > curl is a piece of software that spends 99.99% > of its time waiting for network traffic. You're commenting on an article that shows that in some cases 30% of the time was spent on malloc() overhead.
- pjmlp 10y agoWhich C library from which C compiler?
- blibble 10y agowell yes, they removed the network overhead and disk overhead by running it on localhost and writing to /dev/null, and once you remove two bottlenecks you'll find that something else becomes the new bottleneck libc is not the OS, it's the standard library for C, in the same way the java class library isn't the OS, it's the standard library for java
- avar 10y agoYou seem insistent on having some side-thread argument with me for some reason. > once you remove two bottlenecks... The article we're both commenting on goes to great length to point this out. I don't know why you're trying to convince me or anyone else of this. Yes, in a network library malloc overhead isn't usually a worthwhile thing to worry about, but libcurl is hardly just any random library. > libc is not the OS, it's the standard library for C. I'm not trying to convince you that you should consider libc part of the OS. I was merely clarifying that in the context of my comment, which you replied to assuming I just meant the kernel, that I was talking about the C library as well. FWIW lots of people use "Operating System" to mean the set of software you get by default. OpenBSD, GNU, OSX, Android etc. definitely consider libc to be part of the OS, and Android probably considers its java class library part of the base OS. If everyone meant "the kernel" by "the OS" we'd just say "the kernel". You don't have to agree with that, but correcting people on that point is about as futile as trying to convince people that your use of "hacker" is correct.
- tinus_hn 10y agoSurprisingly, when you remove the actual work, 100% of the time spent is overhead! What a waste!
- kevin_thibedeau 10y agoOn Linux. Basic core libs like libCurl can find their way into MMU-less embedded systems with relatively crude malloc()s. Sane allocation behavior is needed to ward off fragmentation that hurts the entire system.
- derefr 10y agoWouldn't it be better to write a shim library that provides Linux-like MMU behavior at a userspace per-process level, and then target libs like libcurl to it? It'd be a sort of "memory policy driver" that you'd port to an embedded OS.
- Sanddancer 10y agoYou really don't want library code providing or dictating malloc behavior, that's how things like OpenSSL and Heartbleed end up happening. It may be slower, but dealing with the politics of a particular malloc can be uglier than just making your code work well enough most of the time.
- adrianN 10y agoPulling a chunk off a freelist and maintaining that free list is most definitely not free.
- blibble 10y agoso what, to return from a freelist you alter the head pointer to point to the next item and return it that's what, 2 or 3 instructions? even if you push that upto a few hundred it's still effectively free compared to anything else curl is going to do (e.g. writing to the disk)
- nostrebored 10y agoyou have to handle the case in which you have no remaining items as well, breaking up a chunk log(chunkToBreak/chunkSizeNeeded) times. And in the case that you have no viable chunk to use, you need to call back to the OS.
- dxhdr 10y agoA structure allocated in one chunk has cache locality and will be significantly faster to access than dereferencing multiple pointers. The allocation itself is not the bottleneck, it's the poor choice of data structure.
- vvanders 10y agoYup, malloc/free isn't usually algorithmically expensive but the cache misses tend to be the brutal part. You'll spend 15 cycles doing work on 600 cycles of cache fetching. Plus you've just evicted some cache for your local scope.
- pjmlp 10y agoThat is compiler and OS specific behavior.
- Taniwha 10y agoMaybe more than you thing, if you're running a multithreaded app malloc/free/new/delete need to have mutexes to protect the heap data structures, that can also lead you into the kernel