4 ms·
does anyone familiar with the art have thoughts on why only tcmalloc switched from thread caches to cpu caches? would it make linux behavior diverge too much fr
by skavi 8d ago
does anyone familiar with the art have thoughts on why only tcmalloc switched from thread caches to cpu caches? would it make linux behavior diverge too much from other platforms?
- enduku 8d agoI think tcmalloc gains on thread churn and oversubscription by going the cpu-cache route on Linux. on other platforms, I am not so sure but that can be offset by say a treiber-stack like setup for cross-thread frees/teardowns. So lesser code for Linux for similar fastpath design I guess.
- jeffbee 8d ago> on other platforms, I am not so sure tcmalloc only works on Linux.
- rwmj 8d ago(Not an expert but ...) unless you pin threads to cores, which is not the default and somewhat awkward in Linux for user applications, having a per-thread cache doesn't really make sense as your thread could be moved to another core and then your cache will no longer be local to the physical cache.
- skavi 8d agoi think we agree that per cpu caching seems superior. i’m looking for the other side of this. most allocators seem to have stuck with per thread.
- Someone 8d agoIf you use a thread-local data structure, your allocator can pretend that it is running on a single-core, single-task system. If you use a CPU-local data structure, you must handle the case where, mid-way through a call to your allocator, the CPU runs a second thread that makes another call to your allocator (and that, too, can get interrupted by another thread that allocates memory, etc.) That makes thread-local easier to implement and likely faster (it doesn’t require any memory barriers in the fast path) Also, good schedulers try to avoid moving threads between CPUs. The better they manage to do that, the lower the cost of having per thread data structures (there likely still is a price, as there most of the time are more threads than CPUs on a system)
- skavi 8d agohttps://google.github.io/tcmalloc/rseq.html https://google.github.io/tcmalloc/rseq.html i don’t believe rseq based cpu local caches require memory barriers on the fast path.
- fc417fc802 8d agoSetting aside whether or not you can pull off a lock free approach here we can be certain of a couple things. There will be at least some overhead that must be paid somewhere even if that's on a separate management thread. And there will be a lot of additional complexity because that's just how concurrency always is. Meanwhile the better the scheduler performs the more competitive the thread local approach becomes.
- skavi 8d agohttps://docs.kernel.org/userspace-api/rseq.html https://docs.kernel.org/userspace-api/rseq.html cost for interruption in an rseq critical section is that the PC gets overwritten to the rseq abort entry point before the task is rescheduled. no management thread necessary. should be fairly minimal cost, especially assuming interruptions in the critical section are rare.
- fc417fc802 8d agoThat's certainly interesting but I don't see how it would change my answer to you. Your question was why projects don't switch. My answer was because doing so seems likely to be a wash at absolute best. Giving it some more thought, I expect caches will typically be wiped out by a context switch. So the only place rseq is likely to benefit an allocator is on systems with multiple NUMA nodes where you'd like to make sure any management code isn't paying a penalty by hitting the wrong address range. IIUC rseq (ie CPU local data) is primarily good for two things. The first being obviating the need for atomics (specifically the resultant cache line ping-pong) but thread local data already accomplishes that. The second being massive oversubscription of physical CPU cores (ie tens of thousands of threads) where TLS becomes utterly wasteful while also thrashing the cache.
- jeffbee 8d agoExactly. You want memory arenas that are hot in this CPU's caches. If your thread moves, its per-thread caches are now elsewhere. Original TCMalloc was developed in the days of 2-4 core servers. Current TCMalloc was an evolution in the context of 32+ core servers.
- skavi 8d agomy feeling is that the space efficiency gains are probably more significant than the reduction in core migration costs. many applications have far more threads than the system has cores.
- jeffbee 8d agoSure, also true that the per-CPU scheme co-evolved with the proliferation of services with thread-per-request architectures having way more TIDs than cores.
- loeg 8d agoDon't you just take the current cpuid when you go to access the cache again? Then you mutex and access the per-cpu state. There is a tiny race window but 99.99% of the time you will be hitting the same cpu's cache as you just identified, and there will be ~zero contention. The main issue is thread preemption while you're holding a per-core cache mutex. Some other thread can't do meaningful work using the cache while the holder is sleeping.