3 ms·
(Not an expert but ...) unless you pin threads to cores, which is not the default and somewhat awkward in Linux for user applications, having a per-thread cache
by rwmj 8d ago
(Not an expert but ...) unless you pin threads to cores, which is not the default and somewhat awkward in Linux for user applications, having a per-thread cache doesn't really make sense as your thread could be moved to another core and then your cache will no longer be local to the physical cache.
- skavi 8d agoi think we agree that per cpu caching seems superior. i’m looking for the other side of this. most allocators seem to have stuck with per thread.
- Someone 8d agoIf you use a thread-local data structure, your allocator can pretend that it is running on a single-core, single-task system. If you use a CPU-local data structure, you must handle the case where, mid-way through a call to your allocator, the CPU runs a second thread that makes another call to your allocator (and that, too, can get interrupted by another thread that allocates memory, etc.) That makes thread-local easier to implement and likely faster (it doesn’t require any memory barriers in the fast path) Also, good schedulers try to avoid moving threads between CPUs. The better they manage to do that, the lower the cost of having per thread data structures (there likely still is a price, as there most of the time are more threads than CPUs on a system)
- skavi 8d agohttps://google.github.io/tcmalloc/rseq.html https://google.github.io/tcmalloc/rseq.html i don’t believe rseq based cpu local caches require memory barriers on the fast path.
- fc417fc802 8d agoSetting aside whether or not you can pull off a lock free approach here we can be certain of a couple things. There will be at least some overhead that must be paid somewhere even if that's on a separate management thread. And there will be a lot of additional complexity because that's just how concurrency always is. Meanwhile the better the scheduler performs the more competitive the thread local approach becomes.
- skavi 8d agohttps://docs.kernel.org/userspace-api/rseq.html https://docs.kernel.org/userspace-api/rseq.html cost for interruption in an rseq critical section is that the PC gets overwritten to the rseq abort entry point before the task is rescheduled. no management thread necessary. should be fairly minimal cost, especially assuming interruptions in the critical section are rare.
- fc417fc802 7d agoThat's certainly interesting but I don't see how it would change my answer to you. Your question was why projects don't switch. My answer was because doing so seems likely to be a wash at absolute best. Giving it some more thought, I expect caches will typically be wiped out by a context switch. So the only place rseq is likely to benefit an allocator is on systems with multiple NUMA nodes where you'd like to make sure any management code isn't paying a penalty by hitting the wrong address range. IIUC rseq (ie CPU local data) is primarily good for two things. The first being obviating the need for atomics (specifically the resultant cache line ping-pong) but thread local data already accomplishes that. The second being massive oversubscription of physical CPU cores (ie tens of thousands of threads) where TLS becomes utterly wasteful while also thrashing the cache.
- skavi 7d ago> seems likely to be a wash at absolute best maybe i give it too much weight, but: > massive oversubscription of physical CPU cores (ie tens of thousands of threads) where TLS becomes utterly wasteful while also thrashing the cache seems worth solving to me
- fc417fc802 7d agoAre you sure you don't just have an axe to grind? Because that is an exceedingly uncommon edge case. Typically you only have a few threads and you aren't inundating the allocator with requests thus it is unlikely to make any practical difference. If we do decide to concern ourselves with performance TLS has zero overhead and doesn't suffer from contention while rseq (at minimum) carries a penalty if preempted and involves setting a flag plus exhibits a data dependence for the address offset (the latter since AFAIK compilers don't natively support it as they do TLS). So while I'm certainly open to benchmarks to me it very much looks like a mixed bag that only comes up when you're already in questionable territory to begin with. In the event that we do step outside the norm I'd guess that a handful of threads with contention is a much more common scenario than thousands of threads per physical core exhibiting only minimal preemption. I also expect something like a web server servicing thousands of requests in parallel to use an event loop instead of spawning an equivalent number of threads. I'm struggling to come up with a scenario where you haven't already fatally shot yourself in the foot and this remains a useful optimization to make. It's certainly relevant if you're using fibers (green threads, whatever you want to call them) but at that point you aren't in c calling malloc and your language runtime will (one hopes) already be taking care of all this for you.
- Someone 8d agohttps://lwn.net/Articles/1033957/ https://lwn.net/Articles/1033957/: rseq_cs The rseq_cs field is a pointer to a struct rseq_cs. Is is NULL when no rseq assembly block critical section is active for the registered thread. Setting it to point to a critical section descriptor (struct rseq_cs) marks the beginning of the critical section. I’m not sure I fully understand that man page (it never seems to say callers have to clear that field at the end of a critical section, for example), but doesn’t that mean the caller has to guarantee setting rseq_cs happens_before any code in the critical section? That’s a memory barrier.
- Veserv 8d agoThat is because you do not need to clear the field at the end of a critical section. It contains the contiguous instruction range where it fires so there is no problem with leaving it active forever unless you have another critical section where you want to use it. No explicit memory barrier is required anywhere as the value is only read in supervisor mode and a privilege switch implicitly issues a LS-LS barrier on all major architectures. Even if you did not want to rely on that, you would only need a single S-LS barrier when you store the control structure the very first time.
- Someone 8d ago> That is because you do not need to clear the field at the end of a critical section. It contains the contiguous instruction range where it fires so there is no problem with leaving it active forever unless you have another critical section where you want to use it. Aha! So, to take advantage of that, a memory allocator uses the same abort handler for all operations?
- ckennelly 7d agoTCMalloc has different abort handlers for each function: If preempted, we need to know where to restart. If `rseq_cs` is no longer describing a relevant address, that is, the program counter has moved past it, the kernel just ignores it.
- 8d ago
- jeffbee 8d agoExactly. You want memory arenas that are hot in this CPU's caches. If your thread moves, its per-thread caches are now elsewhere. Original TCMalloc was developed in the days of 2-4 core servers. Current TCMalloc was an evolution in the context of 32+ core servers.
- skavi 8d agomy feeling is that the space efficiency gains are probably more significant than the reduction in core migration costs. many applications have far more threads than the system has cores.
- jeffbee 8d agoSure, also true that the per-CPU scheme co-evolved with the proliferation of services with thread-per-request architectures having way more TIDs than cores.
- loeg 8d agoDon't you just take the current cpuid when you go to access the cache again? Then you mutex and access the per-cpu state. There is a tiny race window but 99.99% of the time you will be hitting the same cpu's cache as you just identified, and there will be ~zero contention. The main issue is thread preemption while you're holding a per-core cache mutex. Some other thread can't do meaningful work using the cache while the holder is sleeping.