The real downside to reference counting is concurrency. If the Reference Counter lives in one Core A's L1/L2 cache, then Core B needs to write to the reference

The real downside to reference counting is concurrency. If the Reference Counter lives in one Core A's L1/L2 cache, then Core B needs to write to the reference count, the cache line is invalidated, and Core A needs to re-fetch it again. You get two cores fighting over the cache line. Everything still works correctly, but it causes some delays if each core is constantly updating the reference counter. One workaround is giving the threads their own private reference counter (making sure they're on different cache lines), and updating the shared reference counter only when their private reference count reaches 0. Not only do you stop the fighting, you don't even need atomic operations to update the private reference counts anymore.