4 ms·

Ah, thanks for the details, I wasn't referring to NUMA-induced lock contention, but rather lock contention in general. I've seen a workload hit 64 CPUs with 80%

Ah, thanks for the details, I wasn't referring to NUMA-induced lock contention, but rather lock contention in general. I've seen a workload hit 64 CPUs with 80% of CPU time in lock contention (and others in the 10-30%). Now, while that means the developer has a big problem to fix, it also has me wondering about single sockets getting big enough -- 128 CPUs is already hard to use well. Back when I first saw multi-socket systems with 2-4 total CPUs, getting the extra CPUs online was all goodness. But adding another 128 CPUs to my already 128-CPU system, well...is there a point where we can say, in general, that we already have enough cores? In the talk I referred to the 850,000-core GPU, and how I couldn't see that ever working as general purpose CPUs in the software of today. With 3D stacking, I think we'll reach a practical core limit on a single socket, and just won't need the complexity (including NUMA) of multi socket anymore.

dragontamer 5y ago
But its the same in GPU-land. There are plenty of tasks which max out at one block / threadgroup of 1024 CUDA-threads. cudaMemcpy is a silly example (probably maxes out memory bandwidth at just 64 or 32 cores used), but there are plenty of tasks that simply don't scale to the full use of a GPU. Just because some tasks (many tasks?) fail to use more than 32-GPU cores doesn't mean that GPU-parallelism is useless. It just means that when you program those particular tasks, only use 32-GPU cores!! Then use the GPU-cores on _other_ tasks (possibly in parallel). IIRC, cudaMalloc, and many other primitives in the CUDA framework, has been shown to have very little parallelism at all. You need to work at keeping this "sequential-code" outside of your inner loops. (Runs on CUDA-stream #0, which for older hardware at least is sequentially scheduled) ---------- 1. Some tasks can effectively use infinite cores (SIMD-threads really for GPUs... but same idea since a SIMD-lane can largely emulate a thread as long as you're careful about branch divergence) 2. Some tasks can be parallelized at the application / operator level. Run many applications in parallel ("Makefile parallelism") 3. Some tasks (memcpy) are so memory-bound that parallelism will never help. 4. Some tasks have a better solution that becomes feasible with more compute power. --------- Lets take a CPU example: H.264 encoding. IIRC, this task barely scales to 8 cores and has diminishing returns beyond that. But an example of #2 would be Youtube: you have one encoding machine that handles transcoding in parallel. You don't run just 1 instance of the problem (using 8 cores), you run 32 in parallel, and each of those 32-instances can effectively use 8-cores for H264 encoding. And #4 can still happen: H264 is pretty easy for modern computers with little chance of parallelism. Switching to H.265 or even to AV1 will increase the compute power needed, and allow scaling of single tasks up to 16 to 64 cores. Now your 2x128 hypothetical machine can only run 4 transcoding sessions at a time. ----------- The dual-socket machine for a transcoding cluster is still superior over a single-socket machine. 10Gbit Ethernet is more than sufficient to handle 4x AV1 sessions (especially because AV1 is slower than realtime), so right there we've cut the number of Ethernet cables in half, which means we've cut the number of 10 Gbit Switches in half. Having the two sockets share one Ethernet port is an efficiency gain even if you don't have any task-communication going on: if only for the I/O sharing capability of the NUMA Fabric.

dragontamer 5y ago

But its the same in GPU-land. There are plenty of tasks which max out at one block / threadgroup of 1024 CUDA-threads. cudaMemcpy is a silly example (probably maxes out memory bandwidth at just 64 or 32 cores used), but there are plenty of tasks that simply don't scale to the full use of a GPU. Just because some tasks (many tasks?) fail to use more than 32-GPU cores doesn't mean that GPU-parallelism is useless. It just means that when you program those particular tasks, only use 32-GPU cores!! Then use the GPU-cores on _other_ tasks (possibly in parallel). IIRC, cudaMalloc, and many other primitives in the CUDA framework, has been shown to have very little parallelism at all. You need to work at keeping this "sequential-code" outside of your inner loops. (Runs on CUDA-stream #0, which for older hardware at least is sequentially scheduled) ---------- 1. Some tasks can effectively use infinite cores (SIMD-threads really for GPUs... but same idea since a SIMD-lane can largely emulate a thread as long as you're careful about branch divergence) 2. Some tasks can be parallelized at the application / operator level. Run many applications in parallel ("Makefile parallelism") 3. Some tasks (memcpy) are so memory-bound that parallelism will never help. 4. Some tasks have a better solution that becomes feasible with more compute power. --------- Lets take a CPU example: H.264 encoding. IIRC, this task barely scales to 8 cores and has diminishing returns beyond that. But an example of #2 would be Youtube: you have one encoding machine that handles transcoding in parallel. You don't run just 1 instance of the problem (using 8 cores), you run 32 in parallel, and each of those 32-instances can effectively use 8-cores for H264 encoding. And #4 can still happen: H264 is pretty easy for modern computers with little chance of parallelism. Switching to H.265 or even to AV1 will increase the compute power needed, and allow scaling of single tasks up to 16 to 64 cores. Now your 2x128 hypothetical machine can only run 4 transcoding sessions at a time. ----------- The dual-socket machine for a transcoding cluster is still superior over a single-socket machine. 10Gbit Ethernet is more than sufficient to handle 4x AV1 sessions (especially because AV1 is slower than realtime), so right there we've cut the number of Ethernet cables in half, which means we've cut the number of 10 Gbit Switches in half. Having the two sockets share one Ethernet port is an efficiency gain even if you don't have any task-communication going on: if only for the I/O sharing capability of the NUMA Fabric.