3 ms·
Two big developments in recent years are: 1. SIMD "threads" being programmed with traditional programming languages (ie: CUDA C++). These are "false threads",
by dragontamer 8y ago
Two big developments in recent years are:
1. SIMD "threads" being programmed with traditional programming languages (ie: CUDA C++). These are "false threads", they run in groups of 32 (NVidia) or 64 (AMD) and have unique characteristics compared to traditional threads.
2. The push for "Task-based" parallelism. Instead of writing threads, you should write tasks. Tasks are then run on a thread-pool. The difference being: Tasks don't always spin up a new thread. You track "dependencies" between tasks to minimize communication and synchronization instead. Tasks are both more efficient than threads AND easier to reason about. The crazy example is Intel's "Inefficient Fibonacci": https://software.intel.com/en-us/node/506102 https://software.intel.com/en-us/node/506102 . The task-based Fibonacci is both efficient and simple. Threads simply cannot compete.
-------------------
It turns out that threads themselves are both inefficient and complex. Pure performance pushes us towards SIMD / GPGPU compute. Even on CPUs, the fastest computational model is AVX or AVX512.
Simplicity pushes us towards the Tasking model, which happens to be more efficient in practice anyway. Its far cheaper for thread-pools to swap tasks, rather than spinning up threads for the scheduler to pass around and manage.
Classical thread based programming seems like a dead end to me. Its too complex and doesn't give enough returns on modern architectures. The Task model is built on top of threads, but I expect most programmers to switch to the Tasking model, and only to leave "Threads" to the OS-devs or System-level devs far below.
The SIMD model gets everything done efficiently with barriers in most cases (no need for semaphores or mutexes... indeed... mutexes don't work on traditional SIMD GPUs like Pascal or AMD Vega... due to how SIMD Wavefronts execute). Other issues are handled with "CUDA Shared Memory" or "OpenCL Local Memory" and atomics. Its weird to lose the Mutex or Semaphore as a tool, but you learn that you really didn't need it once you learn to use Barriers and Atomics effectively.
SIMD Model gets a LOT done with atomics, popcount, ballot, and various primitives that are alien to CPU Programmers. Watch a "Merge Path based Parallel Merge Sort" on a GPU if you wanna see what I mean. Its very alien, but it works extremely efficiently.
Tasks similarly have a different model compared to Threads. They're the natural successor. Almost everything I can think of can be more elegantly expressed as a Task instead of as a Thread. That'd be OpenMP 4.0, Intel TBB, or Microsoft PPL.