5 ms·
While many valid points, they put GPUs in the specialized processor category to seal their argument. It is technically true, however the trend in GPUs is and ha
by dahart 8y ago
While many valid points, they put GPUs in the specialized processor category to seal their argument. It is technically true, however the trend in GPUs is and has always been toward more general computing, and most computers have GPUs in them. I expect to see commercially viable CPUs with SIMD units (Intel tried and bailed, maybe they’ll try again...) as well as GPUs with virtual & shared memory any day now.
- mattnewport 8y agoGPUs already have virtual addressing and the ability to share memory with the CPU and other GPUs in at least some circumstances. What they don't have is automatic page faulting to persistent storage or fully shared memory with the CPU by default but that is for performance reasons. For most applications of GPUs performance is too important to want either behavior by default. All GPUs are basically full of giant SIMD units and the programming models increasingly expose this. They just don't have a common standard ISA.
- dahart 8y agoTotally agreed, automatic page faulting to disk isn't something you want in a GPU application right this second, but with Moore's law dying, I just expect buses to catch up. And you're also right the programming models do expose the SIMD nature, but at the same time they are becoming increasingly easier, and increasingly more flexible and more general purpose. At least I'm thinking of CUDA and NVIDIAs more recent independent thread scheduling; tolerating higher levels of divergence. I'm less familiar with AMD hardware, but I assume the trends are the same. Really, I'm looking at the trend more than any specifics or constraints of today. GPUs used to be only for rendering triangles, and only a small number of gamers and an even smaller number of graphics researchers cared. Today, the applications are way more mainstream, everyone has GPUs, and interest in AI and crypto and general high performance computing is rivaling that of graphics & games.
- mattnewport 8y agoThe thing is GPUs achieve their performance by embracing certain realities that aren't likely to change any time soon. The classic model of memory reflected in conventional CPU architectures tries to keep up the pretense that all memory accesses are equally fast through lots of levels of caches. GPUs have a model that recognizes memory locality as fundamental and treats general memory access more like IO - something asynchronous that you want to issue a request for and then find other work to do while you wait for a response. Fine gained thread context switching with massive parallelism mostly hides that from the programmer and saves them having to write an await on every memory fetch. CPUs also go to heroic lengths to try and make shared memory transparent with complex cache synchronization logic. GPUs allow you to do that to an extent with atomics but have a programming model that discourages it. That's a model that better aligns with the realities of hardware. Moore's Law was always about number of transistors, not clock speed or single thread performance. GPUs have kept it fairly alive in that regard. Even CPUs continue to increase transistor counts quite effectively, they've just stalled out on clock speed increases and ways to use those transistors to increase single thread performance and a lot of software still can't scale well by just adding cores.
- dahart 8y agoYou're right, and I'd agree with all of that. Maybe one way of expressing my point of view that involves fewer potentially wrong predictions of the future is: I feel like the SIMD programming model is now becoming accepted as general purpose computing, rather than that GPUs are particularly specialized. The GPU model is different than CPU, but not that specialized anymore. I may be drawing a subjective line. I do expect GPU programming to continue getting easier both hardware and software wise, but I also feel like, for exactly all the reasons you mention, more and more people are aware of and accepting the GPU limitations in search of that performance.
- jcranmer 8y ago> I expect to see commercially viable CPUs with SIMD units (Intel tried and bailed, maybe they’ll try again...) I don't think you meant what you wrote here. ARM(/AArch64), PowerPC, MIPS, RISC-V, SPARC, and x86 all have SIMD unit ISAs on them. In fact, on x86-64, I'm annoyed because scalar floating point uses the vector registers anyways, so just glancing for use of %xmm is insufficient to tell you if your code got vectorized. What Intel tried and axed was the many-dumb-x86 core model of Larrabee and the Xeon Phi stuff. It's arguable that a lot of that failure was do to Intel stupidly asserting that you wouldn't need to rewrite your code to make it run on that sort of architecture.
- dahart 8y agoYou’re absolutely right, totally fair point. I was thinking about SIMD-only GPUs with many thousands of parallel threads, but yes of course, small scale SIMD is ubiquitous. Cray did a SIMD CPU before, so it has been commercially viable. I don’t know who else has done it. But don’t you think we’ll start to see mixed mode CPUs that have a lot more GPU style SIMD in them? Like, not just some vector instructions that you sprinkle into CPU code, but maybe the ability to launch a big batch of fully SIMD threads. What I had in my head in my comment was not to suggest nobody has done it, I’m just thinking that in the future we will probably see CPU and GPU merge together. It seems like a strange byproduct of corporate chip-maker history that we currently have CPUs and GPUs that are separate things. Considering the widespread adoption of GPUs for high performance and general purpose computing, I guess I just expect to see processors become dual-mode in the future, rather than have two different single-mode chips.