4 ms·

I tend to disagree. It's true there's a tradeoff but there's also an opportunity cost in staying on the road we're on now (which has no future, at least for CP

I tend to disagree. It's true there's a tradeoff but there's also an opportunity cost in staying on the road we're on now (which has no future, at least for CPUs). CPU performance increases since 2000 have mainly come from longer pipelines, bigger branch prediction logic and larger/deeper caches. Those are all extremely important for single-threaded performance but are not much use for embarrassingly parallel (low branch) computation like in MATLAB/Octave, R, shaders, AI, physics and so on. You're right about Xeon Phi and make an absolutely valid point about the trouble of filling ALUs with data. It was probably destined to fail because most mainstream languages just fall down terribly trying to do multiprocessing. Only a handful like Erlang and Go get it right, but at the time just weren't on the radar. I think the world is treading water with the current CPU/GPU divide and heterogeneous processors like the Cell. Few developers fully utilized the Synergistic Processing Elements (SPEs). These paradigms provide powerful hardware but leave the intricacies to the developer, or sometimes the compiler if they're lucky. That's the part that I'm getting tired of. Companies tout the power of their various technologies without really addressing the hard problems in computer science. I want a paradigm that presents itself as a single unified CPU and memory. I want the compiler and operating system to optimize my code and handle moving my data with copy-on-write. If I had something like this and a decent Actor Model language, then it could trivially emulate things like shaders and we could get rid of all of these domain specific languages. What I'm getting at is that with a few hundred short-pipeline cores from the 90s (say MIPS or PowerPC 600 series) and a content-addressable memory, compute-bound problems become tractable. Why bother with rasterization when you can write a ray tracer in a page of code and get better results? Or imagine being able to run not just one neural net but many in parallel, or let genetic algorithms run for 2 or 3 orders of magnitude more generations. I want raw computing power, not a subset of it filtered through someone else's notion of what I might need.

jcranmer 8y ago
> CPU performance increases since 2000 have mainly come from longer pipelines, bigger branch prediction logic and larger/deeper caches. Those are all extremely important for single-threaded performance but are not much use for embarrassingly parallel (low branch) computation like in MATLAB/Octave, R, shaders, AI, physics and so on. And better misaligned memory handling (Haswell has no latency penalty reading a word that crosses cache lines), better speculative execution, wider SIMD units, more complex SIMD swizzling operations. > That's the part that I'm getting tired of. Companies tout the power of their various technologies without really addressing the hard problems in computer science. I want a paradigm that presents itself as a single unified CPU and memory. I want the compiler and operating system to optimize my code and handle moving my data with copy-on-write. If I had something like this and a decent Actor Model language, then it could trivially emulate things like shaders and we could get rid of all of these domain specific languages. Speaking as someone who writes an optimizing HPC compiler... the idea that you can write a simple program and have it magically parallelize to ultra-high-throughput code is fantasy. We had that idea back in the 1960s. We've still not solved it 50 years later, and we've learned that there are giant barriers towards making these fantasy compilers that are rather fundamental. One major problem you have to consider: communication is expensive. Already, on a single chip, it's impossible for data to move from one end of the chip to the other within a clock cycle (the speed of light in a vacuum takes about 10 times the clock cycle time to do so, and the speed of light in a physical wire is slower than that in a vacuum). So you start paying latency costs to have to move bits of data around. But your interconnect mesh isn't all-to-all, not when you have hundreds of access points. So that means that you have to pay attention to who is communicating to whom, and how much they are doing so. In a pathological case, like reversing a list, you've got all of your data moving across a single link in a 1-D network or only sqrt(#processor size) links in a 2-D network.
TomVDB 8y ago
> Why bother with rasterization when you can write a ray tracer in a page of code and get better results? You do realize that ray tracing is ultimately a problem whose performance is determined by the external memory system? Unless you’re thinking about each mini CPU rendering a bunch of reflective balls where the full scene can be stored locally in each CPU. Because once you don’t, you’ll have to find ways to cover the access latency to the shared memory pool, and before you know it your super simple CPU will looks suspiciously like the shader core of today’s GPUs. Your other examples have similar limitations. The truth is that there are not many problems that can efficiently be mapped to an architecture with tons of small CPUs, some local RAM, and nothing else.

jcranmer 8y ago

> CPU performance increases since 2000 have mainly come from longer pipelines, bigger branch prediction logic and larger/deeper caches. Those are all extremely important for single-threaded performance but are not much use for embarrassingly parallel (low branch) computation like in MATLAB/Octave, R, shaders, AI, physics and so on. And better misaligned memory handling (Haswell has no latency penalty reading a word that crosses cache lines), better speculative execution, wider SIMD units, more complex SIMD swizzling operations. > That's the part that I'm getting tired of. Companies tout the power of their various technologies without really addressing the hard problems in computer science. I want a paradigm that presents itself as a single unified CPU and memory. I want the compiler and operating system to optimize my code and handle moving my data with copy-on-write. If I had something like this and a decent Actor Model language, then it could trivially emulate things like shaders and we could get rid of all of these domain specific languages. Speaking as someone who writes an optimizing HPC compiler... the idea that you can write a simple program and have it magically parallelize to ultra-high-throughput code is fantasy. We had that idea back in the 1960s. We've still not solved it 50 years later, and we've learned that there are giant barriers towards making these fantasy compilers that are rather fundamental. One major problem you have to consider: communication is expensive. Already, on a single chip, it's impossible for data to move from one end of the chip to the other within a clock cycle (the speed of light in a vacuum takes about 10 times the clock cycle time to do so, and the speed of light in a physical wire is slower than that in a vacuum). So you start paying latency costs to have to move bits of data around. But your interconnect mesh isn't all-to-all, not when you have hundreds of access points. So that means that you have to pay attention to who is communicating to whom, and how much they are doing so. In a pathological case, like reversing a list, you've got all of your data moving across a single link in a 1-D network or only sqrt(#processor size) links in a 2-D network.

TomVDB 8y ago

> Why bother with rasterization when you can write a ray tracer in a page of code and get better results? You do realize that ray tracing is ultimately a problem whose performance is determined by the external memory system? Unless you’re thinking about each mini CPU rendering a bunch of reflective balls where the full scene can be stored locally in each CPU. Because once you don’t, you’ll have to find ways to cover the access latency to the shared memory pool, and before you know it your super simple CPU will looks suspiciously like the shader core of today’s GPUs. Your other examples have similar limitations. The truth is that there are not many problems that can efficiently be mapped to an architecture with tons of small CPUs, some local RAM, and nothing else.