4 ms·
I'm not an expert, but my basic knowledge of electronic and with a little intuition, a neuron can hold memory while a transistor cannot. Since processors are ba
by jokoon 9y ago
I'm not an expert, but my basic knowledge of electronic and with a little intuition, a neuron can hold memory while a transistor cannot. Since processors are based on a separated memory and data calculation, it makes sense that a standard unit of a processor that can do both retain memory and do calculation should scale better, just like it seem to do in a biological brain.
Even when you look at CPU performance, you can often pinpoint bottlenecks right at the amount of available L1 or L2 cache. Cache locality has almost always been the limitations of performance, because to process data, you must first access and write data. So if memory is more closely available, then everything should always be fast.
Also, remember you cannot do software on a GPU, because GPUs, even with CUDA or OpenCL, are not built to run software for the simple fact that GPUs don't do error correction. OpenCL and CUDA will only help when processing data that can be parallelized, so where the result will not risk to be jeopardized if errors accumulate.
- twtw 9y agoCould you please clarify your last paragraph? You can definitely "do software on a GPU," and GPUs have as much error correction as anything else.
- steve_musk 9y agoAgreed, the last paragraph makes no sense.
- jokoon 9y agoRead this comment: https://news.ycombinator.com/item?id=16436487 https://news.ycombinator.com/item?id=16436487 You can do software on a GPU, but you cannot have good guarantees.
- twtw 9y agoI don't know about AMD, but NVIDIA GPUs also have internal protection. Tesla and Quadro GPUs have had internal error detection on registers and cache since Fermi at least. But this is totally irrelevant anyway, the comparison we started with is to an analog alternative. Anything analog will have strictly worse noise and error problems. In neural networks, errors are probably not even a problem for analog implementation, so they definitely aren't a problem for GPU implementations.
- jokoon 9y agoI was not talking about neuron networks. But ok, I was not aware that gpu had error correction, how much additional transistor does it take?
- jcranmer 9y agoA 1-bit SRAM cell consists of 6 transistors (or 4 transistors and 2 resistors). While a single transistor doesn't have a capacity for memory, it is trivial to build small circuits of transistors that have memory. The fundamental problem in modern computers isn't memory: it's in moving data around. The speed of light in a vacuum gives you only a few cm of distance to move information in a single clock cycle, and the actual electronic propagation inside the processors is substantially slower. In fact, the governing factor of the size of L1 cache is the time it takes to actually read a value. At the scale of supercomputers, the topology of the interconnect has major implications for the actual performance on HPC applications. Saying that bringing memory closer is the determining factor in speed ignores the fact that the size of memory has implications in the time to access it. The innovation in CPUs has been about minimizing latency essentially by developing better heuristics in what it might be. GPUs innovate by not trying to minimize latency but instead trying to overprovision cores and rely on batched memory access (consequently, GPUs are not good at handling codes that rely on irregular memory access patterns).