3 ms·
I think what people seems to not understand about neuron networks is the memory locality. Neurons pass messages directly between each other, so they always reta
by jokoon 12y ago
I think what people seems to not understand about neuron networks is the memory locality. Neurons pass messages directly between each other, so they always retain their own memory state, which is very different from a CPU or a GPU. CPU and GPU have a shared memory architecture, and thus memory access is generally slower overall, because of how they are designed for programming simplicity. Don't ever forget that computer speed is never about calculation, but always about memory access. A computer speed is always limited by its L1 and L2 cache size.
If you want to draw one image, GPUs are fine, but they are not really able to simulate neuron networks because they don't have as much parallelism or memory locality a neuron network needs. GPU are more parallized than a CPU, yes, thanks to OpenCL. But they're still specialized towards image blitting, not towards massively parallelism.
Neuron networks are not very hard to understand, but if you think about simulating one, you quickly understand that most hardware is just not adequate and unable to run a neuron network properly, it will be just too slow. Computers were never designed and intended to simulate something like a brain.
Also don't forget most algorithm we use today are not parallelizable, most are sequential. We could rethink many algorithms and adapt them, but parallelism is often a huge constraint. Sequentiality is a specialized case of computability if you think about it.
- calibwam 12y agoAbsolutes are dumb. Practically, a CPU is throttled on memory access, but it is not hard to imagine scenarios where the computation is the bottle neck. For instance, if most of the program is run in tight loops, where cache misses are few, and the whole loop can fit in L1, almost the whole overhead would be in computation.
- vidarh 12y agoHave you looked at the Parallela / Epiphany from Adapteva? Their current top model is 64 core, but their roadmap is targetting 4K cores or more per chip. They have local (on core) memory, and a routing mesh to let all cores access each others memory - either for communication or for extra storage, and optionally access the host systems RAM. The chips also expose multiple 10Gbps links that can be used to connect the chips themselves into a bigger mesh. You "pay" for accessing remote memory with additional cycles latency based on distance to the node you want to address, so it massively favours focusing on memory locality. I have two 16 core Parallelas sitting around, but have had less time than I'd hoped to play with them.
- jokoon 12y agoThat seems really interesting, I still wonder about the latency of the board connector, but 6GB/s is really great. Although the real issue with this board is being able to program it effectively.
- vidarh 12y agoYeah, the current boards are very much a means for people to experiment with the architecture first and foremost. I'm very curious to see what will start to happen once they get one step further up from the 64 core chips and start getting more per-core memory... Even more so if they get it onto a PCIe card so I can stick one in my home server to play with instead of yet another small ARM machine (I have a drawer full of ARM computers at this point).
- jokoon 12y agoI'm more interested about inter card latency...
- im2w1l 12y ago>Their current top model is 64 core. Sadly it isn't. They only made prototypes for the kickstarters, it isn't mass produced or available.
- lstyls 12y agoYes but human brains are not random access -- computer memory is. Your analogy holds over distributed networks, where transmission time and data loss are positively correlated with distance. Your attempt to compare human neurons to a CPU core, however, has little value. Neural nets != Human brains