3 ms·

Discussions like this are interesting because the arguments on one side seem ridiculous, yet logically valid and simple to understand/articulate; the arguments

by ddxxdd 7y ago

Discussions like this are interesting because the arguments on one side seem ridiculous, yet logically valid and simple to understand/articulate; the arguments on the other side are complex and difficult to formulate into words. I just got out of the shower, so I'll add my recent showerthought: In a human brain, each neuron is a CPU in and of itself; this means that every neuron in the human brain can run in parallel. In modern computers, the "perceptrons" are virtual, and there are only between 1 and 4 CPUs to allow activation of a specific "perceptron" at a given time. So if you have a biological brain with 100 billion neurons sitting next to a virtual brain with 100 billion perceptrons, the virtual brain will run 100 billion times slower even with equivalent intelligence.

cobbzilla 7y ago
I hear you on the parallelism, the flip side is that each neuron doesn’t need anywhere near the flexibility of a general purpose CPU. I think many of the AI algorithms can run on GPUs, where core counts can get into the thousands (still very far from millions). But these are largely independent — the interconnects between GPUs are not on the scale of neurons. My guess: we develop specialized hardware that, over time, will increasingly resemble organic brain structure.
- 0815test 7y ago
  "Core counts" on GPUs are quite misleading - what are called "cores" on GPU hardware would be called "execution units/ALU's" on the CPU. GPUs have significantly-enhanced memory throughput, and can use a combination of SIMD (with masked/predicated instructions), barrel processing and genuine multicore to excel at embarrassingly-parallel compute. But they're not magic. And if what you care about is running a wide variety of mainstream stats or machine learning algorithms, the "specialized hardware" you mention is the modern GPU, or something very much like it - I don't see any low-hanging fruit that might make some other HW architecture more feasible. Fixed-point compute might get there someday, but it's way too fiddly at present - it's not really clear how to "tweak" mainstream learning algorithms so as to dispense with the large dynamic range that floating point compute provides.

by ddxxdd 7y ago

cobbzilla 7y ago

I hear you on the parallelism, the flip side is that each neuron doesn’t need anywhere near the flexibility of a general purpose CPU. I think many of the AI algorithms can run on GPUs, where core counts can get into the thousands (still very far from millions). But these are largely independent — the interconnects between GPUs are not on the scale of neurons. My guess: we develop specialized hardware that, over time, will increasingly resemble organic brain structure.

0815test 7y ago
"Core counts" on GPUs are quite misleading - what are called "cores" on GPU hardware would be called "execution units/ALU's" on the CPU. GPUs have significantly-enhanced memory throughput, and can use a combination of SIMD (with masked/predicated instructions), barrel processing and genuine multicore to excel at embarrassingly-parallel compute. But they're not magic. And if what you care about is running a wide variety of mainstream stats or machine learning algorithms, the "specialized hardware" you mention is the modern GPU, or something very much like it - I don't see any low-hanging fruit that might make some other HW architecture more feasible. Fixed-point compute might get there someday, but it's way too fiddly at present - it's not really clear how to "tweak" mainstream learning algorithms so as to dispense with the large dynamic range that floating point compute provides.