4 ms·
Question: One problem of current superlarge language models is their high inference cost. Wouldn't that be a perfect application for analog computers, since the
by cubefox 3y ago
Question: One problem of current superlarge language models is their high inference cost. Wouldn't that be a perfect application for analog computers, since they are known to be power efficient? As I understand, the weights of a neural network are approximate values anyway, so it probably wouldn't be a problem when the calculations (vector multiplication or something like that?) aren't perfectly precise.
I assume this idea has long been considered and doesn't actually work for some reason. Perhaps because implementing an order of hundred billion parameters in an analog computer isn't technically feasible? That's about how many transistors are in a CPU, and those are presumably much simpler to implement than the weights ("synapses") of a neural network.
(I know there are special inference accelerator chips, like TPUs or co-processors on smartphone chipsets, but those seem to be fully digital and not actually "neuromorphic".)
- cshimmin 3y agoThis is essentially how the very first neural network, the Perceptron, was implemented in the late 50's. It even has potentiometer dials to set the weights, and they were driven by motors during training! As for feasibility of doing analog inference today, it's an interesting idea and I'm not sure. Billions of parameters would mean billions of wires though, there may be substantial parasitic losses in building such a thing.
- buescher 3y agoCarver Mead tried back in the eighties. There’s a book - and Synaptics.
- cubefox 3y agoThe most advanced form of analog neuromorphic computer seems to be this: https://www.nature.com/articles/s41586-022-04992-8 https://www.nature.com/articles/s41586-022-04992-8 It makes use of a special ReRAM ("memristor") chip. But, as I suspected, it is far too small to run large language models.
- fulafel 3y agoAnalog nature by itself doesn't have to mean individual wires or knobs. I woner what the analog equivalent of a FPGA (mutable connections) would be.
- rapjr9 3y agoSimpler machine learning algorithms are being implemented in analog hardware. It costs much less power: https://sites.dartmouth.edu/odame/research/asthma-symptom-monitoring/ https://sites.dartmouth.edu/odame/research/asthma-symptom-mo...