4 ms·

Hopefully someone close to neural networks can describe how useful this primitive is as a node in neural networks. If it is useful then, is this a better primit

by bdamm 9y ago

Hopefully someone close to neural networks can describe how useful this primitive is as a node in neural networks. If it is useful then, is this a better primitive than simply doing it in "software" using a GPU, for example?

ShabbosGoy 9y ago
Wouldn’t the memtransistor be faster though? Even if you use CUDA, you still have to compile down to assembly and probably use some kernel level module to communicate with the GPU.
- NateyJay 9y ago
  A physical memtransistor will be a lot faster than emulating it in software, but you'll also be limited by the original connections and design of the chip. Not as easy to reconfigure as writing new code.
  - LolWolf 9y ago
    I mean, ideally I think it would be written as an FPGA-like piece of hardware where the connections can be synthesized from a high level description language. A la [0]. —— [0] https://web.stanford.edu/group/brainsinsilicon/index.html https://web.stanford.edu/group/brainsinsilicon/index.html
    - jerf 9y ago
      Then you will rapidly run into the same problems that FPGAs have, which is that all that connective tissue isn't free and has to pay for itself. That turns out to be a pretty steep bar to leap.
      - DesiLurker 9y ago
        Well the problems of FPGA have more to do with synthesibility of the high level HDL and having to fit that generic representation in a predefined LUT model. its alleviated quite a bit when you are designing 'application specific' FPGA with different substrate. IIRC Mathstar tried to do this some time ago and so did Ambrics. at the time it suffered from a solution looking for problem syndrome & failed. the big idea here is that IF your base element (memristor crossbar here) is suitable for such rapidly reconfigurable bus architecture (which it seem like it is) then you can use it to synthesis a single neuron directly. which is a huge leap over the next best GPU/TPU based architecture based on instruction fetch-decode-execute model. based on what I have read few years ago you can have a 20M neurons simulated with memristors in about a cm2 die. that is human level integration density even if you totally ignore the vast difference in switching rate (100Hz vs 1+GHz).
jokoon 9y ago
I'm not an expert, but my basic knowledge of electronic and with a little intuition, a neuron can hold memory while a transistor cannot. Since processors are based on a separated memory and data calculation, it makes sense that a standard unit of a processor that can do both retain memory and do calculation should scale better, just like it seem to do in a biological brain. Even when you look at CPU performance, you can often pinpoint bottlenecks right at the amount of available L1 or L2 cache. Cache locality has almost always been the limitations of performance, because to process data, you must first access and write data. So if memory is more closely available, then everything should always be fast. Also, remember you cannot do software on a GPU, because GPUs, even with CUDA or OpenCL, are not built to run software for the simple fact that GPUs don't do error correction. OpenCL and CUDA will only help when processing data that can be parallelized, so where the result will not risk to be jeopardized if errors accumulate.
- twtw 9y ago
  Could you please clarify your last paragraph? You can definitely "do software on a GPU," and GPUs have as much error correction as anything else.
  - steve_musk 9y ago
    Agreed, the last paragraph makes no sense.
  - jokoon 9y ago
    Read this comment: https://news.ycombinator.com/item?id=16436487 https://news.ycombinator.com/item?id=16436487 You can do software on a GPU, but you cannot have good guarantees.
    - twtw 9y ago
      I don't know about AMD, but NVIDIA GPUs also have internal protection. Tesla and Quadro GPUs have had internal error detection on registers and cache since Fermi at least. But this is totally irrelevant anyway, the comparison we started with is to an analog alternative. Anything analog will have strictly worse noise and error problems. In neural networks, errors are probably not even a problem for analog implementation, so they definitely aren't a problem for GPU implementations.

by bdamm 9y ago

ShabbosGoy 9y ago

Wouldn’t the memtransistor be faster though? Even if you use CUDA, you still have to compile down to assembly and probably use some kernel level module to communicate with the GPU.

NateyJay 9y ago
A physical memtransistor will be a lot faster than emulating it in software, but you'll also be limited by the original connections and design of the chip. Not as easy to reconfigure as writing new code.
- LolWolf 9y ago
  I mean, ideally I think it would be written as an FPGA-like piece of hardware where the connections can be synthesized from a high level description language. A la [0]. —— [0] https://web.stanford.edu/group/brainsinsilicon/index.html https://web.stanford.edu/group/brainsinsilicon/index.html
  - jerf 9y ago
    Then you will rapidly run into the same problems that FPGAs have, which is that all that connective tissue isn't free and has to pay for itself. That turns out to be a pretty steep bar to leap.
    - DesiLurker 9y ago
      Well the problems of FPGA have more to do with synthesibility of the high level HDL and having to fit that generic representation in a predefined LUT model. its alleviated quite a bit when you are designing 'application specific' FPGA with different substrate. IIRC Mathstar tried to do this some time ago and so did Ambrics. at the time it suffered from a solution looking for problem syndrome & failed. the big idea here is that IF your base element (memristor crossbar here) is suitable for such rapidly reconfigurable bus architecture (which it seem like it is) then you can use it to synthesis a single neuron directly. which is a huge leap over the next best GPU/TPU based architecture based on instruction fetch-decode-execute model. based on what I have read few years ago you can have a 20M neurons simulated with memristors in about a cm2 die. that is human level integration density even if you totally ignore the vast difference in switching rate (100Hz vs 1+GHz).

jokoon 9y ago

I'm not an expert, but my basic knowledge of electronic and with a little intuition, a neuron can hold memory while a transistor cannot. Since processors are based on a separated memory and data calculation, it makes sense that a standard unit of a processor that can do both retain memory and do calculation should scale better, just like it seem to do in a biological brain. Even when you look at CPU performance, you can often pinpoint bottlenecks right at the amount of available L1 or L2 cache. Cache locality has almost always been the limitations of performance, because to process data, you must first access and write data. So if memory is more closely available, then everything should always be fast. Also, remember you cannot do software on a GPU, because GPUs, even with CUDA or OpenCL, are not built to run software for the simple fact that GPUs don't do error correction. OpenCL and CUDA will only help when processing data that can be parallelized, so where the result will not risk to be jeopardized if errors accumulate.

twtw 9y ago
Could you please clarify your last paragraph? You can definitely "do software on a GPU," and GPUs have as much error correction as anything else.
- steve_musk 9y ago
  Agreed, the last paragraph makes no sense.
- jokoon 9y ago
  Read this comment: https://news.ycombinator.com/item?id=16436487 https://news.ycombinator.com/item?id=16436487 You can do software on a GPU, but you cannot have good guarantees.
  - twtw 9y ago
    I don't know about AMD, but NVIDIA GPUs also have internal protection. Tesla and Quadro GPUs have had internal error detection on registers and cache since Fermi at least. But this is totally irrelevant anyway, the comparison we started with is to an analog alternative. Anything analog will have strictly worse noise and error problems. In neural networks, errors are probably not even a problem for analog implementation, so they definitely aren't a problem for GPU implementations.