3 ms·

I say the biggest missing link remains a complete lack of modeling whatsoever for some time domain. Representation of time delay between neurons and other eleme

I say the biggest missing link remains a complete lack of modeling whatsoever for some time domain. Representation of time delay between neurons and other elements can encode a lot of information. As it currently stands, everything in popular AI architectures is evaluated in a completely timeless manner. I believe very deeply that spiking neural networks are the path to AGI. I also believe that the amount of compute ultimately required to realize practical SNNs is going to turn out to be shockingly small.

l33t7332273 3y ago
I thought there were NN architectures where some layers’ outputs are sent to layers not directly in front of them. Wouldn’t that essentially capture some simulation of a time delay?
- bob1029 3y ago
  Is there any inherent delay modeled between any of the layers in your hypothetical NN architecture (assuming non-SNN)? If there are no actual time delays modeled within the network, then whatever it evaluates is representative of the outside state at some instant in time.
cglan 3y ago
I think the problem is that SNNs aren't suited to standard von neuman architecture. I'm in total agreement. That amount of power and compute thrown at language models, and ML in general is shocking relative to the capabilities. For GPT4, it's staggering. Probably on the order of KWHs in power alone for basic tasks. I think a properly modeled architecture would look something like the actor model represented in hardware, where each actor is basically a small extremely power efficient cpu but unfortunately you wouldn't be able to pull that off the shelf.
- bob1029 3y ago
  > I think the problem is that SNNs aren't suited to standard von neuman architecture. This is where we begin to disagree. If you are content with moving from actual real time to synthetic time, all of the potential frustrations fall away again. Think about online vs offline rendering in computer graphics. Theoretically, you could render all of Toy Story on a RasPi4. Most of us would be willing to wait days for answers, assuming the batch size is acceptable and the answers are correct.

l33t7332273 3y ago

I thought there were NN architectures where some layers’ outputs are sent to layers not directly in front of them. Wouldn’t that essentially capture some simulation of a time delay?

bob1029 3y ago
Is there any inherent delay modeled between any of the layers in your hypothetical NN architecture (assuming non-SNN)? If there are no actual time delays modeled within the network, then whatever it evaluates is representative of the outside state at some instant in time.

cglan 3y ago

I think the problem is that SNNs aren't suited to standard von neuman architecture. I'm in total agreement. That amount of power and compute thrown at language models, and ML in general is shocking relative to the capabilities. For GPT4, it's staggering. Probably on the order of KWHs in power alone for basic tasks. I think a properly modeled architecture would look something like the actor model represented in hardware, where each actor is basically a small extremely power efficient cpu but unfortunately you wouldn't be able to pull that off the shelf.

bob1029 3y ago
> I think the problem is that SNNs aren't suited to standard von neuman architecture. This is where we begin to disagree. If you are content with moving from actual real time to synthetic time, all of the potential frustrations fall away again. Think about online vs offline rendering in computer graphics. Theoretically, you could render all of Toy Story on a RasPi4. Most of us would be willing to wait days for answers, assuming the batch size is acceptable and the answers are correct.