7 ms·

Meh, people have been trying to ape biological networks for decades and they consistently produce poor functioning versions of the leading edge... Why wouldn't

Meh, people have been trying to ape biological networks for decades and they consistently produce poor functioning versions of the leading edge... Why wouldn't we have already seen spiking networks contribute something by now?

forrest2 4y ago
I think people said the same thing about NNs in general before we hit a scale where they started performing magic. There could be exponential or quadratic scaling laws with any of these black boxes that makes one approach suddenly extremely viable or even dominant.
- bob1029 4y ago
  > There could be exponential or quadratic scaling laws with any of these black boxes that makes one approach suddenly extremely viable or even dominant. The reason I like the CPU approach is the memory scaling is bonkers compared to GPU. You can buy a server that has 12TB of DRAM (in stock right now) for the cost of 1 of those H100 GPU systems. This is enough memory to hold over 3 trillion parameters with full 32-bit FP resolution. Employ some downsampling and you could get even more ridiculous. If 12TB isn't enough, you can always reach for things like RDMA and high speed interconnects. You could probably get 100 trillion parameters into 1 rack. At some point you'll need to add hierarchy to the SNN so that multiple racks & datacenters can work together. Imagine the power savings... It's not exactly a walk in the park, but those DIMMs are very eco friendly compared to GPUs. You don't need a whole lot of CPU cores in my proposal either. 8-16 very fast cores per box would probably be more than enough, looking at how fintech does things. 1 thread is actually running the entire show in my current prototype. The other threads are for spike timers & managing other external signals.
  - suby 4y ago
    Is your current prototype open source?
    - Ambix 4y ago
      Not the TS, but that's actually the same goal I have in mind with [0] project. Right now I'm building my homelab server which aimed to fit 1 TB RAM and 2 CPUs with ~100 cores total. It will cost like 0.1% of what I need to pay for GPU cluster with the same memory size :) [0] https://github.com/gotzmann/llama.go/ https://github.com/gotzmann/llama.go/
habitue 4y ago
> Why wouldn't we have already seen spiking networks contribute something by now? This presupposes we've explored this space thoroughly, and we haven't. When everything you do with NNs improves results (how it mostly is now), that means not enough people are trying out ideas and new things. I don't think you can invoke EMH-like reasoning quite yet. Give us a nice long winter like physics has had, and then we can use this heuristic.
- sdenton4 4y ago
  It's not strictly a question of trying Everything and seeing what sticks. There's a hypothesis in the parent comment - better handling of the time domain will lead to better modeling - which is actually fairly independent of architecture. So, there's going to be a number of possible ways to build better time modeling, ranging from tweaks to existing architecture to completely rebuilding Rome. So, if better time modeling really is a limitation, you don't need to rebuild Rome to find out. In fact, I might argue that S4 layers already provide this improved time handling in the current world, and is proving very successful, which would again widen the moat for SNNs.

forrest2 4y ago

I think people said the same thing about NNs in general before we hit a scale where they started performing magic. There could be exponential or quadratic scaling laws with any of these black boxes that makes one approach suddenly extremely viable or even dominant.

bob1029 4y ago
> There could be exponential or quadratic scaling laws with any of these black boxes that makes one approach suddenly extremely viable or even dominant. The reason I like the CPU approach is the memory scaling is bonkers compared to GPU. You can buy a server that has 12TB of DRAM (in stock right now) for the cost of 1 of those H100 GPU systems. This is enough memory to hold over 3 trillion parameters with full 32-bit FP resolution. Employ some downsampling and you could get even more ridiculous. If 12TB isn't enough, you can always reach for things like RDMA and high speed interconnects. You could probably get 100 trillion parameters into 1 rack. At some point you'll need to add hierarchy to the SNN so that multiple racks & datacenters can work together. Imagine the power savings... It's not exactly a walk in the park, but those DIMMs are very eco friendly compared to GPUs. You don't need a whole lot of CPU cores in my proposal either. 8-16 very fast cores per box would probably be more than enough, looking at how fintech does things. 1 thread is actually running the entire show in my current prototype. The other threads are for spike timers & managing other external signals.
- suby 4y ago
  Is your current prototype open source?
  - Ambix 4y ago
    Not the TS, but that's actually the same goal I have in mind with [0] project. Right now I'm building my homelab server which aimed to fit 1 TB RAM and 2 CPUs with ~100 cores total. It will cost like 0.1% of what I need to pay for GPU cluster with the same memory size :) [0] https://github.com/gotzmann/llama.go/ https://github.com/gotzmann/llama.go/

habitue 4y ago

> Why wouldn't we have already seen spiking networks contribute something by now? This presupposes we've explored this space thoroughly, and we haven't. When everything you do with NNs improves results (how it mostly is now), that means not enough people are trying out ideas and new things. I don't think you can invoke EMH-like reasoning quite yet. Give us a nice long winter like physics has had, and then we can use this heuristic.

sdenton4 4y ago
It's not strictly a question of trying Everything and seeing what sticks. There's a hypothesis in the parent comment - better handling of the time domain will lead to better modeling - which is actually fairly independent of architecture. So, there's going to be a number of possible ways to build better time modeling, ranging from tweaks to existing architecture to completely rebuilding Rome. So, if better time modeling really is a limitation, you don't need to rebuild Rome to find out. In fact, I might argue that S4 layers already provide this improved time handling in the current world, and is proving very successful, which would again widen the moat for SNNs.