4 ms·

I've found that intentionally causing abrupt (but reasonable) changes to hyperparameters in evolutionary spiking neural network simulations (I.e between each ge

I've found that intentionally causing abrupt (but reasonable) changes to hyperparameters in evolutionary spiking neural network simulations (I.e between each generation) results in far more robust simulations that will meet fitness criteria with less likelihood of getting stuck somewhere. The tradeoff being that simulating will take longer, but this may be worth things like reducing the chances of your resource requirements going asymptotic half way through. My current method of perturbation is to cap the total # of activations per candidate globally. When at least 3 improvements in the best global fitness score have been achieved, I cut the global activation limit in half. Obviously, this is fairly catastrophic for the current population. Each generation can increase the limit by 1% if no improvements made. This provides aggressive selection pressure for more efficient networks and forces a different kind of partial restart of the training process (I.e. do what you just did with 50% resources now). Very often this does result in optimization of the network. It also seems to make the networks rebound faster after restarts. At first, a restart can take 10-20 generations to recover. By generation 1000, this is closer to 1-2.

_yb2s 2y ago
Very cool, but I'm surprised you're sharing this on here and not in a job interview with a deep learning startup and/or an arXiv paper.
- bob1029 2y ago
  > a job interview with a deep learning startup I am not aware of any DL startups that are interested in techniques which are incompatible with GPUs and back propagation. I think this is a solo journey through the dark forest for now.
  - inciampati 2y ago
    Eloquent, realistic, and sad. I'm living my own version of this (in biotech). The journey is the wonder.
  - phyalow 2y ago
    Correct me if im wrong, this is just a question originating from my own curiosity, but could you not apply the same or similar techniques to classical NN architectures? I dont see exactly how the differentiability of the architecture matters? I.e. in between epochs randomly perturb weight values (similar effect I understand to introducing scheduled noise or non-isomorphic transforms on the features), implement a scheduled causal drop in/out scheme etc? I know of people that train models these ways as a form of regularisation and they claim better fits too. Enjoy your adventures in the Dark Forest.
    - bob1029 2y ago
      > I dont see exactly how the differentiability of the architecture matters? I don't know that classical architectures are infeasible to perturb as noted in other comments. I just think we have a really interesting knob to turn in this specific case that doesn't seem to exist in other architectures. How would you go about constraining the resource consumption (by up to half!) of a classical ANN while preserving realistic hopes that it will recover between training epochs? And, we are talking real resource reduction here not some academic thing. You can see the difference in task manager. That's actually the primary tool I've been using to determine how to tune the parameters of this scheme.
      - SuchAnonMuchWow 2y ago
        There has been some work to dynamically reduce the compute required by a network. See for example: https://arxiv.org/abs/2404.02258 https://arxiv.org/abs/2404.02258 They have a fixed compute budget which is lower than what the LLM need, and dynamically decide to allocate this compute budget to different part of the network. So its not exactly what you propose since here the compute budget is fixed (that's the point of the paper: to make the network learn how to allocate the resources by itself) but its dynamic for each part of the network, so it shows that its possible.
  - sva_ 2y ago
    > I think this is a solo journey through the dark forest for now. In academia there's quite a bit of stuff on neuromorphic computing going on.
- sethammons 2y ago
  Your surprise breaks my brain. Why would someone "keep it to themselves" outside some white paper they won't have time to promote or for a job interview that they might not need? Is your thought that it is some novel approach that nobody is doing and has obvious market implications? Even then, the implications matter only if you are in a position to take advantage. The Internet used to be a place people shared cool shit because it was cool, not for some monetization scheme or personal gain beyond the joy of sharing. Content creators (as a job) and hustle culture are a blight.
  - baxtr 2y ago
    They are sharing it here?! It’s not like OP is keeping it to themselves
  - _yb2s 2y ago
    As I'm sure you're aware deep learning work better is the most hot and highly funded thing right now... there is nothing wrong with making sure you get the credit for a new technique, and it in no way precludes widespread free sharing of the idea. I am an academic researcher (not in this field) and everything I do is 100% free and public, but I still either patent or publish it, because getting the credit is what allows me to keep doing what I love. I actually hate that part, and would rather just do the work, but I already push that line too far sometimes, and risk losing my funding aka "publish or perish."
- authorfly 2y ago
  Respect to the OP as spiking networks are a different domain, but the field of Graceful Degradation and Perturbations itself is almost older than the field of Neural Networks itself. Generally, the bitter lesson occurs - these techniques show some kind of useful robustness at parameter M, but fail to show as much of a useful robustness relatively as parameter M*100. For example, before attention, using Drop out (in essence a form of forcing robustness) was one of the most common techniques. The quality of robustness or its resulting sub parts (e.g. the tradeoff between precision and recall) is incorporated as a measure in some of the fields basic scores, such as harmonic F1 score. Still it, like alternate activation functions, data beyond matrices and vectors and non-linear networks, remain very unstudied.
JackeJR 2y ago
I am quite curious about your approach, Do you have an example code somewhere?
- bob1029 2y ago
  I'd start here: https://arxiv.org/abs/2406.12552 https://arxiv.org/abs/2406.12552 I will say that less is more and "The Bitter Lesson" applies here. Chasing biologically-inspired rabbits, such as STDP and LIF (see paper above/wikipedia), does seem to be a waste of time, especially when we have this thing entirely outside of biology that can arbitrarily replicate, serialize, mutate and simulate billions of instances of the same parent candidate in minutes-hours. Leaky charge carriers and inability to persist learned weights between candidate instantiations are limitations, not features to emulate. Imagine if you could be reincarnated with all of the exact knowledge you have today. Then clone that 1000 times and apply subtle mutations to it. Then, put these clones in an FFA arena and see who wins based upon a very well crafted fitness function. Then, do all of that over and over thousands of times per hour.
  - jcims 2y ago
    >Chasing biologically-inspired rabbits, such as STDP and LIF (see paper above/wikipedia), does seem to be a waste of time Unless physiological compatibility with a biological brain is considered an interesting endpoint? If you think about Neuralink for example, wouldn’t it be interesting if our brains could directly engage the model? Not just a translation layer but perceive it directly and natively though some kind of synaptic modem that converts analog exchanges of neurotransmitters to the synthetic network in the digital domain.
    - bob1029 2y ago
      > Unless physiological compatibility with a biological brain is considered an interesting endpoint? Excellent point. I am focused entirely on synthetic simulations without biological integration (for now). I think that could be an interesting next step - determining a way to modify a SNN that was trained in a synthetic time domain such that it can process real time signals. Training these things online is not really an option. You have to run a large # of offline simulations before you can find the right networks. Learning rules like STDP could theoretically address the online learning problem, but I couldn't find any traction down that path yet.
taylorius 2y ago
Firstly, I'm coming at this from the pint of view of standard feed forward NNs, rather than spiking neural networks, so apologies in advance if I am way off-base. By global activation limit, I assume you mean the total sum of neuron activation during one forward pass? So halving it would strongly encourage sparsity? How do you enforce the reduced limit? Just put it in the loss function? Or does the spiking nature allow you to "sparsify" the spiking by raising a threshold? Very very interesting stuff. I'm working with differential evolution, and incorporate correlation measures between candidates to (try to) allow for multiple divergent solutions, and it also feels fairly off the beaten track, compared to mainstream AI. But who cares, let's enjoy the dark forest! :-)
- bob1029 2y ago
  > By global activation limit, I assume you mean the total sum of neuron activation during one forward pass? Yes. For each candidate network I keep a counter of the total # of neuron activations. This figure is constrained by a global limit (which varies based on prior performance). We simply break out of the simulation loop once the allotted resources are exhauted. This means that more efficient networks are more likely to score higher.
seanhunter 2y ago
> I've found that intentionally causing abrupt (but reasonable) changes to hyperparameters in evolutionary spiking neural network simulations (I.e between each generation) results in far more robust simulations that will meet fitness criteria with less likelihood of getting stuck somewhere. If I understand you correctly this is similar to the philosophy behind simulated annealing isn't it? The idea being that adding some peturbation can be enough to jog the evolution out of local minima and you probably want to reduce the amount of those jogs over time so it ends up converging. https://en.wikipedia.org/wiki/Simulated_annealing https://en.wikipedia.org/wiki/Simulated_annealing
- neom 2y ago
  Not op but what they described is indeed sim annealing coupled with relaxing feedback loops. This approach would in theory result in prevention of stagnation, forced adaptation and over time, creating some form of resiliency. Pretty cool stuff!
- bob1029 2y ago
  I was actually thinking about simulated annealing as I got into this. I've found that reducing this specific mode of perturbation over time has not been necessary. The simulation will oscillate its way right into a perfect fitness score assuming everything else is good. Convergence seems to rely much more heavily on the quality of the fitness function and population management (i.e. size, selection criteria, mutation strategies). I apply random mutations to delays and weights, but using a spectrum of mutation each time (I.e. some clones are heavily mutated while others are barely touched). This spectra of mutation is constant all the way through the simulation.
lo0dot0 2y ago
Some of you may die... But that's a sacrifice i'm willing to make.

_yb2s 2y ago

Very cool, but I'm surprised you're sharing this on here and not in a job interview with a deep learning startup and/or an arXiv paper.

bob1029 2y ago
> a job interview with a deep learning startup I am not aware of any DL startups that are interested in techniques which are incompatible with GPUs and back propagation. I think this is a solo journey through the dark forest for now.
- inciampati 2y ago
  Eloquent, realistic, and sad. I'm living my own version of this (in biotech). The journey is the wonder.
- phyalow 2y ago
  Correct me if im wrong, this is just a question originating from my own curiosity, but could you not apply the same or similar techniques to classical NN architectures? I dont see exactly how the differentiability of the architecture matters? I.e. in between epochs randomly perturb weight values (similar effect I understand to introducing scheduled noise or non-isomorphic transforms on the features), implement a scheduled causal drop in/out scheme etc? I know of people that train models these ways as a form of regularisation and they claim better fits too. Enjoy your adventures in the Dark Forest.
  - bob1029 2y ago
    > I dont see exactly how the differentiability of the architecture matters? I don't know that classical architectures are infeasible to perturb as noted in other comments. I just think we have a really interesting knob to turn in this specific case that doesn't seem to exist in other architectures. How would you go about constraining the resource consumption (by up to half!) of a classical ANN while preserving realistic hopes that it will recover between training epochs? And, we are talking real resource reduction here not some academic thing. You can see the difference in task manager. That's actually the primary tool I've been using to determine how to tune the parameters of this scheme.
    - SuchAnonMuchWow 2y ago
      There has been some work to dynamically reduce the compute required by a network. See for example: https://arxiv.org/abs/2404.02258 https://arxiv.org/abs/2404.02258 They have a fixed compute budget which is lower than what the LLM need, and dynamically decide to allocate this compute budget to different part of the network. So its not exactly what you propose since here the compute budget is fixed (that's the point of the paper: to make the network learn how to allocate the resources by itself) but its dynamic for each part of the network, so it shows that its possible.
- sva_ 2y ago
  > I think this is a solo journey through the dark forest for now. In academia there's quite a bit of stuff on neuromorphic computing going on.
sethammons 2y ago
Your surprise breaks my brain. Why would someone "keep it to themselves" outside some white paper they won't have time to promote or for a job interview that they might not need? Is your thought that it is some novel approach that nobody is doing and has obvious market implications? Even then, the implications matter only if you are in a position to take advantage. The Internet used to be a place people shared cool shit because it was cool, not for some monetization scheme or personal gain beyond the joy of sharing. Content creators (as a job) and hustle culture are a blight.
- baxtr 2y ago
  They are sharing it here?! It’s not like OP is keeping it to themselves
- _yb2s 2y ago
  As I'm sure you're aware deep learning work better is the most hot and highly funded thing right now... there is nothing wrong with making sure you get the credit for a new technique, and it in no way precludes widespread free sharing of the idea. I am an academic researcher (not in this field) and everything I do is 100% free and public, but I still either patent or publish it, because getting the credit is what allows me to keep doing what I love. I actually hate that part, and would rather just do the work, but I already push that line too far sometimes, and risk losing my funding aka "publish or perish."
authorfly 2y ago
Respect to the OP as spiking networks are a different domain, but the field of Graceful Degradation and Perturbations itself is almost older than the field of Neural Networks itself. Generally, the bitter lesson occurs - these techniques show some kind of useful robustness at parameter M, but fail to show as much of a useful robustness relatively as parameter M*100. For example, before attention, using Drop out (in essence a form of forcing robustness) was one of the most common techniques. The quality of robustness or its resulting sub parts (e.g. the tradeoff between precision and recall) is incorporated as a measure in some of the fields basic scores, such as harmonic F1 score. Still it, like alternate activation functions, data beyond matrices and vectors and non-linear networks, remain very unstudied.

JackeJR 2y ago

I am quite curious about your approach, Do you have an example code somewhere?

bob1029 2y ago
I'd start here: https://arxiv.org/abs/2406.12552 https://arxiv.org/abs/2406.12552 I will say that less is more and "The Bitter Lesson" applies here. Chasing biologically-inspired rabbits, such as STDP and LIF (see paper above/wikipedia), does seem to be a waste of time, especially when we have this thing entirely outside of biology that can arbitrarily replicate, serialize, mutate and simulate billions of instances of the same parent candidate in minutes-hours. Leaky charge carriers and inability to persist learned weights between candidate instantiations are limitations, not features to emulate. Imagine if you could be reincarnated with all of the exact knowledge you have today. Then clone that 1000 times and apply subtle mutations to it. Then, put these clones in an FFA arena and see who wins based upon a very well crafted fitness function. Then, do all of that over and over thousands of times per hour.
- jcims 2y ago
  >Chasing biologically-inspired rabbits, such as STDP and LIF (see paper above/wikipedia), does seem to be a waste of time Unless physiological compatibility with a biological brain is considered an interesting endpoint? If you think about Neuralink for example, wouldn’t it be interesting if our brains could directly engage the model? Not just a translation layer but perceive it directly and natively though some kind of synaptic modem that converts analog exchanges of neurotransmitters to the synthetic network in the digital domain.
  - bob1029 2y ago
    > Unless physiological compatibility with a biological brain is considered an interesting endpoint? Excellent point. I am focused entirely on synthetic simulations without biological integration (for now). I think that could be an interesting next step - determining a way to modify a SNN that was trained in a synthetic time domain such that it can process real time signals. Training these things online is not really an option. You have to run a large # of offline simulations before you can find the right networks. Learning rules like STDP could theoretically address the online learning problem, but I couldn't find any traction down that path yet.

taylorius 2y ago

Firstly, I'm coming at this from the pint of view of standard feed forward NNs, rather than spiking neural networks, so apologies in advance if I am way off-base. By global activation limit, I assume you mean the total sum of neuron activation during one forward pass? So halving it would strongly encourage sparsity? How do you enforce the reduced limit? Just put it in the loss function? Or does the spiking nature allow you to "sparsify" the spiking by raising a threshold? Very very interesting stuff. I'm working with differential evolution, and incorporate correlation measures between candidates to (try to) allow for multiple divergent solutions, and it also feels fairly off the beaten track, compared to mainstream AI. But who cares, let's enjoy the dark forest! :-)

bob1029 2y ago
> By global activation limit, I assume you mean the total sum of neuron activation during one forward pass? Yes. For each candidate network I keep a counter of the total # of neuron activations. This figure is constrained by a global limit (which varies based on prior performance). We simply break out of the simulation loop once the allotted resources are exhauted. This means that more efficient networks are more likely to score higher.

seanhunter 2y ago

> I've found that intentionally causing abrupt (but reasonable) changes to hyperparameters in evolutionary spiking neural network simulations (I.e between each generation) results in far more robust simulations that will meet fitness criteria with less likelihood of getting stuck somewhere. If I understand you correctly this is similar to the philosophy behind simulated annealing isn't it? The idea being that adding some peturbation can be enough to jog the evolution out of local minima and you probably want to reduce the amount of those jogs over time so it ends up converging. https://en.wikipedia.org/wiki/Simulated_annealing https://en.wikipedia.org/wiki/Simulated_annealing

neom 2y ago
Not op but what they described is indeed sim annealing coupled with relaxing feedback loops. This approach would in theory result in prevention of stagnation, forced adaptation and over time, creating some form of resiliency. Pretty cool stuff!
bob1029 2y ago
I was actually thinking about simulated annealing as I got into this. I've found that reducing this specific mode of perturbation over time has not been necessary. The simulation will oscillate its way right into a perfect fitness score assuming everything else is good. Convergence seems to rely much more heavily on the quality of the fitness function and population management (i.e. size, selection criteria, mutation strategies). I apply random mutations to delays and weights, but using a spectrum of mutation each time (I.e. some clones are heavily mutated while others are barely touched). This spectra of mutation is constant all the way through the simulation.

lo0dot0 2y ago

Some of you may die... But that's a sacrifice i'm willing to make.