4 ms·

Very cool, but I'm surprised you're sharing this on here and not in a job interview with a deep learning startup and/or an arXiv paper.

by _yb2s 2y ago

bob1029 2y ago
> a job interview with a deep learning startup I am not aware of any DL startups that are interested in techniques which are incompatible with GPUs and back propagation. I think this is a solo journey through the dark forest for now.
- inciampati 2y ago
  Eloquent, realistic, and sad. I'm living my own version of this (in biotech). The journey is the wonder.
- phyalow 2y ago
  Correct me if im wrong, this is just a question originating from my own curiosity, but could you not apply the same or similar techniques to classical NN architectures? I dont see exactly how the differentiability of the architecture matters? I.e. in between epochs randomly perturb weight values (similar effect I understand to introducing scheduled noise or non-isomorphic transforms on the features), implement a scheduled causal drop in/out scheme etc? I know of people that train models these ways as a form of regularisation and they claim better fits too. Enjoy your adventures in the Dark Forest.
  - bob1029 2y ago
    > I dont see exactly how the differentiability of the architecture matters? I don't know that classical architectures are infeasible to perturb as noted in other comments. I just think we have a really interesting knob to turn in this specific case that doesn't seem to exist in other architectures. How would you go about constraining the resource consumption (by up to half!) of a classical ANN while preserving realistic hopes that it will recover between training epochs? And, we are talking real resource reduction here not some academic thing. You can see the difference in task manager. That's actually the primary tool I've been using to determine how to tune the parameters of this scheme.
    - SuchAnonMuchWow 2y ago
      There has been some work to dynamically reduce the compute required by a network. See for example: https://arxiv.org/abs/2404.02258 https://arxiv.org/abs/2404.02258 They have a fixed compute budget which is lower than what the LLM need, and dynamically decide to allocate this compute budget to different part of the network. So its not exactly what you propose since here the compute budget is fixed (that's the point of the paper: to make the network learn how to allocate the resources by itself) but its dynamic for each part of the network, so it shows that its possible.
- sva_ 2y ago
  > I think this is a solo journey through the dark forest for now. In academia there's quite a bit of stuff on neuromorphic computing going on.
sethammons 2y ago
Your surprise breaks my brain. Why would someone "keep it to themselves" outside some white paper they won't have time to promote or for a job interview that they might not need? Is your thought that it is some novel approach that nobody is doing and has obvious market implications? Even then, the implications matter only if you are in a position to take advantage. The Internet used to be a place people shared cool shit because it was cool, not for some monetization scheme or personal gain beyond the joy of sharing. Content creators (as a job) and hustle culture are a blight.
- baxtr 2y ago
  They are sharing it here?! It’s not like OP is keeping it to themselves
- _yb2s 2y ago
  As I'm sure you're aware deep learning work better is the most hot and highly funded thing right now... there is nothing wrong with making sure you get the credit for a new technique, and it in no way precludes widespread free sharing of the idea. I am an academic researcher (not in this field) and everything I do is 100% free and public, but I still either patent or publish it, because getting the credit is what allows me to keep doing what I love. I actually hate that part, and would rather just do the work, but I already push that line too far sometimes, and risk losing my funding aka "publish or perish."
authorfly 2y ago
Respect to the OP as spiking networks are a different domain, but the field of Graceful Degradation and Perturbations itself is almost older than the field of Neural Networks itself. Generally, the bitter lesson occurs - these techniques show some kind of useful robustness at parameter M, but fail to show as much of a useful robustness relatively as parameter M*100. For example, before attention, using Drop out (in essence a form of forcing robustness) was one of the most common techniques. The quality of robustness or its resulting sub parts (e.g. the tradeoff between precision and recall) is incorporated as a measure in some of the fields basic scores, such as harmonic F1 score. Still it, like alternate activation functions, data beyond matrices and vectors and non-linear networks, remain very unstudied.

Very cool, but I'm surprised you're sharing this on here and not in a job interview with a deep learning startup and/or an arXiv paper.

by _yb2s 2y ago

Very cool, but I'm surprised you're sharing this on here and not in a job interview with a deep learning startup and/or an arXiv paper.

bob1029 2y ago

> a job interview with a deep learning startup I am not aware of any DL startups that are interested in techniques which are incompatible with GPUs and back propagation. I think this is a solo journey through the dark forest for now.

inciampati 2y ago
Eloquent, realistic, and sad. I'm living my own version of this (in biotech). The journey is the wonder.
phyalow 2y ago
Correct me if im wrong, this is just a question originating from my own curiosity, but could you not apply the same or similar techniques to classical NN architectures? I dont see exactly how the differentiability of the architecture matters? I.e. in between epochs randomly perturb weight values (similar effect I understand to introducing scheduled noise or non-isomorphic transforms on the features), implement a scheduled causal drop in/out scheme etc? I know of people that train models these ways as a form of regularisation and they claim better fits too. Enjoy your adventures in the Dark Forest.
- bob1029 2y ago
  > I dont see exactly how the differentiability of the architecture matters? I don't know that classical architectures are infeasible to perturb as noted in other comments. I just think we have a really interesting knob to turn in this specific case that doesn't seem to exist in other architectures. How would you go about constraining the resource consumption (by up to half!) of a classical ANN while preserving realistic hopes that it will recover between training epochs? And, we are talking real resource reduction here not some academic thing. You can see the difference in task manager. That's actually the primary tool I've been using to determine how to tune the parameters of this scheme.
  - SuchAnonMuchWow 2y ago
    There has been some work to dynamically reduce the compute required by a network. See for example: https://arxiv.org/abs/2404.02258 https://arxiv.org/abs/2404.02258 They have a fixed compute budget which is lower than what the LLM need, and dynamically decide to allocate this compute budget to different part of the network. So its not exactly what you propose since here the compute budget is fixed (that's the point of the paper: to make the network learn how to allocate the resources by itself) but its dynamic for each part of the network, so it shows that its possible.
sva_ 2y ago
> I think this is a solo journey through the dark forest for now. In academia there's quite a bit of stuff on neuromorphic computing going on.

sethammons 2y ago

Your surprise breaks my brain. Why would someone "keep it to themselves" outside some white paper they won't have time to promote or for a job interview that they might not need? Is your thought that it is some novel approach that nobody is doing and has obvious market implications? Even then, the implications matter only if you are in a position to take advantage. The Internet used to be a place people shared cool shit because it was cool, not for some monetization scheme or personal gain beyond the joy of sharing. Content creators (as a job) and hustle culture are a blight.

baxtr 2y ago
They are sharing it here?! It’s not like OP is keeping it to themselves
_yb2s 2y ago
As I'm sure you're aware deep learning work better is the most hot and highly funded thing right now... there is nothing wrong with making sure you get the credit for a new technique, and it in no way precludes widespread free sharing of the idea. I am an academic researcher (not in this field) and everything I do is 100% free and public, but I still either patent or publish it, because getting the credit is what allows me to keep doing what I love. I actually hate that part, and would rather just do the work, but I already push that line too far sometimes, and risk losing my funding aka "publish or perish."

authorfly 2y ago

Respect to the OP as spiking networks are a different domain, but the field of Graceful Degradation and Perturbations itself is almost older than the field of Neural Networks itself. Generally, the bitter lesson occurs - these techniques show some kind of useful robustness at parameter M, but fail to show as much of a useful robustness relatively as parameter M*100. For example, before attention, using Drop out (in essence a form of forcing robustness) was one of the most common techniques. The quality of robustness or its resulting sub parts (e.g. the tradeoff between precision and recall) is incorporated as a measure in some of the fields basic scores, such as harmonic F1 score. Still it, like alternate activation functions, data beyond matrices and vectors and non-linear networks, remain very unstudied.