4 ms·

Not sure what "stabilization" means in this context. But isn't what they call sleeping basically regularization? In particular, mini-batch and drop-out??

ouid 6y ago
I don't know what you think regularization means, but there is no such thing as "just regularization". Not all regularization is equal, and different types of regularization combine in unpredictable ways. The regularization occurring in the context of this research is as follows. The model that they are creating is a sparse model. This means that, for the vast majority of possible inputs, the machine should produce no hypothesis whatsoever about what it is looking at. In fact, a good sparse representation only produces a representation when the input is in the space of valid inputs. If the model is given a valid input, it should come up with a representation of that input, if it is given garbage, it should prefer to not try. Signal is pretty easy to find, you just open your eyes, explore your environment, and update your weights accordingly. We would alternatively like to train the model to recognize when it is looking at nothing, but we have to be pretty careful about what we show it. If any signal is present at all, we definitely don't want to discourage our model from finding it. That was, after all, the point of the model in the first place. Fortunately, because of the size of the space, we can be certain that if we completely shut off the inputs and feed it noise instead, it is very very far from a valid input, and anything that it includes in its representation was included in error.
deleted 6y ago
[deleted]
jhanschoo 6y ago
It's important to note that what is being investigated are neuromorphic processors. i.e. attempts to model biological brains. A paragraph from the article directly addresses why conventional regularization isn't used. > “The issue of how to keep learning systems from becoming unstable really only arises when attempting to utilize biologically realistic, spiking neuromorphic processors or when trying to understand biology itself,” said Los Alamos computer scientist and study coauthor Garrett Kenyon. “The vast majority of machine learning, deep learning, and AI researchers never encounter this issue because in the very artificial systems they study they have the luxury of performing global mathematical operations that have the effect of regulating the overall dynamical gain of the system.”
- erikerikson 6y ago
  See also: https://en.wikipedia.org/wiki/Spiking_neural_network https://en.wikipedia.org/wiki/Spiking_neural_network https://en.m.wikipedia.org/wiki/Hopfield_network https://en.m.wikipedia.org/wiki/Hopfield_network

ouid 6y ago

I don't know what you think regularization means, but there is no such thing as "just regularization". Not all regularization is equal, and different types of regularization combine in unpredictable ways. The regularization occurring in the context of this research is as follows. The model that they are creating is a sparse model. This means that, for the vast majority of possible inputs, the machine should produce no hypothesis whatsoever about what it is looking at. In fact, a good sparse representation only produces a representation when the input is in the space of valid inputs. If the model is given a valid input, it should come up with a representation of that input, if it is given garbage, it should prefer to not try. Signal is pretty easy to find, you just open your eyes, explore your environment, and update your weights accordingly. We would alternatively like to train the model to recognize when it is looking at nothing, but we have to be pretty careful about what we show it. If any signal is present at all, we definitely don't want to discourage our model from finding it. That was, after all, the point of the model in the first place. Fortunately, because of the size of the space, we can be certain that if we completely shut off the inputs and feed it noise instead, it is very very far from a valid input, and anything that it includes in its representation was included in error.

deleted 6y ago

[deleted]

jhanschoo 6y ago

It's important to note that what is being investigated are neuromorphic processors. i.e. attempts to model biological brains. A paragraph from the article directly addresses why conventional regularization isn't used. > “The issue of how to keep learning systems from becoming unstable really only arises when attempting to utilize biologically realistic, spiking neuromorphic processors or when trying to understand biology itself,” said Los Alamos computer scientist and study coauthor Garrett Kenyon. “The vast majority of machine learning, deep learning, and AI researchers never encounter this issue because in the very artificial systems they study they have the luxury of performing global mathematical operations that have the effect of regulating the overall dynamical gain of the system.”

erikerikson 6y ago
See also: https://en.wikipedia.org/wiki/Spiking_neural_network https://en.wikipedia.org/wiki/Spiking_neural_network https://en.m.wikipedia.org/wiki/Hopfield_network https://en.m.wikipedia.org/wiki/Hopfield_network