3 ms·

So instead of the mathematically sound approach of sampling from the categorical output distribution (that the model parameters were fitted to with maximum like

So instead of the mathematically sound approach of sampling from the categorical output distribution (that the model parameters were fitted to with maximum likelihood) you just inject some noise in the activations, and then pick the highest probability output?

gwern 3y ago
Injecting noise has many mathematically-sound interpretations, like the Bayesian interpretations of dropout for ensembling or posterior sampling.
zarzavat 3y ago
It’s a biologically inspired approach, hence the name. Biological neurons don’t work based on sampling functions, they have inherent non-determinism distributed throughout the network, and receptors to enable runtime modulation of the network’s function. This dynamic modulation is seemingly at odds with mathematical soundness. Whether mathematical soundness should be a goal or not is a question to be answered empirically.
- madamelic 3y ago
  Thank you for the explaination. I had to chew on this for a minute or two to put it together. This explaination helped a lot.