3 ms·

clicking through the neural networks api: ANEURALNETWORKS_RELU6 The output is calculated using this formula: output = min(6, max(0, input)) is that a joke/e

by singularity2001 9y ago

clicking through the neural networks api: ANEURALNETWORKS_RELU6 The output is calculated using this formula: output = min(6, max(0, input)) is that a joke/easter-egg? also: is there an official java wrapper for the neural networks ndk api?

mcintyre1994 9y ago
It's in TensorFlow: https://www.tensorflow.org/api_docs/python/tf/nn/relu6 https://www.tensorflow.org/api_docs/python/tf/nn/relu6 And even points to a paper: http://www.cs.utoronto.ca/~kriz/conv-cifar10-aug2010.pdf http://www.cs.utoronto.ca/~kriz/conv-cifar10-aug2010.pdf I'm not sure why they didn't make it a variable maximum though, or what the advantage is over multiplying a sigmoid by 6.
- clarle 9y ago
  There's an explanation here: https://stackoverflow.com/questions/47220595/why-the-6-in-relu6 https://stackoverflow.com/questions/47220595/why-the-6-in-re... It's an arbitrary value, but 6 fits best into 8 bits.
jamesonthecrow 9y ago
Not a joke / easter-egg. RELU6 is an activation function commonly used in deep convolutional neural networks. It comes up fairly often in mobile machine learning cases because it's used in Google's optimized MobileNet architecture and would cause errors when trying to convert to run on device. The original paper detailing the function is here (PDF warning): http://www.cs.utoronto.ca/~kriz/conv-cifar10-aug2010.pdf http://www.cs.utoronto.ca/~kriz/conv-cifar10-aug2010.pdf "Our ReLU units differ from those of [8] in two respects. First, we cap the units at 6, so our ReLU activation function is: y = min(max(x, 0), 6). In our tests, this encourages the model to learn sparse features earlier. In the formulation of [8], this is equivalent to imagining that each ReLU unit consists of only 6 replicated bias-shifted Bernoulli units, rather than an infinite amount. We will refer to ReLU units capped at n as ReLU-n units."

mcintyre1994 9y ago

It's in TensorFlow: https://www.tensorflow.org/api_docs/python/tf/nn/relu6 https://www.tensorflow.org/api_docs/python/tf/nn/relu6 And even points to a paper: http://www.cs.utoronto.ca/~kriz/conv-cifar10-aug2010.pdf http://www.cs.utoronto.ca/~kriz/conv-cifar10-aug2010.pdf I'm not sure why they didn't make it a variable maximum though, or what the advantage is over multiplying a sigmoid by 6.

clarle 9y ago
There's an explanation here: https://stackoverflow.com/questions/47220595/why-the-6-in-relu6 https://stackoverflow.com/questions/47220595/why-the-6-in-re... It's an arbitrary value, but 6 fits best into 8 bits.

jamesonthecrow 9y ago

Not a joke / easter-egg. RELU6 is an activation function commonly used in deep convolutional neural networks. It comes up fairly often in mobile machine learning cases because it's used in Google's optimized MobileNet architecture and would cause errors when trying to convert to run on device. The original paper detailing the function is here (PDF warning): http://www.cs.utoronto.ca/~kriz/conv-cifar10-aug2010.pdf http://www.cs.utoronto.ca/~kriz/conv-cifar10-aug2010.pdf "Our ReLU units differ from those of [8] in two respects. First, we cap the units at 6, so our ReLU activation function is: y = min(max(x, 0), 6). In our tests, this encourages the model to learn sparse features earlier. In the formulation of [8], this is equivalent to imagining that each ReLU unit consists of only 6 replicated bias-shifted Bernoulli units, rather than an infinite amount. We will refer to ReLU units capped at n as ReLU-n units."