3 ms·

Can someone ELI5 how one increases the size of an NN? If I take the handwritten digit classifier that people use as ML 101, is it just a matter of increasing th

Can someone ELI5 how one increases the size of an NN? If I take the handwritten digit classifier that people use as ML 101, is it just a matter of increasing the size of the hidden layers?

malka 5y ago
Yes, or add more layers
joshvm 5y ago
EfficientNet is a nice simple example of this. It's family of networks with different sizes, controlled by a single scaling parameter. That parameter controls depth, width and resolution - see here for more details, the figure is easy to understand: https://paperswithcode.com/method/efficientnet https://paperswithcode.com/method/efficientnet
teebs 5y ago
You can just do that by adding large hidden layers or more hidden layers, up to a point. But eventually the signal that's coming from the input data is so diluted through all the neurons and layers that your models stop performing better as you add more neurons. Many of the advances in NNs come from structuring the neurons in particular ways - for example, in computer vision, convolutional neural networks. These are kind of a small neural network that looks at each part of the image and gives an output - basically, shrinking and summarizing the image - which is then fed into another small neural network that shrinks & summarizes some more, and so on until you have only one result, like "is this a cat" or something like that. Transformers, which are mostly used for natural language processing, have small neural network layers that let the network figure out which other words in a text are relevant for understanding a given word (or, token, really). These help simplify the problem for the NN so it doesn't have to sort through all the data it has at once, which is a bottleneck on scaling NNs. Not sure that explains it like you're 5, but hopefully it addresses your question

3 ms·

Can someone ELI5 how one increases the size of an NN? If I take the handwritten digit classifier that people use as ML 101, is it just a matter of increasing th

by yalogin 5y ago

Can someone ELI5 how one increases the size of an NN? If I take the handwritten digit classifier that people use as ML 101, is it just a matter of increasing the size of the hidden layers?

malka 5y ago
Yes, or add more layers
joshvm 5y ago
EfficientNet is a nice simple example of this. It's family of networks with different sizes, controlled by a single scaling parameter. That parameter controls depth, width and resolution - see here for more details, the figure is easy to understand: https://paperswithcode.com/method/efficientnet https://paperswithcode.com/method/efficientnet
teebs 5y ago
You can just do that by adding large hidden layers or more hidden layers, up to a point. But eventually the signal that's coming from the input data is so diluted through all the neurons and layers that your models stop performing better as you add more neurons. Many of the advances in NNs come from structuring the neurons in particular ways - for example, in computer vision, convolutional neural networks. These are kind of a small neural network that looks at each part of the image and gives an output - basically, shrinking and summarizing the image - which is then fed into another small neural network that shrinks & summarizes some more, and so on until you have only one result, like "is this a cat" or something like that. Transformers, which are mostly used for natural language processing, have small neural network layers that let the network figure out which other words in a text are relevant for understanding a given word (or, token, really). These help simplify the problem for the NN so it doesn't have to sort through all the data it has at once, which is a bottleneck on scaling NNs. Not sure that explains it like you're 5, but hopefully it addresses your question