4 ms·

More parameters is very exciting! I wonder if anyone here has read about what the relationship between the number of parameters on the network and the size of t

• jureso 10y ago
  Based on our current theoretical understanding the we should expect overfitting. In practice however, a very large number of parameters does not necessarily lead to overfitting. Clearly there is a gap between out theoretical understanding and practice. Check out a recent paper that explores exactly this questions: https://arxiv.org/abs/1611.03530 https://arxiv.org/abs/1611.03530
• nshazeer 10y ago
  I am one of the authors of "Outrageously Large Neural Networks". Yes - overfitting is a problem. We employed dropout to combat overfitting. Even with dropout, we found that adding additional capacity provides diminishing returns once the capacity of the network exceeds the number of examples in the training data (see sec. 5.2). To demonstrate significant gains from really large networks, we had to use huge datasets, up to 100 billion words.
  • petra 10y ago
    Impressive work, mate! Does mixture-of-experts works well the other way around, as a way to minimize power and hardware in common sized problems ? And would it work in low resolution networks, like BinaryConnect ?
• jamesblonde 10y ago
  A good rule of thumb is 1 sample per parameter. So if you have 100m parameters, you should have 100m samples.