4 ms·
> ensemble of transformer models Isn't that just dropout?
by hackerlight 3y ago
> ensemble of transformer models
Isn't that just dropout?
- mikkom 3y agoNo. Why do you think so?
- hackerlight 3y agoGeoffrey Hinton describes dropout that way. It's like you're training different nets each time dropout changes.
- wenc 3y agoDropout is different from ensembles. It is a regularization method. It might look like an ensemble because you’re selecting different subsets but ensembles combine different independent models rather than just subset models.
- wenc 3y agoThat said random forests are an internal ensemble, so I guess that could work. In my mind an ensemble is like a committee. For it to be effective, each member should be independent (able to pick up different signals) and have a greater than random chance of being correct.
- hackerlight 3y agoI am aware it is not literally an ensemble model, but Geoffrey Hinton says it achieves the same thing conceptually and practically.