Regularisation methods like dropout are often good enough that you can build a network with too many parameters (for the amount of data you have) and rely upon

Regularisation methods like dropout are often good enough that you can build a network with too many parameters (for the amount of data you have) and rely upon the regularisation to find the subset of that network that is actually useful. People have recently got good results from also randomly dropping weights, or even whole layers.