Let's for a second assume that we bound the weights within [-1,1], considering that we are using floats to represent them which are a subset of real numbers, by

Let's for a second assume that we bound the weights within [-1,1], considering that we are using floats to represent them which are a subset of real numbers, by increasing the number of hidden neurons, you increase the class of models that you can select from, any single bit change in the weights means a different model. By increasing the class of models in your search space, you increase the number of models that perform good on training, but perform worse on test, i.e. underestimate true loss. Exactly because you select a single model and you have more models that underestimate the true (read test) error, you *may* be more likely to get models with worse performance on the test set compared to using a smaller network exactly because in the smaller network you are searching within a smaller distribution.