There are an infinite number of distributions that can fit the training data well (e.g., one that completely memorize the data and therefore replicate the frequ

There are an infinite number of distributions that can fit the training data well (e.g., one that completely memorize the data and therefore replicate the frequencies). The trick is to find the distributions that generalize well, and here the NN architecture is critical.