12 ms·
You can choose the prior according to any selection rule that does not see the data (actually, you can do more, but justifying this is the realm of empirical Ba
by hodgehog11 18d ago
You can choose the prior according to any selection rule that does not see the data (actually, you can do more, but justifying this is the realm of empirical Bayes and requires some more precise arguments). In this case, you can choose it according to the model size and provided that your Jacobian is full rank, you will get increasing marginal likelihood.
- srean 18d agoWhat threw me off was the (possibly misunderstood) suggestion for minimizing the generalization bound over the prior after the data has been incorporated.
- hodgehog11 18d agoAh, sorry for the misunderstanding, I can see how my comment reads that way. That is done in the Gaussian process context, not in my first example, and yes, it's a dirty idea, but you can justify it using differential privacy arguments (basically you are optimizing few parameters and these do not have full interaction with the data).
- srean 18d agoYeah, I had read one of your parallel comments and understood what you had meant. Differential privacy is a good formulation (well, the only one I know) to deal with the peeking problem in general.