3 ms·

> The lack of understanding of even what a parametric method is, in CS, is very telling: NB. it has nothing to do with the final model having parameters. It is

> The lack of understanding of even what a parametric method is, in CS, is very telling: NB. it has nothing to do with the final model having parameters. It is whether the method assumes a parametrised distribution of its input data. Having done my PhD work on (Bayesian) nonparametric methods I'm struggling to parse this. What is the input data? Just the explanatory (independent) variables? Both explanatory and response (dependent) variables? Are we talking about a joint or conditional parametric distribution? Many parametric methods (e.g. OLS regression) make no assumption about the distribution of the explanatory variables. Many "nonparametric" methods do make parametric assumptions about the conditional distribution of the response variable(s) conditioned on the explanatory variable(s) (e.g. GP regression). I don't see how this works as a classification for whether a method is "nonparametric" or not.

mjburgess 5y ago
So i'd put parametric vs. non-para on a scale. It can be done in terms of the final model parameters, even -- but this may seem initially weird. If the parameters of the predictive model are a weakly compressive function of the dataset, then your method is non-parametric. If your parameters are extremely compressive it's parametric. Subject to both being low-loss models of the data. Why? Well a non-parametric method is basically one which "uses the data points as its statistical model"; and a parametric method fits the data to a prior low-parameterised model. Eg., linear regression essentially fits the data to a normal distribution = parametric on the mean/std, ie., pdf(Y|X) = N(ax +b, stdev). You fit (a,b) and thus essentially are just "finding a mean". Eg., knn just remembers the dataset itself. So there's a scale from "linear regression to knn", ie., Weights = (Mean, Stdev)... to Weights = (X, Y). The terms parametric and non-parametric are fairly overloaded, so this way of characterising the distinction may either improve or worsen that. Either way, my point is that NN model with a very high parameter count is essentially best analysed as KNN on a weakly compressed feature space. In that sense it is an incredibly obvious and simple algorithm. Incidentally, NN can just be linear regression or KNN if you set it up appropriately. So NN is an alg. which runs "from knn to linear regression" depending on, eg., activation-fns, how hard you regularise it, etc.

mjburgess 5y ago

So i'd put parametric vs. non-para on a scale. It can be done in terms of the final model parameters, even -- but this may seem initially weird. If the parameters of the predictive model are a weakly compressive function of the dataset, then your method is non-parametric. If your parameters are extremely compressive it's parametric. Subject to both being low-loss models of the data. Why? Well a non-parametric method is basically one which "uses the data points as its statistical model"; and a parametric method fits the data to a prior low-parameterised model. Eg., linear regression essentially fits the data to a normal distribution = parametric on the mean/std, ie., pdf(Y|X) = N(ax +b, stdev). You fit (a,b) and thus essentially are just "finding a mean". Eg., knn just remembers the dataset itself. So there's a scale from "linear regression to knn", ie., Weights = (Mean, Stdev)... to Weights = (X, Y). The terms parametric and non-parametric are fairly overloaded, so this way of characterising the distinction may either improve or worsen that. Either way, my point is that NN model with a very high parameter count is essentially best analysed as KNN on a weakly compressed feature space. In that sense it is an incredibly obvious and simple algorithm. Incidentally, NN can just be linear regression or KNN if you set it up appropriately. So NN is an alg. which runs "from knn to linear regression" depending on, eg., activation-fns, how hard you regularise it, etc.