3 ms·

I might argue that computer science theory is not fundamentally harder, so much as the available tooling is far more sophisticated. Not so many years ago, the

I might argue that computer science theory is not fundamentally harder, so much as the available tooling is far more sophisticated. Not so many years ago, the function for mean and median needed to be written rather than accessed within native libraries. Now, complex neural network architectures are available to all. Alas, simply gaining that access does not in any imply deeper understanding of what is occurring under the hood. Perhaps the higher technical demands is simply a byproduct of the more complex "off-the-shelf" solutions and the need to do more than simply pip install, point, and click?

mjburgess 5y ago
I'd say its more the opposite. The lack of much "academic & public understanding" of neural networks comes down to the severe lack of statistical modelling and empirical theory-building knowledge in the compu-sci community. NNs are not a hard thing to understand, it's just regression -- there's parameteric and non-parametric. And NNs are, just like any generic approximator algorithm, a largely non-parametric method. The lack of understanding of even what a parametric method is, in CS, is very telling: NB. it has nothing to do with the final model having parameters. It is whether the method assumes a parametrised distribution of its input data. Non-parameteric methods are well-understood, they aren't magic, and they aren't very hard to characterise. Rather, it is exactly in those areas where compu-sci people are most qualified that the mathematics gets the hardest. Much of the low-hanging fruit has been picked (Turing, et al.) and today comes the hard part of the thorniest problems.
- CrazyStat 5y ago
  > The lack of understanding of even what a parametric method is, in CS, is very telling: NB. it has nothing to do with the final model having parameters. It is whether the method assumes a parametrised distribution of its input data. Having done my PhD work on (Bayesian) nonparametric methods I'm struggling to parse this. What is the input data? Just the explanatory (independent) variables? Both explanatory and response (dependent) variables? Are we talking about a joint or conditional parametric distribution? Many parametric methods (e.g. OLS regression) make no assumption about the distribution of the explanatory variables. Many "nonparametric" methods do make parametric assumptions about the conditional distribution of the response variable(s) conditioned on the explanatory variable(s) (e.g. GP regression). I don't see how this works as a classification for whether a method is "nonparametric" or not.
  - mjburgess 5y ago
    So i'd put parametric vs. non-para on a scale. It can be done in terms of the final model parameters, even -- but this may seem initially weird. If the parameters of the predictive model are a weakly compressive function of the dataset, then your method is non-parametric. If your parameters are extremely compressive it's parametric. Subject to both being low-loss models of the data. Why? Well a non-parametric method is basically one which "uses the data points as its statistical model"; and a parametric method fits the data to a prior low-parameterised model. Eg., linear regression essentially fits the data to a normal distribution = parametric on the mean/std, ie., pdf(Y|X) = N(ax +b, stdev). You fit (a,b) and thus essentially are just "finding a mean". Eg., knn just remembers the dataset itself. So there's a scale from "linear regression to knn", ie., Weights = (Mean, Stdev)... to Weights = (X, Y). The terms parametric and non-parametric are fairly overloaded, so this way of characterising the distinction may either improve or worsen that. Either way, my point is that NN model with a very high parameter count is essentially best analysed as KNN on a weakly compressed feature space. In that sense it is an incredibly obvious and simple algorithm. Incidentally, NN can just be linear regression or KNN if you set it up appropriately. So NN is an alg. which runs "from knn to linear regression" depending on, eg., activation-fns, how hard you regularise it, etc.

mjburgess 5y ago

I'd say its more the opposite. The lack of much "academic & public understanding" of neural networks comes down to the severe lack of statistical modelling and empirical theory-building knowledge in the compu-sci community. NNs are not a hard thing to understand, it's just regression -- there's parameteric and non-parametric. And NNs are, just like any generic approximator algorithm, a largely non-parametric method. The lack of understanding of even what a parametric method is, in CS, is very telling: NB. it has nothing to do with the final model having parameters. It is whether the method assumes a parametrised distribution of its input data. Non-parameteric methods are well-understood, they aren't magic, and they aren't very hard to characterise. Rather, it is exactly in those areas where compu-sci people are most qualified that the mathematics gets the hardest. Much of the low-hanging fruit has been picked (Turing, et al.) and today comes the hard part of the thorniest problems.

CrazyStat 5y ago
> The lack of understanding of even what a parametric method is, in CS, is very telling: NB. it has nothing to do with the final model having parameters. It is whether the method assumes a parametrised distribution of its input data. Having done my PhD work on (Bayesian) nonparametric methods I'm struggling to parse this. What is the input data? Just the explanatory (independent) variables? Both explanatory and response (dependent) variables? Are we talking about a joint or conditional parametric distribution? Many parametric methods (e.g. OLS regression) make no assumption about the distribution of the explanatory variables. Many "nonparametric" methods do make parametric assumptions about the conditional distribution of the response variable(s) conditioned on the explanatory variable(s) (e.g. GP regression). I don't see how this works as a classification for whether a method is "nonparametric" or not.
- mjburgess 5y ago
  So i'd put parametric vs. non-para on a scale. It can be done in terms of the final model parameters, even -- but this may seem initially weird. If the parameters of the predictive model are a weakly compressive function of the dataset, then your method is non-parametric. If your parameters are extremely compressive it's parametric. Subject to both being low-loss models of the data. Why? Well a non-parametric method is basically one which "uses the data points as its statistical model"; and a parametric method fits the data to a prior low-parameterised model. Eg., linear regression essentially fits the data to a normal distribution = parametric on the mean/std, ie., pdf(Y|X) = N(ax +b, stdev). You fit (a,b) and thus essentially are just "finding a mean". Eg., knn just remembers the dataset itself. So there's a scale from "linear regression to knn", ie., Weights = (Mean, Stdev)... to Weights = (X, Y). The terms parametric and non-parametric are fairly overloaded, so this way of characterising the distinction may either improve or worsen that. Either way, my point is that NN model with a very high parameter count is essentially best analysed as KNN on a weakly compressed feature space. In that sense it is an incredibly obvious and simple algorithm. Incidentally, NN can just be linear regression or KNN if you set it up appropriately. So NN is an alg. which runs "from knn to linear regression" depending on, eg., activation-fns, how hard you regularise it, etc.