3 ms·

I thought nonlinearity was very important to be able to make a larger model better than a smaller one? Like so important that tom7 made a half-joke demo with it

I thought nonlinearity was very important to be able to make a larger model better than a smaller one? Like so important that tom7 made a half-joke demo with it: https://yewtu.be/watch?v=Ae9EKCyI1xU https://yewtu.be/watch?v=Ae9EKCyI1xU

epgui 3y ago
Linear models don’t need everything to be linear.
- iamcreasy 3y ago
  I presume you are implying that linear model only mandates linear relationship between predictor and regression coefficients?
  - stdbrouw 3y ago
    A linear relationship between any transformation of the outcome and any transformation of the predictor variables — so the function is linear but the relationship between predictors and outcome can take on almost any shape.
    - iamcreasy 3y ago
      Ah, I missed 'the transformation of outcome' in my mind. Thanks for clearing it up.
  - hackerlight 3y ago
    Linear models are a linear combination of possibly non-linear regressors. The linearity is strictly in the parameters, not in whatever you're adding up. A neural network can be pedantically referred to as a linear model of the form y = a + b*neural_network, for example. Here, y is a linear model (even though neural_network isn't).
    - epgui 3y ago
      IMO it's not very pedantic... It's pretty much exactly what it is! (I'm not too sure about the example equation you give, however)
dist-epoch 3y ago
Well, you can create a non-linear model by piece-wise combining multiple linear models. The famous ReLU non-linearity is just that - two linear functions joined.
- nyrikki 3y ago
  same thing with any feed forward network too. They are all piece-wise linear in respect to inputs. Layers reduce resource requirements and make some patterns easier or even practical to find, but any ANN that is a FNN supervised learning could be represented as a parametric linear regression. Unsupervised learning, that tends to use clustering is harder to visualize but is the same thing. You still have ANNs, which have binary output, which can be viewed through the lens of deciders. They have to have unique successor and predecessor functions. Really this is just set shattering that relates to a finite VC dimensionality being required for something to be PAC learnable. But the title of this is confusing the map for the territory. It isn't that 'Everything is a linear model' but that linear models are the preferred, most practical form. The efforts to leverage spikey neutral networks, which is a more realistic model of cortical neurons, and which have continuous output (or more correctly the computable reals) tend to run into problems like riddled basins. https://arxiv.org/abs/1711.02160 https://arxiv.org/abs/1711.02160 Obviously setting rectified linear unit at 0 = 1 resolves to differentiation problem, but many functions may not be so simple Perhaps a useful lens is how TSP with a discreet Euclidean metric is in NP-complete while the continuous version is in NP-hard. But it isn't that everything is linearizable, but rather that linearized problems tend to be the most practical.
- btdmaster 3y ago
  I see what you mean. Though in my mind, and this is clearly subjective, piece-wise linear is at least less strict than a linear model. (With enough ReLUs, you could get arbitrarily close to a lookup table, which I think would be best described as a nonlinear model.)
tnecniv 3y ago
Nonlinear things start looking like linear things again in very high dimensions
- nyrikki 3y ago
  Only when you dimensions are truly independent and that is a stretch. Really what you are saying is that you are more likely to find a field for your problem, and fields don't exist in more than 2 dimensions. Consider Predator Pray with fear and refuge, which is indeterminate, and not due to a lack of precision but a topological feature where ≥3 open sets share the same boundary set. https://www.sciencedirect.com/science/article/abs/pii/S0960077922010128 https://www.sciencedirect.com/science/article/abs/pii/S09600... General relativity, with 3 spacial and one temporal dimension is another. One lens to consider this is that rotations are hyperbolic due to the lack of independence from the time dimension. Quantum mechanics would have been much more difficult if it didn't have two exit basins. Which is similar to ANNs and linear regressions being binary output. (Some exceptions will orthogonal dimensions like EM)

epgui 3y ago

Linear models don’t need everything to be linear.

iamcreasy 3y ago
I presume you are implying that linear model only mandates linear relationship between predictor and regression coefficients?
- stdbrouw 3y ago
  A linear relationship between any transformation of the outcome and any transformation of the predictor variables — so the function is linear but the relationship between predictors and outcome can take on almost any shape.
  - iamcreasy 3y ago
    Ah, I missed 'the transformation of outcome' in my mind. Thanks for clearing it up.
- hackerlight 3y ago
  Linear models are a linear combination of possibly non-linear regressors. The linearity is strictly in the parameters, not in whatever you're adding up. A neural network can be pedantically referred to as a linear model of the form y = a + b*neural_network, for example. Here, y is a linear model (even though neural_network isn't).
  - epgui 3y ago
    IMO it's not very pedantic... It's pretty much exactly what it is! (I'm not too sure about the example equation you give, however)

dist-epoch 3y ago

Well, you can create a non-linear model by piece-wise combining multiple linear models. The famous ReLU non-linearity is just that - two linear functions joined.

nyrikki 3y ago
same thing with any feed forward network too. They are all piece-wise linear in respect to inputs. Layers reduce resource requirements and make some patterns easier or even practical to find, but any ANN that is a FNN supervised learning could be represented as a parametric linear regression. Unsupervised learning, that tends to use clustering is harder to visualize but is the same thing. You still have ANNs, which have binary output, which can be viewed through the lens of deciders. They have to have unique successor and predecessor functions. Really this is just set shattering that relates to a finite VC dimensionality being required for something to be PAC learnable. But the title of this is confusing the map for the territory. It isn't that 'Everything is a linear model' but that linear models are the preferred, most practical form. The efforts to leverage spikey neutral networks, which is a more realistic model of cortical neurons, and which have continuous output (or more correctly the computable reals) tend to run into problems like riddled basins. https://arxiv.org/abs/1711.02160 https://arxiv.org/abs/1711.02160 Obviously setting rectified linear unit at 0 = 1 resolves to differentiation problem, but many functions may not be so simple Perhaps a useful lens is how TSP with a discreet Euclidean metric is in NP-complete while the continuous version is in NP-hard. But it isn't that everything is linearizable, but rather that linearized problems tend to be the most practical.
btdmaster 3y ago
I see what you mean. Though in my mind, and this is clearly subjective, piece-wise linear is at least less strict than a linear model. (With enough ReLUs, you could get arbitrarily close to a lookup table, which I think would be best described as a nonlinear model.)

tnecniv 3y ago

Nonlinear things start looking like linear things again in very high dimensions

nyrikki 3y ago
Only when you dimensions are truly independent and that is a stretch. Really what you are saying is that you are more likely to find a field for your problem, and fields don't exist in more than 2 dimensions. Consider Predator Pray with fear and refuge, which is indeterminate, and not due to a lack of precision but a topological feature where ≥3 open sets share the same boundary set. https://www.sciencedirect.com/science/article/abs/pii/S0960077922010128 https://www.sciencedirect.com/science/article/abs/pii/S09600... General relativity, with 3 spacial and one temporal dimension is another. One lens to consider this is that rotations are hyperbolic due to the lack of independence from the time dimension. Quantum mechanics would have been much more difficult if it didn't have two exit basins. Which is similar to ANNs and linear regressions being binary output. (Some exceptions will orthogonal dimensions like EM)