5 ms·

Serious (and likely ignorant) question - what does linearity have to do with anything here? linear over what and why does non-linearity make something 'unpredic

Serious (and likely ignorant) question - what does linearity have to do with anything here? linear over what and why does non-linearity make something 'unpredictable'?

tylerhou 6y ago
I assume they are using non-linear to mean non-continuous, which implies that there can be large, hard-to-understand changes in behavior when the input is changed only a small amount.
- kccqzy 6y ago
  Polynomials with large degrees are continuous. It's just that they can still change by a large amount (i.e. having a large derivative) when the input is changed by a small amount. I invite you to construct the Lagrange polynomial (i.e. interpolating polynomial) for points on a nice, simple curve with some noise. They will, by definition, pass through every point given, and yet it will likely behave very badly outside the range of the given points.
  - tylerhou 6y ago
    There is nothing wrong with using a non-linear model, though; x^2 or x^3 regressions make sense on many datasets. Non-continuous is also not the perfect terminology, but I argue that it is more precise than non-linear: the chief idea being that the model "changes unpredictably."
    - kccqzy 6y ago
      Sure you can argue things however you want, if you also decide to ignore hundreds of years of mathematical terminology.
cjhanks 6y ago
If you have ever opened up Excel or a similar program. One of the more useful options is to generate a regression line-fit on your data points. One option is to specify a polynomial function, you can specify how many coefficients you want. One of the measurements is the mean-squared-error between the line-fit and the points. You can add as many polynomial coefficients as you want, and you will be able to decrease the mean squared error. But the more polynomial's you choose, two things will be true: 1. The line-fit will be far more likely to go through the points. 2. At points in the line where there was no data, the line will less approximate the underlying physical reality. That same mathematical property is what is relevant here. There is nothing inherently evil about non-linearity, when the non-linear math model properly maps to the physical reality. But when you over fit a line, many of the functional solutions may be completely wrong.
- EE84M3i 6y ago
  I'm confused. I agree that overfitting can lead to very bad models. But, what I don't understand is that I thought that "linear" in ML contexts was normally used in the sense of 'linear transformations', which is a sense of linear that 'line-fit' from excel isn't -- it's affine. Is a linear model with thousands/millions of weights/parameters (like deep learning models) really substantially simpler to understand? Can it do anything useful? [1]: https://en.wikipedia.org/wiki/Linear_map https://en.wikipedia.org/wiki/Linear_map
  - cjhanks 6y ago
    I suppose from the perspective of someone implementing these models, yeah - it is linear, but it is not bijective. In a system with only one layer, that manifests as an alias (assuming the output dimensions are smaller). In a system with multiple layers of either `N->M` or `M->N`, those aliases tend to manifest as apparent "non-linearities". So, I guess looking from the bottom up the system may look non-continuous and linear. But if you look from the top down, it would look continuous and non-linear. Really, I am not sure which one is "true".
YeGoblynQueenne 6y ago
Linear models have more bias, so they represent current data less well and are more predictive of future, unseen data (think of a straight line through a point cloud). Non-linear models have more variance so they represent current data better and are less predictive of future, unseen data (think of a line snaking around a point cloud). An added complication is that deep neural net models are, in practice, vectors (or, well, tensors) of numbers so they are difficult to interpret. This and their extreme variance makes it hard to know how they will behave in the future.
- perl4ever 6y ago
  I'm not good at math, but I'm confused by the association of AI with non-linear stuff, setting aside the association of non-linear with "bad". I thought ML involved linear algebra or something (says xkcd!) which would presumably be...linear?
  - klipt 6y ago
    The underlying derivatives are linear (like all derivatives) but neural networks' ability to approximate arbitrary non linear functions is one of their biggest strengths.
    - perl4ever 6y ago
      Yes, so I'm left wondering, when making the association of the math to the badness, how do you decide if the linearity or the non-linearity is the salient part?
      - coldtea 6y ago
        Mathematically, you can think of "linear" AI problems as "easy to solve", and non-linear as "difficult". That's part of what the parent means. Some function being linear means it's easier to guess. If a real world phenomenon is tied to a linear function, then it's easy for AI to guess/approximate.
  - 6y ago

Serious (and likely ignorant) question - what does linearity have to do with anything here? linear over what and why does non-linearity make something 'unpredictable'?

tylerhou 6y ago

I assume they are using non-linear to mean non-continuous, which implies that there can be large, hard-to-understand changes in behavior when the input is changed only a small amount.

kccqzy 6y ago
Polynomials with large degrees are continuous. It's just that they can still change by a large amount (i.e. having a large derivative) when the input is changed by a small amount. I invite you to construct the Lagrange polynomial (i.e. interpolating polynomial) for points on a nice, simple curve with some noise. They will, by definition, pass through every point given, and yet it will likely behave very badly outside the range of the given points.
- tylerhou 6y ago
  There is nothing wrong with using a non-linear model, though; x^2 or x^3 regressions make sense on many datasets. Non-continuous is also not the perfect terminology, but I argue that it is more precise than non-linear: the chief idea being that the model "changes unpredictably."
  - kccqzy 6y ago
    Sure you can argue things however you want, if you also decide to ignore hundreds of years of mathematical terminology.

cjhanks 6y ago

If you have ever opened up Excel or a similar program. One of the more useful options is to generate a regression line-fit on your data points. One option is to specify a polynomial function, you can specify how many coefficients you want. One of the measurements is the mean-squared-error between the line-fit and the points. You can add as many polynomial coefficients as you want, and you will be able to decrease the mean squared error. But the more polynomial's you choose, two things will be true: 1. The line-fit will be far more likely to go through the points. 2. At points in the line where there was no data, the line will less approximate the underlying physical reality. That same mathematical property is what is relevant here. There is nothing inherently evil about non-linearity, when the non-linear math model properly maps to the physical reality. But when you over fit a line, many of the functional solutions may be completely wrong.

EE84M3i 6y ago
I'm confused. I agree that overfitting can lead to very bad models. But, what I don't understand is that I thought that "linear" in ML contexts was normally used in the sense of 'linear transformations', which is a sense of linear that 'line-fit' from excel isn't -- it's affine. Is a linear model with thousands/millions of weights/parameters (like deep learning models) really substantially simpler to understand? Can it do anything useful? [1]: https://en.wikipedia.org/wiki/Linear_map https://en.wikipedia.org/wiki/Linear_map
- cjhanks 6y ago
  I suppose from the perspective of someone implementing these models, yeah - it is linear, but it is not bijective. In a system with only one layer, that manifests as an alias (assuming the output dimensions are smaller). In a system with multiple layers of either `N->M` or `M->N`, those aliases tend to manifest as apparent "non-linearities". So, I guess looking from the bottom up the system may look non-continuous and linear. But if you look from the top down, it would look continuous and non-linear. Really, I am not sure which one is "true".

YeGoblynQueenne 6y ago

Linear models have more bias, so they represent current data less well and are more predictive of future, unseen data (think of a straight line through a point cloud). Non-linear models have more variance so they represent current data better and are less predictive of future, unseen data (think of a line snaking around a point cloud). An added complication is that deep neural net models are, in practice, vectors (or, well, tensors) of numbers so they are difficult to interpret. This and their extreme variance makes it hard to know how they will behave in the future.

perl4ever 6y ago
I'm not good at math, but I'm confused by the association of AI with non-linear stuff, setting aside the association of non-linear with "bad". I thought ML involved linear algebra or something (says xkcd!) which would presumably be...linear?
- klipt 6y ago
  The underlying derivatives are linear (like all derivatives) but neural networks' ability to approximate arbitrary non linear functions is one of their biggest strengths.
  - perl4ever 6y ago
    Yes, so I'm left wondering, when making the association of the math to the badness, how do you decide if the linearity or the non-linearity is the salient part?
    - coldtea 6y ago
      Mathematically, you can think of "linear" AI problems as "easy to solve", and non-linear as "difficult". That's part of what the parent means. Some function being linear means it's easier to guess. If a real world phenomenon is tied to a linear function, then it's easy for AI to guess/approximate.
- 6y ago