4 ms·

I see a lot of comments here assuming "linear model" means "can't model nonlinearities." Absolutely not the case. Splines can easily take care of that. The "lin

I see a lot of comments here assuming "linear model" means "can't model nonlinearities." Absolutely not the case. Splines can easily take care of that. The "linear" part of linear model just means "linear in the predictor space." You can add a non-linear predictor easily via spline basis (similar/sometimes identical to "kernels" in ML). My series of lm/glm/gam/gamm revelations was: 1. All t-tests and ANOVA flavors are just linear models 2. Linear models are just a special case of generalized linear models (GLMs), which can deal with binary or count data too 3. All linear models and GLMs are just special cases of generalized linear mixed models, which can deal with repeated measures, grouped data, and other non-iid clustering 4. Linearity is usually a bad assumption, which can easily be overcome via splines 5. Estimating a spline smoothing penalty is the same thing as estimating the variance for a random effect in a a mixed model, so #3 and #4 can be combined for free And then you end up with a generalized additive mixed model (GAMM), which can model smooth nonlinear functions of many variables, smooth interaction surfaces between variables (e.g. latitude/longitude), handle repeated measurements and grouping, and deals with many types of outcomes, including continuous, binary yes/no, count, ordinal categories, or survival time measurements. All while yielding statistically valid confidence intervals, and typically only taking a few minutes of CPU time even on datasets with hundreds of thousands / millions of datapoints.

carlthome 3y ago
This sounds really cool but was hard to digest for me as a ML Engineer who came into work just around deep learning and DNNs. Is there some go-to practice material I could look at? Splines I haven't touched since numerical computing exercises in school.
- goosedragons 3y ago
  Simon Wood's Generalized Additive Model book.
  - levocardia 3y ago
    Or, for a more approachable treatment, Semiparametric Regression with R by Harezlak, Ruppert, and Wand. A middle ground between Wood's book (which is comprehensive but can dip into math that's way over my head at times) and H/R/W is Semiparametric Regression by Ruppert, Wand, and Carroll. I have also heard great things about Frank Harrell's Regression Modeling Strategies which uses a slightly different approach (still spline-based though), but I haven't read it. His other writing is fantastic though.
    - disgruntledphd2 3y ago
      > Frank Harrell's Regression Modeling Strategies which uses a slightly different approach (still spline-based though), but I haven't read it. Very very good indeed, with the exception that he basically ignores compute time and efficiency. I learned a bunch, but applying his approach to the kinds of datasets I deal with (much larger and with pretty strict compute budgets) was very difficult.
- levocardia 3y ago
  If you know DNNs, you'll find the introduction of splines in Semiparametric Regression in R very intuitive - splines are introduced using what the authors call a "truncated line basis" but you already know it as the RELU function, just with a bias. Indeed, even the penalization of splines will look extremely familiar: it's basically just L2 regularization to induce smoothness. You might also enjoy reading the "Neural Additive Model" paper from Hinton's lab, which is basically GAMs using a separate DNN as a "spline basis" for each input variable.
deleted 3y ago
[deleted]
idiotsecant 3y ago
Widely known in the controls engineering world for the last hundred years as 'anything is linear if you zoom in far enough'
- vitus 3y ago
  Well, subject to certain constraints. You aren't guaranteed that your equilibria behave the same in the linearization of a nonlinear system if your Jacobian has any eigenvalues with real part of 0. https://en.wikipedia.org/wiki/Hartman%E2%80%93Grobman_theorem https://en.wikipedia.org/wiki/Hartman%E2%80%93Grobman_theore...
- new_username01 3y ago
  It's amazing how many ideas I've seen being "rediscovered" these days from control theory.
- kqr 3y ago
  Wait isn't that just an application of Taylor expansion which analysis has known about for several hundreds of years?
  - f1shy 3y ago
    Taylor is only for some functions, must be continuos and differentiable to n grades. In control theory you break the functions, so discontinuities can also be considered.
- rcfox 3y ago
  Alternatively, anything is linear if you add enough dimensions.
mjcohen 3y ago
In the 70's in CS grad school at USC, I wrote an adaptive least squares cubic spline fit routine. Kept subdividing intervals and fitting a least squares cubic spline in each interval until a criteria was met. Fun times. Don't have the code or writeup any more.
EvgeniyZh 3y ago
"You can make linear model non linear by adding non linearity" is kind of vacuous statement, isn't it
- ivansavz 3y ago
  No no, linear just means some expression of the form y = b1*x1 + b2*x2 + ... + bp*xp which is in the heart of the model (perhaps more specific description would be linear combination). Whatever you call it, it's just some quantity y, that is constructed via an additive process from components x1...xp, and each component is multiplied by some constant (the coefficient of the linear combination). This linear combination "core" of the model can be used directly (in which case it is geometrically a hyperplane), but you could also use non-linear transformation of the inputs or the outputs. E.g. x1 could be a spline basis function, or log-transformed data, or anything. Similarly, the output y can be passed though some non-linearity to force a particular output (like in logistic regression). Hence the statement is not vacuous
  - EvgeniyZh 3y ago
    Well yeah, but what is the non trivial statement here? Linear model is linear, and if you add to your model non-linear steps (such as pre- or post- processing) then it is not longer linear and can handle non-linear data. In the same spirit, you could say that GPT4 is a linear model (token logits) with a very large preprocessing (transformer). Maybe what OP meant is that particular non linearities (splines) allow to keep some goodies from the linear model in non-linear settings. Still it's not clear for me in which settings exactly such models are good enough
2-718-281-828 3y ago
> Splines can easily take care of that. within set boundaries.
usgroup 3y ago
GLMs are a non linear transformation on an output followed by linear modelling. They are referred to as “linear models” but you might as well then also consider NNs as linear models, or any model at all which ends with addition as the final step.
fho 3y ago
Can you recommend some resources to go from step 3 to 4? So far I have been successfully using GLMMs where appropriate, but then jumped to implementing completely arbitrary models by fitting them to the data (plus bootstrapping). But that takes ages for most problems as you say

carlthome 3y ago

This sounds really cool but was hard to digest for me as a ML Engineer who came into work just around deep learning and DNNs. Is there some go-to practice material I could look at? Splines I haven't touched since numerical computing exercises in school.

goosedragons 3y ago
Simon Wood's Generalized Additive Model book.
- levocardia 3y ago
  Or, for a more approachable treatment, Semiparametric Regression with R by Harezlak, Ruppert, and Wand. A middle ground between Wood's book (which is comprehensive but can dip into math that's way over my head at times) and H/R/W is Semiparametric Regression by Ruppert, Wand, and Carroll. I have also heard great things about Frank Harrell's Regression Modeling Strategies which uses a slightly different approach (still spline-based though), but I haven't read it. His other writing is fantastic though.
  - disgruntledphd2 3y ago
    > Frank Harrell's Regression Modeling Strategies which uses a slightly different approach (still spline-based though), but I haven't read it. Very very good indeed, with the exception that he basically ignores compute time and efficiency. I learned a bunch, but applying his approach to the kinds of datasets I deal with (much larger and with pretty strict compute budgets) was very difficult.
levocardia 3y ago
If you know DNNs, you'll find the introduction of splines in Semiparametric Regression in R very intuitive - splines are introduced using what the authors call a "truncated line basis" but you already know it as the RELU function, just with a bias. Indeed, even the penalization of splines will look extremely familiar: it's basically just L2 regularization to induce smoothness. You might also enjoy reading the "Neural Additive Model" paper from Hinton's lab, which is basically GAMs using a separate DNN as a "spline basis" for each input variable.

deleted 3y ago

[deleted]

idiotsecant 3y ago

Widely known in the controls engineering world for the last hundred years as 'anything is linear if you zoom in far enough'

vitus 3y ago
Well, subject to certain constraints. You aren't guaranteed that your equilibria behave the same in the linearization of a nonlinear system if your Jacobian has any eigenvalues with real part of 0. https://en.wikipedia.org/wiki/Hartman%E2%80%93Grobman_theorem https://en.wikipedia.org/wiki/Hartman%E2%80%93Grobman_theore...
new_username01 3y ago
It's amazing how many ideas I've seen being "rediscovered" these days from control theory.
kqr 3y ago
Wait isn't that just an application of Taylor expansion which analysis has known about for several hundreds of years?
- f1shy 3y ago
  Taylor is only for some functions, must be continuos and differentiable to n grades. In control theory you break the functions, so discontinuities can also be considered.
rcfox 3y ago
Alternatively, anything is linear if you add enough dimensions.

mjcohen 3y ago

In the 70's in CS grad school at USC, I wrote an adaptive least squares cubic spline fit routine. Kept subdividing intervals and fitting a least squares cubic spline in each interval until a criteria was met. Fun times. Don't have the code or writeup any more.

EvgeniyZh 3y ago

"You can make linear model non linear by adding non linearity" is kind of vacuous statement, isn't it

ivansavz 3y ago
No no, linear just means some expression of the form y = b1*x1 + b2*x2 + ... + bp*xp which is in the heart of the model (perhaps more specific description would be linear combination). Whatever you call it, it's just some quantity y, that is constructed via an additive process from components x1...xp, and each component is multiplied by some constant (the coefficient of the linear combination). This linear combination "core" of the model can be used directly (in which case it is geometrically a hyperplane), but you could also use non-linear transformation of the inputs or the outputs. E.g. x1 could be a spline basis function, or log-transformed data, or anything. Similarly, the output y can be passed though some non-linearity to force a particular output (like in logistic regression). Hence the statement is not vacuous
- EvgeniyZh 3y ago
  Well yeah, but what is the non trivial statement here? Linear model is linear, and if you add to your model non-linear steps (such as pre- or post- processing) then it is not longer linear and can handle non-linear data. In the same spirit, you could say that GPT4 is a linear model (token logits) with a very large preprocessing (transformer). Maybe what OP meant is that particular non linearities (splines) allow to keep some goodies from the linear model in non-linear settings. Still it's not clear for me in which settings exactly such models are good enough

2-718-281-828 3y ago

> Splines can easily take care of that. within set boundaries.

usgroup 3y ago

GLMs are a non linear transformation on an output followed by linear modelling. They are referred to as “linear models” but you might as well then also consider NNs as linear models, or any model at all which ends with addition as the final step.

fho 3y ago

Can you recommend some resources to go from step 3 to 4? So far I have been successfully using GLMMs where appropriate, but then jumped to implementing completely arbitrary models by fitting them to the data (plus bootstrapping). But that takes ages for most problems as you say