10 ms·
Everything is a linear model
- SubiculumCode 3y agoI find it irritating that the article mentions repeated measures, but does not try them, much less a mixed effects model. Yes, they are linear models with more parameters, but doing it in lm would be a special kind of madness
- simulo 3y agoI knew the linked-in-the-article https://lindeloev.github.io/tests-as-linear/ https://lindeloev.github.io/tests-as-linear/ which is also great. A bit meta on the widespread use of linear models: "Transcending General Linear Reality" by Andrew Abbott, DOI:10.2307/202114
- ivansavz 3y agoHere is the Python port of test-as-linear developed by George Ho (eigenfoo): https://github.com/minireference/tests-as-linear/blob/bugfix_and_updates/tests-as-linear.ipynb https://github.com/minireference/tests-as-linear/blob/bugfix... I'm linking to my fork of it because I've added some fixes and filled in some of the missing parts (e.g. Welch's t-test).
- LifeIsBio 3y agoI read this article when I was in grad school 5 years ago. Absolutely love it and talk about it to this day. It really makes me frustrated about the ways I was introduced to statistics: brute force memorization of seeming arbitrary formulas.
- btdmaster 3y agoI thought nonlinearity was very important to be able to make a larger model better than a smaller one? Like so important that tom7 made a half-joke demo with it: https://yewtu.be/watch?v=Ae9EKCyI1xU https://yewtu.be/watch?v=Ae9EKCyI1xU
- epgui 3y agoLinear models don’t need everything to be linear.
- iamcreasy 3y agoI presume you are implying that linear model only mandates linear relationship between predictor and regression coefficients?
- stdbrouw 3y agoA linear relationship between any transformation of the outcome and any transformation of the predictor variables — so the function is linear but the relationship between predictors and outcome can take on almost any shape.
- iamcreasy 3y agoAh, I missed 'the transformation of outcome' in my mind. Thanks for clearing it up.
- hackerlight 3y agoLinear models are a linear combination of possibly non-linear regressors. The linearity is strictly in the parameters, not in whatever you're adding up. A neural network can be pedantically referred to as a linear model of the form y = a + b*neural_network, for example. Here, y is a linear model (even though neural_network isn't).
- epgui 3y agoIMO it's not very pedantic... It's pretty much exactly what it is! (I'm not too sure about the example equation you give, however)
- dist-epoch 3y agoWell, you can create a non-linear model by piece-wise combining multiple linear models. The famous ReLU non-linearity is just that - two linear functions joined.
- dboreham 3y agoHmm...I thought everything was an Eigenfunction.
- jwilber 3y agoAnother fun stats X is really Y: estimating the Area Under the Curve metric (AUC) is equivalent to the Wilcoxon-Mann-Whitney test! https://rmets.onlinelibrary.wiley.com/doi/abs/10.1256/003590002320603584 https://rmets.onlinelibrary.wiley.com/doi/abs/10.1256/003590...
- t_mann 3y agoStatistics is more than hypothesis testing, but you'll get surprisingly far without straying too far from linear models - I remember a Stats prof saying 'most of classical Statistics is GLM [0]' [0] https://en.wikipedia.org/wiki/Generalized_linear_model https://en.wikipedia.org/wiki/Generalized_linear_model
- lanstin 3y agoWhich means it really all just finding hyperplanes that are near the data.
- sebastianavina 3y ago"Classification of mathematical problems as linear and nonlinear is like classification of the Universe as bananas and non-bananas. " and everything turns around the same principles. For example dynamical models and PID controls. yet solving a banana, is the only thing we really know how to do. So we end up fitting everything in our banana models.
- Tommah 3y agoLinear algebra was the first math class I took in undergrad. I thought the next one was going to be non-linear algebra! But it wasn't.
- whatshisface 3y agoI disagree with the implication that linearity is an unnatural concept, it appears whenever the changes being studied are small relative to the key parameters that determine the system. Every system is linear for small perturbations. Even logic gates; in negative feedback they can form passable inverting amplifiers. In a place as big as the universe it is rather common for two things to be very different in scale and yet interacting.
- notjoemama 3y agoSo, Lady Finger, not Cavendish. Got it.
- ofrzeta 3y agoI thought everything was a power function.
- smitty1e 3y agoI thought everything was a graph.
- optimalsolver 3y agoNah, it's all just ifs and for-loops: https://www.reddit.com/media?url=https%3A%2F%2Fi.redd.it%2F48xp6c3blm381.jpg https://www.reddit.com/media?url=https%3A%2F%2Fi.redd.it%2F4...
- sva_ 3y ago(2022)
- chmaynard 3y agoRSS feed: https://danielroelfs.com/blog/index.xml https://danielroelfs.com/blog/index.xml
- matteoraso 3y agoA cool thing about linear models is that they can be used to model non-linear correlations by using transformations. For example, an exponential function can be made linear if you just take the logarithm.
- johnsutor 3y agoSee the topic of GLMs: https://en.wikipedia.org/wiki/Generalized_linear_model https://en.wikipedia.org/wiki/Generalized_linear_model
- Horffupolde 3y agoNot all functions have a trivial transformation.
- gwern 3y agoDoesn't have to be 'trivial'. It could be a trillion-parameter MoE LLM spitting out 10,000-large embeddings. But a lot of that will act linearly and can, in theory, be put into your linear model easily & profitably.
- richrichie 3y agoYou can state and prove theorems with linear models. You can do inference and testing. This means papers and academics naturally love them. And therefore, they are everywhere. Not the case with non-linear models. We need to throw computers at them.
- whatshisface 3y agoI think that's an overstatement. You can consider the nonlinearity as a perturbation of the linear theory, study the topology of the solutions or their symmetries, or sometimes even find exact solutions, like exist for a handful of transistor circuits.
- richrichie 3y agoIf the model is y = a + b * x + e, then we can precisely state probability distributions of a and b given x and y for reasonably general assumptions about error e. If the model is y = f(x) + e, where for example f is a neural net, there is not much we can say about the “quality” of parameters of f. That is, we can’t attach confidence interval. We will have to resort to expensive simulations and for most practical applications this is not workable (size of dataset). What you say is useful in a different context. Local linearization is a very powerful idea.
- whatshisface 3y agoMaybe not if there are many more parameters in f than there are samples in the dataset, but if there are a handful of parameters in f then a typical step is to take the inverse of the derivative of goodness-of-fit with respect to each fit parameter, to determine the precision with which each parameter was determined by the fit.
- lngnmn2 3y ago[dead]
- cs702 3y agoMaybe it's because to scientists and statisticians comfortable with linear algebra, everything looks like a linear model.
- levocardia 3y agoI see a lot of comments here assuming "linear model" means "can't model nonlinearities." Absolutely not the case. Splines can easily take care of that. The "linear" part of linear model just means "linear in the predictor space." You can add a non-linear predictor easily via spline basis (similar/sometimes identical to "kernels" in ML). My series of lm/glm/gam/gamm revelations was: 1. All t-tests and ANOVA flavors are just linear models 2. Linear models are just a special case of generalized linear models (GLMs), which can deal with binary or count data too 3. All linear models and GLMs are just special cases of generalized linear mixed models, which can deal with repeated measures, grouped data, and other non-iid clustering 4. Linearity is usually a bad assumption, which can easily be overcome via splines 5. Estimating a spline smoothing penalty is the same thing as estimating the variance for a random effect in a a mixed model, so #3 and #4 can be combined for free And then you end up with a generalized additive mixed model (GAMM), which can model smooth nonlinear functions of many variables, smooth interaction surfaces between variables (e.g. latitude/longitude), handle repeated measurements and grouping, and deals with many types of outcomes, including continuous, binary yes/no, count, ordinal categories, or survival time measurements. All while yielding statistically valid confidence intervals, and typically only taking a few minutes of CPU time even on datasets with hundreds of thousands / millions of datapoints.
- carlthome 3y agoThis sounds really cool but was hard to digest for me as a ML Engineer who came into work just around deep learning and DNNs. Is there some go-to practice material I could look at? Splines I haven't touched since numerical computing exercises in school.
- goosedragons 3y agoSimon Wood's Generalized Additive Model book.
- levocardia 3y agoOr, for a more approachable treatment, Semiparametric Regression with R by Harezlak, Ruppert, and Wand. A middle ground between Wood's book (which is comprehensive but can dip into math that's way over my head at times) and H/R/W is Semiparametric Regression by Ruppert, Wand, and Carroll. I have also heard great things about Frank Harrell's Regression Modeling Strategies which uses a slightly different approach (still spline-based though), but I haven't read it. His other writing is fantastic though.
- CassandraOakly 3y ago[flagged]
- 2-718-281-828 3y agoisn't every non linear model fundamentally linear if it is run on a von neumann computer?
- kqr 3y agoThis is why I feel like I couldn't become a credible AI/ML consultant. I would just throw everything into a linear model, make some progress, and call it a day.
- blitzar 3y ago99% of AI / ML consultants are just throwing everything into a linear model, make no progress, slap a big AI sticker on the box and call it a day. You would have fitted right in.
- ronald_raygun 3y agoWasn’t there an OpenAI paper where they showed that MM multiplication with the numerical imprecision of floats was enough to get general function learning? You might not even need anything else…
- blitzar 3y agoOne almost certainly does not need anything else ... unless what you want is big piles of investor cash, a Scrooge McDuck swimming pool quantity of investor cash - then you need to call whatever maths you do AI.
- funcDropShadow 3y agoEverything is linear if plotted log-log with a fat magic marker (Mar's Law) [1]. [1]: https://spacecraft.ssl.umd.edu/akins_laws.html https://spacecraft.ssl.umd.edu/akins_laws.html