4 ms·

if you want to convert people into loving linear models (and you should), we need to make sure that they learn the difference between 'linear models' and 'linea

if you want to convert people into loving linear models (and you should), we need to make sure that they learn the difference between 'linear models' and 'linear models fit using OLS' i've met smart people that cant wrap their head around how it's possible to create linear model where the number of parameters exceeds the number of data points (that's an OLS restriction). or they're worried about how they can apply their formula for calculating the std error on the parameters. bruh, it's the future and we have big computers. just bootstrap em and don't make any assumptions.

hansvm 2y ago
> where the number of parameters exceeds the number of data points Linear models have many solutions fitting the data exactly in that parameter regime, many more fitting it approximately for any metric still satisfying the idea that identical outputs are preferable, and sometimes multiple solutions even with more data. So.....not just for OLS, but for most metrics (where you'd prefer to match or approximately match the data), the parameters are underconstrained. How much that matters depends on lots of things. If you have additional constraints (a common one that's particularly easy to program is looking for a minimum-norm solution), that trivially solves the problem. Otherwise, you might still have issues. E.g., non-minimum-norm solutions often perform badly on slightly out-of-distribution samples (since those extra basis vectors were unconstrained and thus might be large). Is there something I'm missing where 'linear models' are used to represent something wildly different than I'm used to? Are people using norms with discontinuities or something in practice? Is the criticism of OLS perhaps unrelated to the overparameterization issue? I think I'm missing some detail that would relate all of those.
tylerrobinson 2y ago
Okay, I’ll bite. > If you want to convert people into loving linear models (and you should), we need to make sure that they learn the difference between 'linear models' and 'linear models fit using OLS' Help me understand the pitch. What linear models are you referring to here that aren’t estimated with OLS? How should I wrap my head around having more parameters than observations?
- solresol 2y ago
  Linear models that aren't estimated with OLS: - Theil-Sen - Huber - RANSAC Models that can cope with more parameters than observations: - Ridge - Lasso

hansvm 2y ago

> where the number of parameters exceeds the number of data points Linear models have many solutions fitting the data exactly in that parameter regime, many more fitting it approximately for any metric still satisfying the idea that identical outputs are preferable, and sometimes multiple solutions even with more data. So.....not just for OLS, but for most metrics (where you'd prefer to match or approximately match the data), the parameters are underconstrained. How much that matters depends on lots of things. If you have additional constraints (a common one that's particularly easy to program is looking for a minimum-norm solution), that trivially solves the problem. Otherwise, you might still have issues. E.g., non-minimum-norm solutions often perform badly on slightly out-of-distribution samples (since those extra basis vectors were unconstrained and thus might be large). Is there something I'm missing where 'linear models' are used to represent something wildly different than I'm used to? Are people using norms with discontinuities or something in practice? Is the criticism of OLS perhaps unrelated to the overparameterization issue? I think I'm missing some detail that would relate all of those.

tylerrobinson 2y ago

Okay, I’ll bite. > If you want to convert people into loving linear models (and you should), we need to make sure that they learn the difference between 'linear models' and 'linear models fit using OLS' Help me understand the pitch. What linear models are you referring to here that aren’t estimated with OLS? How should I wrap my head around having more parameters than observations?

solresol 2y ago
Linear models that aren't estimated with OLS: - Theil-Sen - Huber - RANSAC Models that can cope with more parameters than observations: - Ridge - Lasso