4 ms·

It’s logically obvious. You can recreate any linear regression using a neural network. So a neural network approach will always be at least as good as linear r

It’s logically obvious. You can recreate any linear regression using a neural network. So a neural network approach will always be at least as good as linear regression. But a neural network can model non-linear relationships as well. Now ask yourself, what is the likelihood of there being a non-linear relationship in the data? As with most real-world data, the likelihood of non-linear relationships is extremely high. That’s why we can be quite certain a neural network approach will able to outperform linear regression.

Calavar 3y ago
> It’s logically obvious. You can recreate any linear regression using a neural network. So a neural network approach will always be at least as good as linear regression. Well, maybe it's logically obvious, but it's also empirically false: https://www.youtube.com/watch?v=x7psGHgatGM https://www.youtube.com/watch?v=x7psGHgatGM (Skip to 13:36 to get to the point) > But a neural network can model non-linear relationships as well. Now ask yourself, what is the likelihood of there being a non-linear relationship in the data? As with most real-world data, the likelihood of non-linear relationships is extremely high. That’s why we can be quite certain a neural network approach will able to outperform linear regression. As I've said elsewhere in this thread, a deep neural network of N layers can be decomposed into a complex nonlinear function of N - 1 layers followed by a linear combination of the nonlinear features generated that function. There's no law that says that you have to use a neural network to generate your nonlinear features. You can use any method you like and then linearly combine those.
- 4death4 3y ago
  I don’t think you understand what you’re saying. “A linear combination of non-linear features” is not what people mean when they talk about linear regression. And even if they were, a neural network will do a much better job of generating the non-linear features than you will be hand. So again, it’s logically obvious that a neural network will be capable of performing better than simple linear regression.
  - Calavar 3y ago
    > I don’t think you understand what you’re saying. “A linear combination of non-linear features” is not what people mean when they talk about linear regression. According to who? Did you watch the talk I linked? Because in that video, NeurIPS gave their test of time award to a paper where they generated nonlinear features and used those to train a linear classifier. I guess the committee at NeurIPS also didn't understand what they are saying? > And even if they were, a neural network will do a much better job of generating the non-linear features than you will be hand. So again, it’s logically obvious that a neural network will be capable of performing better than simple linear regression. Again, logically obvious but often empirically false.

Calavar 3y ago

> It’s logically obvious. You can recreate any linear regression using a neural network. So a neural network approach will always be at least as good as linear regression. Well, maybe it's logically obvious, but it's also empirically false: https://www.youtube.com/watch?v=x7psGHgatGM https://www.youtube.com/watch?v=x7psGHgatGM (Skip to 13:36 to get to the point) > But a neural network can model non-linear relationships as well. Now ask yourself, what is the likelihood of there being a non-linear relationship in the data? As with most real-world data, the likelihood of non-linear relationships is extremely high. That’s why we can be quite certain a neural network approach will able to outperform linear regression. As I've said elsewhere in this thread, a deep neural network of N layers can be decomposed into a complex nonlinear function of N - 1 layers followed by a linear combination of the nonlinear features generated that function. There's no law that says that you have to use a neural network to generate your nonlinear features. You can use any method you like and then linearly combine those.

4death4 3y ago
I don’t think you understand what you’re saying. “A linear combination of non-linear features” is not what people mean when they talk about linear regression. And even if they were, a neural network will do a much better job of generating the non-linear features than you will be hand. So again, it’s logically obvious that a neural network will be capable of performing better than simple linear regression.
- Calavar 3y ago
  > I don’t think you understand what you’re saying. “A linear combination of non-linear features” is not what people mean when they talk about linear regression. According to who? Did you watch the talk I linked? Because in that video, NeurIPS gave their test of time award to a paper where they generated nonlinear features and used those to train a linear classifier. I guess the committee at NeurIPS also didn't understand what they are saying? > And even if they were, a neural network will do a much better job of generating the non-linear features than you will be hand. So again, it’s logically obvious that a neural network will be capable of performing better than simple linear regression. Again, logically obvious but often empirically false.