3 ms·

> If we can't know in advance, how can you expect a glorified Markov Chain to magically figure it out? If it could - and it can't, but if it could - how would y

by yccs27 5y ago

> If we can't know in advance, how can you expect a glorified Markov Chain to magically figure it out? If it could - and it can't, but if it could - how would you know it did it correctly? We might not know anything about them in advance, but the patterns are there and could maybe be extracted from the some training data. If only you had a statistical model that was flexible enough to find them… Validation is then as easy as running the model on some examples outside the training set. > No, it gives an infinite number of sine waves added up together. Yeah, and after Doppler shift it is still an infinite number of sine waves - no immediate information gained. Of course, if there are characteristics in the original noise and its frequency distribution, you could try to find those in the doppler-shifted signal. How would you determine the characteristics? From a dataset of examples, I guess. So now the problem is: recognize a pattern from examples and try to find it in new instances. Sounds like the kind of problem ML has found success in. (If you're now thinking "we don't need ML, just some advanced statistics"… Well ML is often basically a statistical model with lots and lots of parameters.)

TeMPOraL 5y ago
> Validation is then as easy as running the model on some examples outside the training set. Only if you can trust the data gathered from that validation to be representative. You can do that easily when you understand the statistics your model is doing - which is the case with an "old-school" ML solution, but not so with DNNs. This gets worse the more complex your problem is. I can expect a DNN to pick up the correct frequency patterns in audio time series quickly, as it stands out in the solution space - but with more variables, more criteria, we know it takes ludicrous amounts of data for the model to start returning good results, and it still often fixates on dubious variables. And then you have to ask yourself - what are your error bars? With a classical approach to estimating train velocity from sound, your results will be reasonably bounded, and won't surprise you. With a DNN, all bets are off. > How would you determine the characteristics? From a dataset of examples, I guess. And physics. In this case, a human can apply their understanding of physics to determine what characteristics to expect, verify they exist in the dataset, and encode that knowledge in the solution. A DNN will have to figure this out on its own, and we have no good way to verify it did it correctly (and isn't just overfit on something that's strongly but incidentally correlated). I agree there are plenty of problems where we don't have a good "first principles" solution - where we're just looking for correlations. DNNs automate this nicely. But such models belong to the category of untrusted ones - they might seem to work now, but because of their opaqueness, we can't treat past performance as a strong indicator of reliability. > Well ML is often basically a statistical model with lots and lots of parameters. Yes. But I think it matters if people know what those parameters do.

by yccs27 5y ago

TeMPOraL 5y ago

> Validation is then as easy as running the model on some examples outside the training set. Only if you can trust the data gathered from that validation to be representative. You can do that easily when you understand the statistics your model is doing - which is the case with an "old-school" ML solution, but not so with DNNs. This gets worse the more complex your problem is. I can expect a DNN to pick up the correct frequency patterns in audio time series quickly, as it stands out in the solution space - but with more variables, more criteria, we know it takes ludicrous amounts of data for the model to start returning good results, and it still often fixates on dubious variables. And then you have to ask yourself - what are your error bars? With a classical approach to estimating train velocity from sound, your results will be reasonably bounded, and won't surprise you. With a DNN, all bets are off. > How would you determine the characteristics? From a dataset of examples, I guess. And physics. In this case, a human can apply their understanding of physics to determine what characteristics to expect, verify they exist in the dataset, and encode that knowledge in the solution. A DNN will have to figure this out on its own, and we have no good way to verify it did it correctly (and isn't just overfit on something that's strongly but incidentally correlated). I agree there are plenty of problems where we don't have a good "first principles" solution - where we're just looking for correlations. DNNs automate this nicely. But such models belong to the category of untrusted ones - they might seem to work now, but because of their opaqueness, we can't treat past performance as a strong indicator of reliability. > Well ML is often basically a statistical model with lots and lots of parameters. Yes. But I think it matters if people know what those parameters do.