4 ms·
In classical estimation tasks (state estimation, control, SLAM, etc..) techniques for dealing with outlier data are well established (RANSAC and it's siblings,
by rsp1984 6y ago
In classical estimation tasks (state estimation, control, SLAM, etc..) techniques for dealing with outlier data are well established (RANSAC and it's siblings, M-Estimators, etc..) and known as "robust estimation". Depending on how much processing you throw at it these techniques can successfully deal with 90% corrupt datasets (10% inliers).
For some reason though I've never heard of "robust ML" and I always wonder why. The only thing I hear about is increasing NN sizes and increasing model complexity. Many smart people are working on this so I suppose there's a reason for that but it's just not very obvious to me. Can someone with knowledge of the matter provide some insight?
- beforeolives 6y agoIsn't what you're describing simply regularization?
- mlthoughts2018 6y agoRobust potential functions are a huge part of ML. Many people have researched robust potential functions for use as loss functions. Its use in ML algorithms predates use for SLAM, tracking algorithms, etc., which only used it after classical ML. Neural nets typically don’t benefit much from it because you can use batch normalization, dropout and clever activation functions to achieve the same results, by having the network learn diminished sensitivity to outliers that produce neurons which saturate the low end of an activation function. This is preferable because many of the robust potential functions involve absolute values, order statistics and other non-differentiable quantities that are hard to put into backpropagation-based optimizers. You almost always would need to relax the loss function to something that trades off smoothness against outlier robustness, where convergence will be slower and slower as you crank the trade off closer to outlier robustness.