4 ms·

I think the biggest advancement that could come out of the ML space right now is the equivalent of Big O notation for ML. Our current algorithms suck, and the

I think the biggest advancement that could come out of the ML space right now is the equivalent of Big O notation for ML. Our current algorithms suck, and the first step to making them suck less is to precisely measure just how much they suck.

currymj 7y ago
i mean for many classic ML models you have this — literally big O notation for the number of samples required to learn, number of iterations to convergence, cost per iteration, etc. You even get the cases where the algorithm with the best big-O bound is totally impractical due to constant factors. The problem is just that all this goes out the window for deep learning, because nobody really knows how to prove much of anything about neural networks.
tixocloud 7y ago
It’s similar to our viewpoint as well. Currently, it’s easy to see gains because we’re mainly going from highly manual processes to 1st generation AI applications. Future generation AI applications will mostly be optimization and squeezing out very tiny performance gains. We want to be able to highlight eventually how much things will cost and whether it’s even worth spending so much for a tiny amount of gain. There’s so much hype right now that silly money is just being thrown at new tech without fully understanding the value you’re getting.
6gvONxR4sf7o 6y ago
We have these for many models and parts of models. On one side, you have sample complexity [0] (how much data do you need?). On the other side, in deep learning you also have optimization convergence rates (how many steps do you need find the right parameters?). For example, newton's method converges at a quadratic rate for many problems. I know people work on similar theories for non-convex optimization, but the actual work there is outside what I know. It would be great to have a breakthrough there. [0] https://en.wikipedia.org/wiki/Sample_complexity https://en.wikipedia.org/wiki/Sample_complexity

currymj 7y ago

i mean for many classic ML models you have this — literally big O notation for the number of samples required to learn, number of iterations to convergence, cost per iteration, etc. You even get the cases where the algorithm with the best big-O bound is totally impractical due to constant factors. The problem is just that all this goes out the window for deep learning, because nobody really knows how to prove much of anything about neural networks.

tixocloud 7y ago

It’s similar to our viewpoint as well. Currently, it’s easy to see gains because we’re mainly going from highly manual processes to 1st generation AI applications. Future generation AI applications will mostly be optimization and squeezing out very tiny performance gains. We want to be able to highlight eventually how much things will cost and whether it’s even worth spending so much for a tiny amount of gain. There’s so much hype right now that silly money is just being thrown at new tech without fully understanding the value you’re getting.

6gvONxR4sf7o 6y ago

We have these for many models and parts of models. On one side, you have sample complexity [0] (how much data do you need?). On the other side, in deep learning you also have optimization convergence rates (how many steps do you need find the right parameters?). For example, newton's method converges at a quadratic rate for many problems. I know people work on similar theories for non-convex optimization, but the actual work there is outside what I know. It would be great to have a breakthrough there. [0] https://en.wikipedia.org/wiki/Sample_complexity https://en.wikipedia.org/wiki/Sample_complexity