3 ms·
This period is only a phase while CNNs offer the best bang. As algorithms improve, the amount of data needed to train a model will drop 1000x making the big5's
by sitkack 9y ago
This period is only a phase while CNNs offer the best bang. As algorithms improve, the amount of data needed to train a model will drop 1000x making the big5's data hoards not worth nearly as much. In five years, the power will shift away from petabyte sized datasets.
- charlysl 9y agoMy understanding is that data needed (aka data complexity) is independent of the algorithm. It depends only on model complexity (roughly # of free parameters in model). AFAIK this is a fundamental principle of ML that has been proven mathematically and which is inescapable. https://en.m.wikipedia.org/wiki/VC_dimension https://en.m.wikipedia.org/wiki/VC_dimension
- ekidd 9y ago> AFAIK this is a fundamental principle of ML that has been proven mathematically and which is inescapable. Well, the human brain manages to master several complex tasks using much smaller data sets than current machine learning algorithms. Natural language acquisition, for example, seems to require fewer than 10 million spoken words per year, and even academically successful 12-year-olds might be reading 1 to 4 million words per year. These are not exactly tiny data sets, but they don't require Google's scale to recreate, either. Sure, the human genome probably "knows" what set of models to try when building the brain, which gives it an advantage. But I can't think of any reason why machine learning couldn't ultimately try similar techniques with similarly-sized data sets, and get competitive results.
- charlysl 9y agoYou are right. I didn't phrase it correctly. What is algorithm independent is the theoretical performance bound for a given data complexity. Sure, not all algorithms are the same, but the best performance that can theoretically be achieved with a given iid sample of a given size is algorithm independent. So yes, a better algo can get same performance with less data, but limit stays the same.
- otakucode 9y agoI'm not entirely sure. Yes, you need a mountain of data to train a system very well to be flexible. However, once that training is done, you absolutely do not need 'cloud' resources whatsoever. You could run an already-trained NN of any sort on an embedded processor running on tiny batteries. They require extremely little processing to actually push some input through the network and get the output. The only reason we continue to send all of our data to cloud companies to have the speech recognition performed there is because those companies have perverse incentives to snoop and spy and profile and target aggressively. Technically speaking, all of these 'voice assistants' and the like would be far better products if they hosted a local pre-trained network that did all of the recognition. Latency is the biggest challenge to these systems, and that simply will never be solved so long as the recognition happens on a server miles away from where the user is speaking. The speed of light, at a minimum, comes into play. What would be very interesting would be if there was a development of a continuously learning system which actually performs better training itself to recognize one or a few users voices without the burden of carrying the weights necessary to also recognize the voices of people with accents on the other side of the planet from anyone who will ever be within earshot of it. That would be an even more overt and active disincentive to sending all your data away "for speech recognition" (the spying is a fringe benefit!) and sitting around waiting for the cloud to get back to you.... and I imagine we'd see it ignored. The limitations today aren't technical, they're organizational and business oriented. And those things don't generally change in concert with technical changes.