4 ms·

I disagree. Time and time again data advantages and computing power advantages are clobbered by algorithm advances. For instance, there is no amount of cluster

I disagree. Time and time again data advantages and computing power advantages are clobbered by algorithm advances. For instance, there is no amount of cluster power that'll let you best resnet with a non-residual network. Well perhaps there is, but even Google's mighty clusters can't make up the difference. In audio, end-to-end models that take 2 weeks to train on a dual GTX 1080 machine, and take > 1 week to train on an entire datacenter beat everything we've ever come up with in linguistics. And it was recently shown that taking the convolutions and redoing in hand-coded assembly gains you more performance than the quite-meager performance a cluster gets you. The same goes for data. We can outperform VGGNet trained on all of imagenet (~ 15 million images) using resnet trained on a 1% random sampling of imagenet. We have papers putting forth statistics on specific model architectures outperforming with < 0.1% of data of benchmarks and just-barely-worse performance using a decent, but fixed and tiny number of data. I understand why Google and Microsoft are pushing "you can't beat clusters", but anyone fiddling with batch-size on actual models will quickly realize that it's ... let's say "riddled with MAJOR caveats" (using clusters implicitly forces large batch sizes, which costs you a LOT in training efficiency, especially once you get past the first few rounds, and the better your model, the more it costs).

Barrin92 9y ago
>The same goes for data. We can outperform VGGNet trained on all of imagenet (~ 15 million images) using resnet trained on a 1% random sampling of imagenet. We have papers putting forth statistics on specific model architectures outperforming with < 0.1% of data of benchmarks and just-barely-worse performance using a decent, but fixed and tiny number of data. okay granted, but isn't this largely part of the academic field, openly accessible and transferable? People at Baidu and Tencent are likely up to date on the state of the art. Contrast this for example with the American military edge or the economics of the German Mittelstand. There's a degree of ingrained knowledge that gives countries decades of advance, but academic research can often be transferred. Understanding in ML to me seems to fall mostly in the second category, and when that category dominates then data and policy really does become relevant. I'm not sure we have seen an ML intellectual arms race yet. For the most part techniques seem to spread fast and as a result data functions like oil of sorts and gives the biggest players the largest advantage.
- chillee 9y ago
  You see the overall big techniques being published, but there's still a huge gap between papers being published and actual implementation in products. Implementing papers into problem-specific domains always introduced all sorts of challenges as well as new structure to take advantage of. For example, none of the openly available machine translation implementations are anywhere near the proprietary ones.
  - candiodari 9y ago
    True, but that's actually because the implementation in products is worse (older) in the giants, compared to the state of the art. Why ? I don't know, but I suspect there's a lot of reasons: 1) they actually have an older implementation. Not many people have working linguistic voice recognition, as that's really, really hard, and you're just not going to make one of those without a 100-strong multidisciplinary team. There's no real business need to replace it (yet). 2) once you have a multidisciplinary team, the multidisciplinary aspect of it is the main reason for the size of the org. Moving to a pure-AI solution would mean 80% or so of the department would become useless and unable to contribute. 3) I bet the linguistic model looks a whole lot more comfortable to the executives than a pure AI algorithm. After all, a linguistic model will not make "hidden mistakes" (mistakes that the system makes but have never been programmed in). And let's not forget that the last few hidden mistake in a highly public model was confusing African Americans with, shall we say, animal byproducts. Needless to say, this was NOT good PR-wise. (by the way you should try to have a realistic "let's do this with AI" talk with a senior manager, and you'll see what I mean. "Can you guarantee it won't make mistakes ?", "Nope. In fact I will pretty much guarantee mistakes. It's like a person. It might purposefully make mistakes in the sense that it causes a disaster in one area because it improves it's metrics". "Okay. Can you at least tell me why it made decisions ?", "No. Impossible. Also: please don't believe any AI researcher claiming otherwise". You're asking extremely risk-averse people to take a big leap) 4) Career-opposition. In the large orgs, the senior engineers have their senior position because they improved step 57 of algorithm 21 by 5%. Making proposals to replace everything after step 3 with an end-to-end model ... they will "politely sabotage" it. (e.g. demand guarantees that it won't make mistakes. Request papers proving that it outperforms, not just the function, but every individual step. Demand they illustrate that thousands of slight mistakes won't ever happen, ...) (you know, like factory workers demanding robot features BECAUSE they figured out that they're ridiculously hard. They have no use for the factory. E.g. demands that a robot responds intelligently to a human walking by ... on a floor where no humans are allowed if any machinery is running. Or demand physical separation of robot action radii, when the software supports those robots working together and this is in fact used in said production line. Makes no sense whatsoever. What are they doing ? They think they're defending their jobs) I've seen several startup products based on Deepmind's Wavenet, but Microsoft and Google's voice recognition have both said that theirs is based on a linguistic accoustic model. You know, the huge, very very complex, dozens of different components, each their own specialization. So the opposite of what you'd think is actually what's happening. The organizations doing the cutting edge research are not ahead of the curve, they're behind, far behind, and falling more day by day. Or to put it another way, you want to see innovation happen ? Find companies that would be dead and bankrupt without that innovation. Google and Microsoft, those are not it. Even facebook is better.

Barrin92 9y ago

>The same goes for data. We can outperform VGGNet trained on all of imagenet (~ 15 million images) using resnet trained on a 1% random sampling of imagenet. We have papers putting forth statistics on specific model architectures outperforming with < 0.1% of data of benchmarks and just-barely-worse performance using a decent, but fixed and tiny number of data. okay granted, but isn't this largely part of the academic field, openly accessible and transferable? People at Baidu and Tencent are likely up to date on the state of the art. Contrast this for example with the American military edge or the economics of the German Mittelstand. There's a degree of ingrained knowledge that gives countries decades of advance, but academic research can often be transferred. Understanding in ML to me seems to fall mostly in the second category, and when that category dominates then data and policy really does become relevant. I'm not sure we have seen an ML intellectual arms race yet. For the most part techniques seem to spread fast and as a result data functions like oil of sorts and gives the biggest players the largest advantage.

chillee 9y ago
You see the overall big techniques being published, but there's still a huge gap between papers being published and actual implementation in products. Implementing papers into problem-specific domains always introduced all sorts of challenges as well as new structure to take advantage of. For example, none of the openly available machine translation implementations are anywhere near the proprietary ones.
- candiodari 9y ago
  True, but that's actually because the implementation in products is worse (older) in the giants, compared to the state of the art. Why ? I don't know, but I suspect there's a lot of reasons: 1) they actually have an older implementation. Not many people have working linguistic voice recognition, as that's really, really hard, and you're just not going to make one of those without a 100-strong multidisciplinary team. There's no real business need to replace it (yet). 2) once you have a multidisciplinary team, the multidisciplinary aspect of it is the main reason for the size of the org. Moving to a pure-AI solution would mean 80% or so of the department would become useless and unable to contribute. 3) I bet the linguistic model looks a whole lot more comfortable to the executives than a pure AI algorithm. After all, a linguistic model will not make "hidden mistakes" (mistakes that the system makes but have never been programmed in). And let's not forget that the last few hidden mistake in a highly public model was confusing African Americans with, shall we say, animal byproducts. Needless to say, this was NOT good PR-wise. (by the way you should try to have a realistic "let's do this with AI" talk with a senior manager, and you'll see what I mean. "Can you guarantee it won't make mistakes ?", "Nope. In fact I will pretty much guarantee mistakes. It's like a person. It might purposefully make mistakes in the sense that it causes a disaster in one area because it improves it's metrics". "Okay. Can you at least tell me why it made decisions ?", "No. Impossible. Also: please don't believe any AI researcher claiming otherwise". You're asking extremely risk-averse people to take a big leap) 4) Career-opposition. In the large orgs, the senior engineers have their senior position because they improved step 57 of algorithm 21 by 5%. Making proposals to replace everything after step 3 with an end-to-end model ... they will "politely sabotage" it. (e.g. demand guarantees that it won't make mistakes. Request papers proving that it outperforms, not just the function, but every individual step. Demand they illustrate that thousands of slight mistakes won't ever happen, ...) (you know, like factory workers demanding robot features BECAUSE they figured out that they're ridiculously hard. They have no use for the factory. E.g. demands that a robot responds intelligently to a human walking by ... on a floor where no humans are allowed if any machinery is running. Or demand physical separation of robot action radii, when the software supports those robots working together and this is in fact used in said production line. Makes no sense whatsoever. What are they doing ? They think they're defending their jobs) I've seen several startup products based on Deepmind's Wavenet, but Microsoft and Google's voice recognition have both said that theirs is based on a linguistic accoustic model. You know, the huge, very very complex, dozens of different components, each their own specialization. So the opposite of what you'd think is actually what's happening. The organizations doing the cutting edge research are not ahead of the curve, they're behind, far behind, and falling more day by day. Or to put it another way, you want to see innovation happen ? Find companies that would be dead and bankrupt without that innovation. Google and Microsoft, those are not it. Even facebook is better.