8 ms·

Let's propose a hypothetical scenario. Imagine an investigator in a large research lab somewhere makes a revolutionary AI breakthrough. Not just an incremental

Let's propose a hypothetical scenario. Imagine an investigator in a large research lab somewhere makes a revolutionary AI breakthrough. Not just an incremental step, but one that achieves 99.9999% success in image object recognition (current state-of-the-art using GPU-accelerated Sparse ConvNets puts Univ. of Warwick's Dr. Ben Graham's team in the lead at about 95%[1]). What would be the incentive to make the result public? Certainly if the discoverer were a grad student at Berkeley they may throw a brief summary onto Arxiv the next day to get some peer validation. But if it were the ML team at Uber's new secretive lab in Pittsburgh? Would they not have a compelling interest in keeping it under the kimono? There are several possible outcomes for this game to play out. But for right now let's bask in the warmth of our open source learning toolkit abundance and concentrate on the real question of the day: now that we can teach machines, are there better ways to use their time then ascertaining what makes a top selfie? [1] https://www.kaggle.com/c/cifar-10/leaderboard/public https://www.kaggle.com/c/cifar-10/leaderboard/public

segmondy 11y ago
Never underestimate hubris and greed. If they made it know, they gain respect, they attract top notch researchers & thieves, the market rewards them with funding/buying more shares.
chimtim 11y ago
This is not how research works or a new fundamental technique is discovered. Any result needs to stand the rigor of peer-review before the said breakthrough can be conclusively proven (more unbiased minds trying to find the limitations of the approach and applying the said approach to various problems). Also this process is slow, and requires many PhDs, students building upon one another's work. Also, going from 96-99 may not be really revolutionary, what really matters is how you got there e.g. was new research needed to detect new unicorn classes or scenes, or if it was simply solved by great engineering, more data, etc.
- coldtea 11y ago
  >Any result needs to stand the rigor of peer-review before the said breakthrough can be conclusively proven Not if you can easily verify the claim is true with sample images (in the situation given). Then you don't need to care for any "peer review". In the same vein, if I come up with a data structure implementation that is 100x faster than common techniques, it just has to pass a suite of tests. No peer review needed. If some Intel engineer finds a revolutionary, 100% speed improving cache prediction method, ditto. And for lots of other cases. As soon as you check it works, you can start patenting and marketing pronto.
  - wfo 11y ago
    Sure this is the case sometimes but generally you need to check and make sure you aren't gaming some odd property of the tests somehow; does your algorithm give up speed somewhere in order to recover it somewhere else. Is there a worst case in the tests you're not noticing. It's easy to miss something small. Proprietary algorithms are by and large not fundamental general breakthroughs in the field, rather they are incremental upgrades or modifications or tradeoffs or completely non-general algorithms for a very specialized purpose that happens to align with the business goals of the company
  - lhc- 11y ago
    By patenting though, you reveal your technique (that's the whole point: you get exclusive rights in exchange for disseminating knowledge). Once you file and receive your patent, you've basically published your results and everyone can benefit (although they'll have to pay you, of course).
- yalogin 11y ago
  That is not how Google works. They keep their new fangled systems under wraps and only release the details after a few years (likely 10). Most definitely research is done through the rigor of peer reviews but in a commercial setting that is almost never the case.
  - plonh 11y ago
    More like 5 years, and shrinking.
    - chimtim 11y ago
      My comment and the article are about AI research specifically, where there is lots of exciting new research due to the success of deep learning. And since the area is new, there is lot of low hanging fruit and challenging problems, which researchers are eager to solve and publish, and as a result Apple is suffering. So it does not matter how Google research worked earlier or if other areas of Google research continue to wait for a 5 year timer.
- jandrewrogers 11y ago
  This is a strange view of computer science research. In most of the cases of material advances outside the academic setting I am familiar with, the researchers always demonstrate or prove their result to a high degree of rigor themselves. None of this requires an army of PhDs (or even one) to rigorously inspect the result unless it is so badly represented as to be opaque. It is not like they sit around coming up with random ideas for how to solve a problem and leave it at that. Peer review is useful for checking the result but you aren't supposed to rely on peer review to filter garbage ideas. I know a few professional computer science researchers, people who churn out new and better algorithms (some radically so) in multiple domains on a surprisingly consistent basis for private companies. At least half their job is actually the easily understandable explanation and proof of why the algorithm must have the claimed properties. Their job isn't to produce academic papers but to solve algorithm problems for software engineers writing real code, so a lot of focus is on reduction to practice and clear explanation of the mechanics.
- onion2k 11y ago
  Any result needs to stand the rigor of peer-review before the said breakthrough can be conclusively proven. Apple aren't in the business of doing research for the sake of provable science though. They do research to make things they can sell. The goal is very different.
  - sangnoir 11y ago
    >The goal is very different. The article agrees, and suggests that because of this, Apple will miss out on the top talent that prefers to publish
wfo 11y ago
This is a potential pitfall of private research. Ideally any real scientist who is earnest and dedicated about his or her work and is passionate about its benefit to humanity or contribution to human knowledge would, upon making a breakthrough like that, feel the need to share it. Look at the plot of Atlas Shrugged: completely unrealistic; Galt is a disgusting sociopath who invents a way to remove resource constraints from the world, essentially destroying markets and capitalism, poverty and hunger, and refuses to share it with humanity so he can hold out for... resources in exchange. These things don't happen. Scientists don't try to keep their work under wraps -- it's the business people who own them that do. The secret private work that these people are doing is a chain around their neck which stops them from being real scientists. Some are willing to give that up for money and work at a lab that doesn't publish. Some are willing to give that up out of a sense of patriotic duty and work in intelligence. It's fantastic that there are private research labs like MSR that pay well and DO publish so there isn't a forced choice between money and being a real scientist for the lucky few who work there.
robotresearcher 11y ago
The incentive for academic labs and researchers to go public is very clear: publications and kudos in the community, which leads to career advancement and plentiful research funding. You don't get super-rich like this, but you get the things that people really want. Respect of your peers, feeling like you are doing something useful, and enough money to run a lab and live a decent middle-class lifestyle, so you can get on with the work you love. That's a good life. (Also you might want to stay in touch with your grad students who are driven to get rich. Can't hurt.)
mizzao 11y ago
The reality is that's not how research works. Inspiration and breakthroughs don't come out of thin air; they come from the freedom to interact with other scientists, process their ideas and work, and then combine them with a new perspective which is tested and then occasionally turns out to be better than the state of the art. The process of research builds a lot of tacit knowledge that can't just be picked up by secretly picketing machine learning conferences. It's a lot of hard work, and the benefits to doing it in collaboration with the wider scientific community far outweigh the added value from being able to keep some marginal contribution secret. The top-secret, non-collaborating research lab always falls behind, with perhaps the rare exception of certain large and well-funded government organizations.
Houshalter 11y ago
Well they could patent it. This is literally the purpose of patents. You can't patent it if you don't publish, and if you do patent it, you have every incentive to publish it. To get other people to use it and pay you licensing fees, or just to brag and attract better scientists to work at your company. I do hate patents, and I think they are often given to ideas that seems simple, trivial, or iterative, rather than ground breaking. Stuff that probably would have been invented by someone else a few years, or even months, later. But for your hypothetical "revolutionary AI breakthrough", that's the ideal use case for them.

segmondy 11y ago

Never underestimate hubris and greed. If they made it know, they gain respect, they attract top notch researchers & thieves, the market rewards them with funding/buying more shares.

chimtim 11y ago

This is not how research works or a new fundamental technique is discovered. Any result needs to stand the rigor of peer-review before the said breakthrough can be conclusively proven (more unbiased minds trying to find the limitations of the approach and applying the said approach to various problems). Also this process is slow, and requires many PhDs, students building upon one another's work. Also, going from 96-99 may not be really revolutionary, what really matters is how you got there e.g. was new research needed to detect new unicorn classes or scenes, or if it was simply solved by great engineering, more data, etc.

coldtea 11y ago
>Any result needs to stand the rigor of peer-review before the said breakthrough can be conclusively proven Not if you can easily verify the claim is true with sample images (in the situation given). Then you don't need to care for any "peer review". In the same vein, if I come up with a data structure implementation that is 100x faster than common techniques, it just has to pass a suite of tests. No peer review needed. If some Intel engineer finds a revolutionary, 100% speed improving cache prediction method, ditto. And for lots of other cases. As soon as you check it works, you can start patenting and marketing pronto.
- wfo 11y ago
  Sure this is the case sometimes but generally you need to check and make sure you aren't gaming some odd property of the tests somehow; does your algorithm give up speed somewhere in order to recover it somewhere else. Is there a worst case in the tests you're not noticing. It's easy to miss something small. Proprietary algorithms are by and large not fundamental general breakthroughs in the field, rather they are incremental upgrades or modifications or tradeoffs or completely non-general algorithms for a very specialized purpose that happens to align with the business goals of the company
- lhc- 11y ago
  By patenting though, you reveal your technique (that's the whole point: you get exclusive rights in exchange for disseminating knowledge). Once you file and receive your patent, you've basically published your results and everyone can benefit (although they'll have to pay you, of course).
yalogin 11y ago
That is not how Google works. They keep their new fangled systems under wraps and only release the details after a few years (likely 10). Most definitely research is done through the rigor of peer reviews but in a commercial setting that is almost never the case.
- plonh 11y ago
  More like 5 years, and shrinking.
  - chimtim 11y ago
    My comment and the article are about AI research specifically, where there is lots of exciting new research due to the success of deep learning. And since the area is new, there is lot of low hanging fruit and challenging problems, which researchers are eager to solve and publish, and as a result Apple is suffering. So it does not matter how Google research worked earlier or if other areas of Google research continue to wait for a 5 year timer.
jandrewrogers 11y ago
This is a strange view of computer science research. In most of the cases of material advances outside the academic setting I am familiar with, the researchers always demonstrate or prove their result to a high degree of rigor themselves. None of this requires an army of PhDs (or even one) to rigorously inspect the result unless it is so badly represented as to be opaque. It is not like they sit around coming up with random ideas for how to solve a problem and leave it at that. Peer review is useful for checking the result but you aren't supposed to rely on peer review to filter garbage ideas. I know a few professional computer science researchers, people who churn out new and better algorithms (some radically so) in multiple domains on a surprisingly consistent basis for private companies. At least half their job is actually the easily understandable explanation and proof of why the algorithm must have the claimed properties. Their job isn't to produce academic papers but to solve algorithm problems for software engineers writing real code, so a lot of focus is on reduction to practice and clear explanation of the mechanics.
onion2k 11y ago
Any result needs to stand the rigor of peer-review before the said breakthrough can be conclusively proven. Apple aren't in the business of doing research for the sake of provable science though. They do research to make things they can sell. The goal is very different.
- sangnoir 11y ago
  >The goal is very different. The article agrees, and suggests that because of this, Apple will miss out on the top talent that prefers to publish

wfo 11y ago

This is a potential pitfall of private research. Ideally any real scientist who is earnest and dedicated about his or her work and is passionate about its benefit to humanity or contribution to human knowledge would, upon making a breakthrough like that, feel the need to share it. Look at the plot of Atlas Shrugged: completely unrealistic; Galt is a disgusting sociopath who invents a way to remove resource constraints from the world, essentially destroying markets and capitalism, poverty and hunger, and refuses to share it with humanity so he can hold out for... resources in exchange. These things don't happen. Scientists don't try to keep their work under wraps -- it's the business people who own them that do. The secret private work that these people are doing is a chain around their neck which stops them from being real scientists. Some are willing to give that up for money and work at a lab that doesn't publish. Some are willing to give that up out of a sense of patriotic duty and work in intelligence. It's fantastic that there are private research labs like MSR that pay well and DO publish so there isn't a forced choice between money and being a real scientist for the lucky few who work there.

robotresearcher 11y ago

The incentive for academic labs and researchers to go public is very clear: publications and kudos in the community, which leads to career advancement and plentiful research funding. You don't get super-rich like this, but you get the things that people really want. Respect of your peers, feeling like you are doing something useful, and enough money to run a lab and live a decent middle-class lifestyle, so you can get on with the work you love. That's a good life. (Also you might want to stay in touch with your grad students who are driven to get rich. Can't hurt.)

mizzao 11y ago

The reality is that's not how research works. Inspiration and breakthroughs don't come out of thin air; they come from the freedom to interact with other scientists, process their ideas and work, and then combine them with a new perspective which is tested and then occasionally turns out to be better than the state of the art. The process of research builds a lot of tacit knowledge that can't just be picked up by secretly picketing machine learning conferences. It's a lot of hard work, and the benefits to doing it in collaboration with the wider scientific community far outweigh the added value from being able to keep some marginal contribution secret. The top-secret, non-collaborating research lab always falls behind, with perhaps the rare exception of certain large and well-funded government organizations.

Houshalter 11y ago

Well they could patent it. This is literally the purpose of patents. You can't patent it if you don't publish, and if you do patent it, you have every incentive to publish it. To get other people to use it and pay you licensing fees, or just to brag and attract better scientists to work at your company. I do hate patents, and I think they are often given to ideas that seems simple, trivial, or iterative, rather than ground breaking. Stuff that probably would have been invented by someone else a few years, or even months, later. But for your hypothetical "revolutionary AI breakthrough", that's the ideal use case for them.