28 ms·

While there is a lot to get excited about with ML both as a consumer and as a software developer I can't help feeling a pang of sadness. Big data and in this c

by bnchrch 10y ago

While there is a lot to get excited about with ML both as a consumer and as a software developer I can't help feeling a pang of sadness. Big data and in this case the relationship (graph) between big data points are whats needed to make great ML/AI products. By nature the only companies that will ever have access to this data in a meaningful way are going to be larger companies: Google, Amazon, Apple, ect. Because of this I worry that small upstarts may never be able to compete on these type of products in a viable way as the requirements to build these features are so easily defensible by the larger incumbents. I hope this is not the case but I'm getting less and less optimistic when I see articles like this.

gordon_freeman 10y ago
It depends on the vision and products too imo. Tesla is much smaller car company than Honda. But Musk just announced that they have collected 222m Autopilot miles from Tesla cars. I assume they have been feeding all that data to ML/AI systems to create fully autonomous vehicles (Level 4). Now I don't think Honda has designed its cars in such a way that allows it to collect data from its Honda Sensing features while the cars are driven.
notliketherest 10y ago
Yes, and it's a land grab for those domains with untapped data and potential for ML/AI related products. New startups that can identify these untapped areas will succeed.
AndrewKemendo 10y ago
I wrote about this a few months ago [1] The one thing that makes me feel slightly better about this is the recently announced AmaGoogFaceSoft partnership on training data open sourcing [2]. I say slightly because they still control the pipes and what is released - so they'll never likely release the good data, also cause it would probably cause privacy issues if they did. It's incumbent on us as the little guys and girls to make new data pipes through innovative products. [1] https://medium.com/@andrewkemendo/the-ai-revolution-will-be-different-from-the-microprocessor-revolution-cb417d324203#.qzp87ojum https://medium.com/@andrewkemendo/the-ai-revolution-will-be-... [2]http://www.partnershiponai.org/ http://www.partnershiponai.org/
tim_sw 10y ago
The techniques they used in the post actually try to overcome the lack of labeled data. (by using semi-supervised techniques/label propagation) You are right that Google has a lot more raw data, but that is much cheaper to obtain, and depending on your problem domain, there could be a bunch of public datasets that are available for you to cobble together. Some of these big tech companies don't necessarily have the right training data for their applications either. For example, Adam Coates from Baidu outlined some of the ways that they generated a better training set for their speech recognition use case here: https://youtu.be/g-sndkf7mCs?t=3817 https://youtu.be/g-sndkf7mCs?t=3817
dunkelheit 10y ago
This is data capitalism. In ordinary capitalism the rich get richer, and in data capitalism those who have data, build superior products, get more users and thus more data, while others starve. Good datasets can be bought but it is scary in its own way.
- bufordsharkley 10y ago
  Is open data + copyleft licenses a possibility?
  - dunkelheit 10y ago
    It is not always feasible - consider for example the dataset of all gmail messages.
- edanm 10y ago
  "In ordinary capitalism the rich get richer [...]". Errr, not really? I mean, at a minimum you're making a somewhat controversial statement - lots of people (me included) wouldn't automatically agree that the rich get richer, especially if you're thinking "only" the rich get richer, which is what is implied by the comment above. (As evidence, see established companies that go bankrupt, and new startups which rise from nothing to become dominant players, much like Google itself, which has existed for a relatively brief period, or something like Uber). So let's hope that data capitalism is like regular capitalism!
  - dunkelheit 10y ago
    Well, things are of course much more complicated than I have presented in my short comment. And economists are still debating whether Piketty's conclusions are valid. But the dynamic I have described is certainly there. And it can become dominant.
  - currysoup_t 10y ago
    Most successful (Valley) startups don't start from nothing, far from it, they start from a vast network of other rich peoples money, usually VC, occasionally the state.
notduncansmith 10y ago
We could build software to help people collect their own data, and make it easier to do things with that information (combination of education, motivation, complementary tools, etc).
choxi 10y ago
A lot of the datasets behind the big image recognition and reinforcement learning (especially recently) have been open sourced: - https://research.googleblog.com/2016/09/announcing-youtube-8m-large-and-diverse.html?m=1 https://research.googleblog.com/2016/09/announcing-youtube-8... - https://techcrunch.com/2016/10/05/udacity-open-sources-an-additional-183gb-of-driving-data/ https://techcrunch.com/2016/10/05/udacity-open-sources-an-ad... - http://image-net.org/ http://image-net.org/ There are also more paid, private dataset services now: - https://fantasydata.com/pricing/nfl-data-api.aspx https://fantasydata.com/pricing/nfl-data-api.aspx - https://www.quandl.com/ https://www.quandl.com/ I could see there being a lot of opportunity for startups that democratize data, which would then unlock more startups to build novel ML architectures on top of.
- Chronic9q 10y ago
  I know for a fact (and you can check online too), that Google's non-public datasets are magnitudes larger in training size and far superior in learning signal (labels are very close to their task). Google has thousands of real robotic manipulator arms to train their reinforcement learning algorithms. Algorithms trained using the OpenAI gym are practically guaranteed to fail in the real world since OpenAI gym is (designed as) a perfect simulator. Don't even get me started on internal image and speech recognition datasets collected from YouTube, search indexing, and Hangouts/Duo calls. There is no way in hell you can compete with this. The closest you have is Facebook. But they've built an AI that has just learned how to play Go. Let alone defeat a world champion. Hell, Google is already working on beating humans at Starcraft.
babo 10y ago
This technique could help to break this monopoly of big data as it relies less on labelling which is even harder and more expensive for a large dataset than the actual data.
mindcrime 10y ago
By nature the only companies that will ever have access to this data in a meaningful way are going to be larger companies: Google, Amazon, Apple, ect. Because of this I worry that small upstarts may never be able to compete on these type of products in a viable way as the requirements to build these features are so easily defensible by the larger incumbents. Wow, what's with all the fatalism around here? C'mon, people have been making variations of this argument for hundreds of years, but small companies still keep coming along and knocking the KOTM off its perch. There's always a new way to do things, a new angle, a new hack, something that hasn't been done before, that can (somewhat) level the playing field (for a while).
- etangent 10y ago
  1) Buy a $200 drone, fly it around for a day and record video output. 2) Write a web scraper and run it for a day on an AWS instance. 3) Download any of the publicly released data sets. There are lot of interesting ones out there, including medical imaging, etc. 4) Hack a video game engine to generate data. 5) Write your own data generator (could even be a generative NN model). Not all data can be simulated, but when it can be, doing so is incredibly powerful. A few of the above options were not available just 5-7 years ago. Data is becoming more publicly accessible, not less.
  - jbpetersen 10y ago
    That doesn't preclude the same outcome as globalization where the "rich" accumulate a much greater proportion of overall wealth while the "poor" are still left much better off in absolute terms. To say nothing of falling in-between.
rspeer 10y ago
The graph algorithm they're describing is basically Manaal Faruqui's "retrofitting", although they don't cite him. I will make the charitable assumption that they came up with it independently (and managed to miss the excitement about it at NAACL 2015). Here's why not to be sad: Retrofitting and its variants are quite effective, and surprisingly, they're not really that computationally intensive. I use an extension of retrofitting to build ConceptNet Numberbatch [1], which is built from open data and is the best-performing semantic model on many word-relatedness benchmarks. The entirety of ConceptNet, including Numberbatch, builds in 5 hours on a desktop computer. Big companies have resources, but they also have inertia. Some problems are solved by throwing all the resources of Google at the problem, but that doesn't mean it's the only way to solve the problem. [1] https://github.com/commonsense/conceptnet-numberbatch https://github.com/commonsense/conceptnet-numberbatch
- igravious 10y ago
  I can't find a single comment thread on HN where ConceptNet is discussed. Bummer. See here: https://hn.algolia.com/?query=conceptnet&sort=byPopularity&prefix&page=0&dateRange=all&type=story https://hn.algolia.com/?query=conceptnet&sort=byPopularity&p... Short of signing up to the mailing list on Google Groups is there any info out there on the net that tells me: a) How ConceptNet relates to efforts like Cyc's OpenCyc and the big G's Knowledge Graph? b) How to add domain specific concepts, attributes, and relations to ConceptNet Both these things are the first that pop into my mind but the website: http://conceptnet5.media.mit.edu/ http://conceptnet5.media.mit.edu/ doesn't seem to cover them.
  - rspeer 10y ago
    Hm, I know I've brought it up before. In relation to OpenCyc: ConceptNet doesn't have CycL or any sort of logical predicates -- the assumption is that you are just doing fuzzy machine learning over ConceptNet. I'm pretty sure you need the rest of Cyc to do anything interesting with CycL, though. ConceptNet takes advantage of linked open data when it can, so it does contain attempted translations of the facts in OpenCyc. Compared to the Google Knowledge Graph: well, you can't really use the Knowledge Graph without being at Google, or making a research agreement with them, right? But from what I've seen of it, and of its predecessor Freebase, I think it focuses a lot on named entities: particularly things you can look up on Wikipedia or things you can buy. Which is fine information. I'd say it's a different segment of linked data than "what words mean", which is ConceptNet's focus. So maybe think of it as more like a bigger WordNet than a smaller Knowledge Graph. How to add domain-specific concepts to ConceptNet: the unsatisfying answer is that you can get the code for ConceptNet and alter the build process to include new data sources (try it on the 5.5 branch if you attempt this). And the startuppy sellout answer is that doing this automatically is one of the things my company Luminoso is for (http://www.luminoso.com http://www.luminoso.com). Thanks for asking, and I'll try to clarify this kind of stuff when I deploy the new site.
WWKong 10y ago
I'm more optimistic about this. I think of it in terms of Big Co has tons of data. Does all the learning. Shares the learning through API for a small fee (very small relative to Small Co having to come up with the learning). For example I upload a picture and Google tells me it is a cat.
artilect 10y ago
Stop thinking about products and money based on ML/AI, start thinking about AGI, research, science and the possibilities AGI could bring. You don't need big data to build AGI. This is a step in the right direction, and from experience, a small research-only startup could achieve very similar results. More than likely, the amount of data is a detriment to creating a better system.
briceb8e 10y ago
Couldn't Blockchain technologies along with big open source projects become a great source of big data? Ideally used with open-source projects. I'm thinking Augur for one but a lot of projects connected to the Ethereum network feels they could be great source of data that can be analyzed. I agree right now big data centralisation is a problem indeed, but I don't believe it will last. I mean, those big companies will remain but Blockchain tech is fairly likely to take a big spot in there with them, I believe.
- singham 10y ago
  You have gone tangent without explaining hows of your opinion.
deleted 10y ago
[deleted]
amelius 10y ago
Even if you have the data, you also need the processing power.

by bnchrch 10y ago

gordon_freeman 10y ago

It depends on the vision and products too imo. Tesla is much smaller car company than Honda. But Musk just announced that they have collected 222m Autopilot miles from Tesla cars. I assume they have been feeding all that data to ML/AI systems to create fully autonomous vehicles (Level 4). Now I don't think Honda has designed its cars in such a way that allows it to collect data from its Honda Sensing features while the cars are driven.

notliketherest 10y ago

Yes, and it's a land grab for those domains with untapped data and potential for ML/AI related products. New startups that can identify these untapped areas will succeed.

AndrewKemendo 10y ago

I wrote about this a few months ago [1] The one thing that makes me feel slightly better about this is the recently announced AmaGoogFaceSoft partnership on training data open sourcing [2]. I say slightly because they still control the pipes and what is released - so they'll never likely release the good data, also cause it would probably cause privacy issues if they did. It's incumbent on us as the little guys and girls to make new data pipes through innovative products. [1] https://medium.com/@andrewkemendo/the-ai-revolution-will-be-different-from-the-microprocessor-revolution-cb417d324203#.qzp87ojum https://medium.com/@andrewkemendo/the-ai-revolution-will-be-... [2]http://www.partnershiponai.org/ http://www.partnershiponai.org/

tim_sw 10y ago

The techniques they used in the post actually try to overcome the lack of labeled data. (by using semi-supervised techniques/label propagation) You are right that Google has a lot more raw data, but that is much cheaper to obtain, and depending on your problem domain, there could be a bunch of public datasets that are available for you to cobble together. Some of these big tech companies don't necessarily have the right training data for their applications either. For example, Adam Coates from Baidu outlined some of the ways that they generated a better training set for their speech recognition use case here: https://youtu.be/g-sndkf7mCs?t=3817 https://youtu.be/g-sndkf7mCs?t=3817

dunkelheit 10y ago

This is data capitalism. In ordinary capitalism the rich get richer, and in data capitalism those who have data, build superior products, get more users and thus more data, while others starve. Good datasets can be bought but it is scary in its own way.

bufordsharkley 10y ago
Is open data + copyleft licenses a possibility?
- dunkelheit 10y ago
  It is not always feasible - consider for example the dataset of all gmail messages.
edanm 10y ago
"In ordinary capitalism the rich get richer [...]". Errr, not really? I mean, at a minimum you're making a somewhat controversial statement - lots of people (me included) wouldn't automatically agree that the rich get richer, especially if you're thinking "only" the rich get richer, which is what is implied by the comment above. (As evidence, see established companies that go bankrupt, and new startups which rise from nothing to become dominant players, much like Google itself, which has existed for a relatively brief period, or something like Uber). So let's hope that data capitalism is like regular capitalism!
- dunkelheit 10y ago
  Well, things are of course much more complicated than I have presented in my short comment. And economists are still debating whether Piketty's conclusions are valid. But the dynamic I have described is certainly there. And it can become dominant.
- currysoup_t 10y ago
  Most successful (Valley) startups don't start from nothing, far from it, they start from a vast network of other rich peoples money, usually VC, occasionally the state.

notduncansmith 10y ago

We could build software to help people collect their own data, and make it easier to do things with that information (combination of education, motivation, complementary tools, etc).

choxi 10y ago

A lot of the datasets behind the big image recognition and reinforcement learning (especially recently) have been open sourced: - https://research.googleblog.com/2016/09/announcing-youtube-8m-large-and-diverse.html?m=1 https://research.googleblog.com/2016/09/announcing-youtube-8... - https://techcrunch.com/2016/10/05/udacity-open-sources-an-additional-183gb-of-driving-data/ https://techcrunch.com/2016/10/05/udacity-open-sources-an-ad... - http://image-net.org/ http://image-net.org/ There are also more paid, private dataset services now: - https://fantasydata.com/pricing/nfl-data-api.aspx https://fantasydata.com/pricing/nfl-data-api.aspx - https://www.quandl.com/ https://www.quandl.com/ I could see there being a lot of opportunity for startups that democratize data, which would then unlock more startups to build novel ML architectures on top of.

Chronic9q 10y ago
I know for a fact (and you can check online too), that Google's non-public datasets are magnitudes larger in training size and far superior in learning signal (labels are very close to their task). Google has thousands of real robotic manipulator arms to train their reinforcement learning algorithms. Algorithms trained using the OpenAI gym are practically guaranteed to fail in the real world since OpenAI gym is (designed as) a perfect simulator. Don't even get me started on internal image and speech recognition datasets collected from YouTube, search indexing, and Hangouts/Duo calls. There is no way in hell you can compete with this. The closest you have is Facebook. But they've built an AI that has just learned how to play Go. Let alone defeat a world champion. Hell, Google is already working on beating humans at Starcraft.

babo 10y ago

This technique could help to break this monopoly of big data as it relies less on labelling which is even harder and more expensive for a large dataset than the actual data.

mindcrime 10y ago

By nature the only companies that will ever have access to this data in a meaningful way are going to be larger companies: Google, Amazon, Apple, ect. Because of this I worry that small upstarts may never be able to compete on these type of products in a viable way as the requirements to build these features are so easily defensible by the larger incumbents. Wow, what's with all the fatalism around here? C'mon, people have been making variations of this argument for hundreds of years, but small companies still keep coming along and knocking the KOTM off its perch. There's always a new way to do things, a new angle, a new hack, something that hasn't been done before, that can (somewhat) level the playing field (for a while).

etangent 10y ago
1) Buy a $200 drone, fly it around for a day and record video output. 2) Write a web scraper and run it for a day on an AWS instance. 3) Download any of the publicly released data sets. There are lot of interesting ones out there, including medical imaging, etc. 4) Hack a video game engine to generate data. 5) Write your own data generator (could even be a generative NN model). Not all data can be simulated, but when it can be, doing so is incredibly powerful. A few of the above options were not available just 5-7 years ago. Data is becoming more publicly accessible, not less.
- jbpetersen 10y ago
  That doesn't preclude the same outcome as globalization where the "rich" accumulate a much greater proportion of overall wealth while the "poor" are still left much better off in absolute terms. To say nothing of falling in-between.

rspeer 10y ago

The graph algorithm they're describing is basically Manaal Faruqui's "retrofitting", although they don't cite him. I will make the charitable assumption that they came up with it independently (and managed to miss the excitement about it at NAACL 2015). Here's why not to be sad: Retrofitting and its variants are quite effective, and surprisingly, they're not really that computationally intensive. I use an extension of retrofitting to build ConceptNet Numberbatch [1], which is built from open data and is the best-performing semantic model on many word-relatedness benchmarks. The entirety of ConceptNet, including Numberbatch, builds in 5 hours on a desktop computer. Big companies have resources, but they also have inertia. Some problems are solved by throwing all the resources of Google at the problem, but that doesn't mean it's the only way to solve the problem. [1] https://github.com/commonsense/conceptnet-numberbatch https://github.com/commonsense/conceptnet-numberbatch

igravious 10y ago
I can't find a single comment thread on HN where ConceptNet is discussed. Bummer. See here: https://hn.algolia.com/?query=conceptnet&sort=byPopularity&prefix&page=0&dateRange=all&type=story https://hn.algolia.com/?query=conceptnet&sort=byPopularity&p... Short of signing up to the mailing list on Google Groups is there any info out there on the net that tells me: a) How ConceptNet relates to efforts like Cyc's OpenCyc and the big G's Knowledge Graph? b) How to add domain specific concepts, attributes, and relations to ConceptNet Both these things are the first that pop into my mind but the website: http://conceptnet5.media.mit.edu/ http://conceptnet5.media.mit.edu/ doesn't seem to cover them.
- rspeer 10y ago
  Hm, I know I've brought it up before. In relation to OpenCyc: ConceptNet doesn't have CycL or any sort of logical predicates -- the assumption is that you are just doing fuzzy machine learning over ConceptNet. I'm pretty sure you need the rest of Cyc to do anything interesting with CycL, though. ConceptNet takes advantage of linked open data when it can, so it does contain attempted translations of the facts in OpenCyc. Compared to the Google Knowledge Graph: well, you can't really use the Knowledge Graph without being at Google, or making a research agreement with them, right? But from what I've seen of it, and of its predecessor Freebase, I think it focuses a lot on named entities: particularly things you can look up on Wikipedia or things you can buy. Which is fine information. I'd say it's a different segment of linked data than "what words mean", which is ConceptNet's focus. So maybe think of it as more like a bigger WordNet than a smaller Knowledge Graph. How to add domain-specific concepts to ConceptNet: the unsatisfying answer is that you can get the code for ConceptNet and alter the build process to include new data sources (try it on the 5.5 branch if you attempt this). And the startuppy sellout answer is that doing this automatically is one of the things my company Luminoso is for (http://www.luminoso.com http://www.luminoso.com). Thanks for asking, and I'll try to clarify this kind of stuff when I deploy the new site.

WWKong 10y ago

I'm more optimistic about this. I think of it in terms of Big Co has tons of data. Does all the learning. Shares the learning through API for a small fee (very small relative to Small Co having to come up with the learning). For example I upload a picture and Google tells me it is a cat.

artilect 10y ago

Stop thinking about products and money based on ML/AI, start thinking about AGI, research, science and the possibilities AGI could bring. You don't need big data to build AGI. This is a step in the right direction, and from experience, a small research-only startup could achieve very similar results. More than likely, the amount of data is a detriment to creating a better system.

briceb8e 10y ago

Couldn't Blockchain technologies along with big open source projects become a great source of big data? Ideally used with open-source projects. I'm thinking Augur for one but a lot of projects connected to the Ethereum network feels they could be great source of data that can be analyzed. I agree right now big data centralisation is a problem indeed, but I don't believe it will last. I mean, those big companies will remain but Blockchain tech is fairly likely to take a big spot in there with them, I believe.

singham 10y ago
You have gone tangent without explaining hows of your opinion.

deleted 10y ago

[deleted]

amelius 10y ago

Even if you have the data, you also need the processing power.