8 ms·

The Machine Learning Race Is Really a Data Race

208 points by prostoalex 8y ago

autokad 8y ago
as a data scientist, it has always been about the data. it has always been a data race. google has known that, which is why they spent billions trying to protect their search moat.
- anongraddebt 8y ago
  This. All things being equal - and assuming similar data quality among competitors - the one with the most data has a stronger strategic position. This is because, all things being equal, more data means a larger number of available choices and insights (I'm making some simplifying assumptions here). Business strategy is fundamentally about trade-offs, though, so we do need a caveat. Naturally, more choices and more insights can, at times, be a weakness. You always have to prioritize, and as data volume increases it doesn't seem the ability for quality prioritization grows in parallel (or at least, necessarily does).
minimaxir 8y ago
The neat thing about big data is that there are massive diminishing returns on the amount of data vs. model quality. What's more important is data quality (e.g. structured and unbiased), and an incumbent with the right approach can have a strong impact.
- ska 8y ago
  I think you are saying in essence, data quality trumps data quantity, which in turn trumps modeling and algorithmics. This seems to hold fairly well, with a few caveats. For example, without enough data you ar wasting your t8me with some techniques. Also “data quality” most often needs to include quality labeling in practice (but you may have meant that inclusively).
- dumbfoundded 8y ago
  It's not just quality but it's also the available features in the data. Quality is just a signal to noise problem. If you have enough data, you generally can reasonably segment it to get higher quality. Obtaining features not present in other data sets is probably the most significant factor. For example, let's say you want to build a speech rec engine. You need 15K hours of data to build/validate a model. How would you get that? You could farm it out to some people on mechanical turk and get 15K hours of audio transcribed. With enough money, you could duplicate the transcriptions enough times to actually be pretty sure about the quality of the data set. If you're clever and have a large enough dataset, segmentation generally gets you decent quality. The big gains come in when you have features not present. For example, google realized when you build a speech rec engine, you can include video data and image processing to actually use the way people move their mouths to significantly increase the quality of an automated transcription.
  - state_less 8y ago
    I think creative new ways of harvesting data will continue to be profitable. That's fine with today's mindset, but over time it seems like we'll want to go beyond big data and think about how it is that a child can become quite capable without seeing millions of instances of people crossing a road, etc... We have to make our machines make models and 'want' to gather data to test them. By having the machines do the work of today's data scientist, they may well put themselves out of business, and quickly everyone else.
DATACOMMANDER 8y ago
The article is decent, but I see two mistakes: the author assumes that AI/ML can’t produce unique insights from public data; and she conflates AI/ML with automation. While it’s true that you won’t find anything that your competitors haven’t if you use the same data and the same AI/ML techniques, there’s nothing stopping companies from differentiating on the techniques in addition to (or even instead of) the data. If you just use plug-and-play AI, then sure, you’ll need a unique data set if you want unique results. The section about finding faster, less error-prone ways to apply existing insights sounds more like automation than AI. There’s certainly overlap, but they’re two different things.
zozbot123 8y ago
Clearly, the solution is to develop a machine learning framework in Rust - Rust is 100% immune to data races!
buboard 8y ago
Data is cheap though and a very effective way to get them is crowdsourcing. The differentiation may not last long.
DrNuke 8y ago
There are reliable ways to generate unbiased, synthetic and even big data for almost any industrial domain out there plus a lot of applied research fields in the STEM curricula. The most important issue nowadays is the accountability of results aka ablation studies following too many false positives and a number of recent, blatant scams.
sirwitti 8y ago
Me personally I'm not 100% convinced that the current ML/AI approaches (that I'm aware of) will yield new big steps forward. Neuromorphic computing could have the potential I think, if we can build hardware that's good enough.
- visarga 8y ago
  This year brought to us huge improvements in handling text (BERT), speech (Google TTS, especially the Allo demo), images (ProGAN, StyleGAN, BigGAN) and activities such as games (Alpha Zero), and robotics. Even music composition is improving a lot. I don't feel it is slowing down yet. And when it will eventually slow down, it will be all for the better - we will have time to reopen all those different approaches that have been more or less ignored because of the DL hype. I think the key fields in AI will become simulator based learning (RL) and graph processing neural nets, because graphs can express any kind of highly dimensional data and are useful for reasoning tasks. They marry the symbolic and connectionist approaches. These two subdomains have had rapid evolution over the last couple of years. They also solve the data problem - in simulation you can produce as much data as you want, and graphs have combinatorial generalisation, thus they work on new configurations without retraining.
- PinkMilkshake 8y ago
  I often feel the same. I could show someone a single photograph of an animal they have never seen before and they will recognize it forever, from different angles, in different lighting, in black and white, probably even from a silhouette. The more I dig in to ML the more it seems like it's cheating, like a mathematical trick. At least the way we are using it. I can't help but feel like it's a game of Pachinko with pixels instead of balls, and neural weights instead of pins and holes. I'm no expert though so take my opinion with a grain of salt.
ohthehugemanate 8y ago
Data quantity and quality are key. Both, though. This is why it's foolish to go up against an ML product from Google, Facebook, or (maybe) Microsoft. You just can't compete with the volume and quality of data they can access. In sectors like automotive, where every brand is competing to try and build the best predictors using only their own data, there is a huge opportunity available for the first two companies to share data with each other. Doubling your data quantity brings a significant improvement to any model, and would put them ahead of any competition. That advantage only grows the more players you add to the sharing pool. I believe that if humanity is really going to harness machine learning, the concept of a bulk data commons is an inevitable requirement. This makes decentralized personal data control, homomorphic encryption, and similar technologies incredibly important.
- nightcracker 8y ago
  > This makes decentralized personal data control, homomorphic encryption, and similar technologies incredibly important. Call me a cynic, but we can't even get companies to do the bare minimum to protect data that is centralized, and you expect them to do all that? I'm not kidding when I say that your last sentence is a bigger, harder and more important task than the machine learning it is supposed to facilitate.
  - ngcc_hk 8y ago
    Yes. Found that sentence very odd if one accept the idea of the more data needed meant sacrify all those values ... otherwise just ask around ... hence may be china and its firms will win as they do not need to.
- jonathankoren 8y ago
  They way you compete with Google et al, is to find a niche they think they’re serving, but in reality completely suck at.
  - hayksaakian 8y ago
    If someone would like an example here, check out this animated video filters (see snapchat, tiktok etc). Just because Google et al have the underlying technology, doesn't mean they're the best at creating a product that uses that tech. Certainly gives them a leg-up to get started, but it's not the end-all be-all.
Xcelerate 8y ago
It's a data race because we've run up against another wall on the algorithms side. Find a technique that works better than GBDT for the same type of problem. Other than some minor tweaks described in the academic literature, it's been a while since something really advanced the state of the art. Small datasets still have massive predictive potential; we just need better algorithms. (As an extreme example, suppose I give you the first 30 digits of pi or e and ask you to predict what comes next. Despite being a small amount of data of low algorithmic complexity, machine learning cannot currently handle this type of problem.)
- ydj 8y ago
  The pi and e example seems more complicated than it looks. If you ask a human who doesn’t know about pi or e, how much effort would it take for them to figure out the next digits? Seems like they’d have to rediscover the math first (or I suppose, perform a google search)
  - montenegrohugo 8y ago
    Yes, it would be a hugely complicated undertaking and probably impossible for most humans with little academic mathematical knowledge. But the point is that it would be possible, which indicates that the problem does not necessarily lie in the amount of data but in the algorithmic approach itself. ML is a great tool that is creating very real and tangible value, but it still has ways to go. Just adding more computational capabilities and more data will only bring marginal improvements.
- 2sk21 8y ago
  I completely agree with this. I subscribe to the Gary Marcus' view that what we need is the marriage of symbolic AI and ML.
nuclx 8y ago
"Machines need a lot more data than humans do in order to get smart" - is that true?
- zozbot123 8y ago
  The general answer is yes - it's the "one shot learning" problem. But humans also have access to a large (if not very well defined) amount of data via background knowledge, which plausibly enables them to apply stronger, more effective priors to any one task.
  - lordnacho 8y ago
    This is the thing. Machines start on a blank slate each time, humans never. So how do you compare the amounts of data needed?
    - ars 8y ago
      So try it: Teach a computer about animals in general. Then show it a single example of a new animal it has never seen, and see how well it would do at telling you if challenge images are or are not that animal. It's not even going to be close to what a human would manage. It's not because of the data, it's because a human understands what he's seeing.
      - lordnacho 8y ago
        Then you're comparing a machine that's never seen anything but animals against a human that has no separation between different kinds of data and can't start on a blank slate. I may have all sorts of images in my mind, allowing me to make the distinction?
      - dmurray 8y ago
        This doesn't sound too different from face recognition systems where you train a network to be able to tell if two faces are the same or different. I don't know what the state of the art in animal recognition is, though.
lordnacho 8y ago
It kinda can't be any other way though. What are models? They're a way to describe the data. So this is where a load of philosophical stuff like Occam's Razor comes in and favours things like having fewer degrees of freedom, lower errors, etc. What is data? It's what makes one model a more likely explanation than another. You can make up any number of ways to describe some phenomenon, but without data there's no way to tell which of them is better. Or rather, you will fall back on some model with fewer specifics, because of those considerations we mentioned earlier. So getting smarter (ie more complex) with models can't help on its own.
- darawk 8y ago
  I'm not sure i'm understanding you, or i'm not sure I agree. Humans, for instance, can learn from substantially less data than (most) machine learning models, if the data is presented in a way that we're optimized for (e.g. human facial recognition). If we can devise ML algorithms capable of generalizing from substantially smaller datasets, while data will still be relevant, it will be a less important bottleneck in the process than it is now.
- joe_the_user 8y ago
  Human beings can be smart in using a fairly small set of data. Humans have an enormous store of data to work with but when confronted with a bit of new data on a new subject, they can sometimes do very well. This is a result of the ML paradigm essentially starting new with whatever data set it is trying to "solve" but overall, the paradigm doesn't have to be that.
  - PeterisP 8y ago
    IMHO pretty much every example I've seen for how humans can get good results with using a fairly small set of data actually involves successful generalization / transfer of "generic life experience" or "generic audiovisual processing" to that new problem instead of actual learning from limited data. We know how to do that in ML, in general - we can do transfer learning from few examples quite well if the underlying generic data is good enough. However, we currently don't have good enough underlying generic data to match the years of generic life data that any human kid has accumulated. A human learning to play an Atari game that involves an agent jumping over a pit has to learn only the mechanics of that game, and that can be done in minutes. Learning that game from scratch, on the other hand, requires also learning interpreting vision and the whole concept that the world has objects that may move around - which takes months of learning for human brain. So comparing sample efficiency is an apples to elephants comparison if we disregard the ability to reuse/transfer knowledge from related tasks that all humans learn during childhood.
miguelrochefort 8y ago
Don't they have more than enough data by now? What are they waiting for?
miguelrochefort 8y ago
What will happen is that we will move toward an agent-centric system like Holochain, and put all data about our environment there. Only when people will realize that transparency is better than privacy, will they start putting all their quantified self data on there. This will become a Decentralized Artificial Intelligence network, from which consciousness and AGI will emerge.
synaesthesisx 8y ago
Mark my words - organizations like Amazon, Google, etc will soon start offering a Data Marketplace (intended for both buyers and sellers of all sorts of "alternative" data - everything from small business metrics to enterprise/b2b and anything in-between. The next logical step would be to offer insight/models as a service as a layer built on top of this.. I've been studying their moves carefully and have no doubt this is where they're headed. While I don't work for Amazon I think AWS in particular is uniquely positioned for these next couple of moves to a level that may make it challenging for others to compete. Interesting times...
- joe_the_user 8y ago
  Indeed, the clever thing would be to have an API flexible enough to allow a user to utilize their own ML architecture and training process but it still keep that user from accessing the raw data. The ideal approach would somehow allow this anonymized access to multiple large databases simultaneously. I don't know how you'd do that but if you claim Etherium would help, you'd arrive at buzzword nirvana even if you were wrong.
- AznHisoka 8y ago
  “I've been studying their moves carefully” What moves specifically?
mslate 8y ago
No, it's a process race. The data is just the most visible "asset" in play. The entities that own the best data mining process will win. This includes data collection + ETL/storage + model training/deployment. Obtaining a vertical monopoly on this process is the goal.
yonkshi 8y ago
The sample efficiency of ML systems are increasing rapidly in the theoretical realm, though the available data are growing magnitudes faster than the progress in ML sample efficiency. The article has a valid point: It's far cheaper to hoard data than to invent a more efficient ML system, thus we will see people race towards data rather than technical complexity.
- fooker 8y ago
  >The sample efficiency of NN are increasing rapidly in the theoretical realm. Can you provide some references for this?
  - yonkshi 8y ago
    Sure. I don't have any holistic survey to prove my point, but an example of recent progress in terms of sample efficiency is this paper[0]. Derivatives of this paper have been used to solve Sudoku[1], Starcraft II[2] and more [3]. This paper enabled more efficient use of data by creating a probabilistic graphical model between logical sets. [0]https://arxiv.org/abs/1706.01427 https://arxiv.org/abs/1706.01427 [1]https://arxiv.org/abs/1711.08028 https://arxiv.org/abs/1711.08028 [2]https://arxiv.org/abs/1806.01830 https://arxiv.org/abs/1806.01830 [3]https://arxiv.org/abs/1806.01822 https://arxiv.org/abs/1806.01822
    - state_less 8y ago
      I like the 3D datasets in these papers since, like a scientist in a lab, you can setup the experiment and explore the domain. Adding time in would be cool too (e.g. Are the blue and red ball going to collide in 10 seconds?) It also helps to be able to show you can answer some of these questions in principle with your model. It gives you hope that it might be able to cover real world images.
gesman 8y ago
THIS. I was just saying to our partner (as well as to my wife!) how lucky we are to work on Healthcare solutions. We have access to data about medications, opioids, patients and physicians behaviors that so very little of others (who can have any clue about data analytics) has. The realization of sitting on a goldmine of impossible-to-access-data + capability of developing cutting edge analytics solutions that could change the world is the best place to be.
- tonyhb 8y ago
  Google partners with the NHS to get medical data.
karmasimida 8y ago
Model is cheap, data is HARD. Plus investing is data is much more predictable: the outcome is always going to get better, though the margin will be diminishing, but better is better. While modeling is not, hiring 100 machine learning 'experts' will not solve the problem 100 times better than 10 of them. On the other side, 100 labelers are surely going to provide 10 times of throughput provided the same upscale.
- aqme28 8y ago
  > though the margin will be diminishing So by the metric of model performance (the only thing that matters here), what you're saying is that hiring more labelers is actually not linear.
  - karmasimida 8y ago
    True, of course. We can only go as high as 100% for accuracy, right? However the labellers are scalable in terms of their throughput and coverage of the data, you can always find bad examples or holes in your current data plane, and that is the time when labelers, not the scientists, are going to rescue you
makewavesnotwar 8y ago
It's incredible. The last company I worked for before going it alone (I was the front-end engineer and moved away from ML based business models) was trying to automate statistical analysis. I came from an academic background in economics and I tried to propose modelling to them in simple terms and they jumped to, "it sounds like you're talking about forecasting, let's go with that." And they started trying to implement python models from academic papers with highly limited training data and my reaction was generally, WTF?! You can't even start to forecast without a reliable base model. But they went ahead trying to sell stuff like churn prediction to companies with 0 understanding of how these models work at the basest levels. And yeah, Google started to throw their hat in the game with Analytics 360 and an enormously larger training base. Amazon's another major player. Weirdly enough though, people do still blindly pay my previous employers to figure stuff out because easy answers are always actionable, even if they're wrong. It's just crazy because the CEO explained to me that lying about the service to potential customers and investors was necessary because "Faking it til you make it" was a sound business principle in his mind like 1980's Michael J Fox was his primary sources of business info. Long story short, don't waste your time with these little companies purporting ML holy grails. They're probably just lying to you, whether intentionally or not. ML is a game for the big boys with access to market level aggregates. The models that last company came up with were wildly inaccurate.
- ryebreadistasty 8y ago
  That's obvious to anyone with any sort of basic idea as to how machine learning works. Feed bots data and test the bots predictions - the more data you have then obviously it'll be better. If you have 1 picture of a bee to test you'd only ever get one very specific shape for a bee from any AI, if you have thousands, you get a much better representation. It's pretty simple.
  - sgt101 8y ago
    This is mostly true at the minute, but not completely and probably not for the long term (10 years +) See http://science.sciencemag.org/content/350/6266/1332.full http://science.sciencemag.org/content/350/6266/1332.full Also you can have problems where more and more data doesn't make an impact - the stock market being a clear example.
craftinator 8y ago
I don't understand how this is a race at all. There is no way to finish it, and "going fast" is likely to cause major issues in developing sound technologies. Maybe changing the attitude of this article to a more scientifically sound position would increase it's relevance.
mark_l_watson 8y ago
I have just started working with Ocean Protocol (https://oceanprotocol.com/ https://oceanprotocol.com/) and I have set up a local meetup in January for local entrepreneurs looking for access to machine learning data.

208 points by prostoalex 8y ago

as a data scientist, it has always been about the data. it has always been a data race. google has known that, which is why they spent billions trying to protect their search moat.

anongraddebt 8y ago
This. All things being equal - and assuming similar data quality among competitors - the one with the most data has a stronger strategic position. This is because, all things being equal, more data means a larger number of available choices and insights (I'm making some simplifying assumptions here). Business strategy is fundamentally about trade-offs, though, so we do need a caveat. Naturally, more choices and more insights can, at times, be a weakness. You always have to prioritize, and as data volume increases it doesn't seem the ability for quality prioritization grows in parallel (or at least, necessarily does).

minimaxir 8y ago

The neat thing about big data is that there are massive diminishing returns on the amount of data vs. model quality. What's more important is data quality (e.g. structured and unbiased), and an incumbent with the right approach can have a strong impact.

ska 8y ago
I think you are saying in essence, data quality trumps data quantity, which in turn trumps modeling and algorithmics. This seems to hold fairly well, with a few caveats. For example, without enough data you ar wasting your t8me with some techniques. Also “data quality” most often needs to include quality labeling in practice (but you may have meant that inclusively).
dumbfoundded 8y ago
It's not just quality but it's also the available features in the data. Quality is just a signal to noise problem. If you have enough data, you generally can reasonably segment it to get higher quality. Obtaining features not present in other data sets is probably the most significant factor. For example, let's say you want to build a speech rec engine. You need 15K hours of data to build/validate a model. How would you get that? You could farm it out to some people on mechanical turk and get 15K hours of audio transcribed. With enough money, you could duplicate the transcriptions enough times to actually be pretty sure about the quality of the data set. If you're clever and have a large enough dataset, segmentation generally gets you decent quality. The big gains come in when you have features not present. For example, google realized when you build a speech rec engine, you can include video data and image processing to actually use the way people move their mouths to significantly increase the quality of an automated transcription.
- state_less 8y ago
  I think creative new ways of harvesting data will continue to be profitable. That's fine with today's mindset, but over time it seems like we'll want to go beyond big data and think about how it is that a child can become quite capable without seeing millions of instances of people crossing a road, etc... We have to make our machines make models and 'want' to gather data to test them. By having the machines do the work of today's data scientist, they may well put themselves out of business, and quickly everyone else.

DATACOMMANDER 8y ago

The article is decent, but I see two mistakes: the author assumes that AI/ML can’t produce unique insights from public data; and she conflates AI/ML with automation. While it’s true that you won’t find anything that your competitors haven’t if you use the same data and the same AI/ML techniques, there’s nothing stopping companies from differentiating on the techniques in addition to (or even instead of) the data. If you just use plug-and-play AI, then sure, you’ll need a unique data set if you want unique results. The section about finding faster, less error-prone ways to apply existing insights sounds more like automation than AI. There’s certainly overlap, but they’re two different things.

zozbot123 8y ago

Clearly, the solution is to develop a machine learning framework in Rust - Rust is 100% immune to data races!

buboard 8y ago

Data is cheap though and a very effective way to get them is crowdsourcing. The differentiation may not last long.

DrNuke 8y ago

There are reliable ways to generate unbiased, synthetic and even big data for almost any industrial domain out there plus a lot of applied research fields in the STEM curricula. The most important issue nowadays is the accountability of results aka ablation studies following too many false positives and a number of recent, blatant scams.

sirwitti 8y ago

Me personally I'm not 100% convinced that the current ML/AI approaches (that I'm aware of) will yield new big steps forward. Neuromorphic computing could have the potential I think, if we can build hardware that's good enough.

visarga 8y ago
This year brought to us huge improvements in handling text (BERT), speech (Google TTS, especially the Allo demo), images (ProGAN, StyleGAN, BigGAN) and activities such as games (Alpha Zero), and robotics. Even music composition is improving a lot. I don't feel it is slowing down yet. And when it will eventually slow down, it will be all for the better - we will have time to reopen all those different approaches that have been more or less ignored because of the DL hype. I think the key fields in AI will become simulator based learning (RL) and graph processing neural nets, because graphs can express any kind of highly dimensional data and are useful for reasoning tasks. They marry the symbolic and connectionist approaches. These two subdomains have had rapid evolution over the last couple of years. They also solve the data problem - in simulation you can produce as much data as you want, and graphs have combinatorial generalisation, thus they work on new configurations without retraining.
PinkMilkshake 8y ago
I often feel the same. I could show someone a single photograph of an animal they have never seen before and they will recognize it forever, from different angles, in different lighting, in black and white, probably even from a silhouette. The more I dig in to ML the more it seems like it's cheating, like a mathematical trick. At least the way we are using it. I can't help but feel like it's a game of Pachinko with pixels instead of balls, and neural weights instead of pins and holes. I'm no expert though so take my opinion with a grain of salt.

ohthehugemanate 8y ago

Data quantity and quality are key. Both, though. This is why it's foolish to go up against an ML product from Google, Facebook, or (maybe) Microsoft. You just can't compete with the volume and quality of data they can access. In sectors like automotive, where every brand is competing to try and build the best predictors using only their own data, there is a huge opportunity available for the first two companies to share data with each other. Doubling your data quantity brings a significant improvement to any model, and would put them ahead of any competition. That advantage only grows the more players you add to the sharing pool. I believe that if humanity is really going to harness machine learning, the concept of a bulk data commons is an inevitable requirement. This makes decentralized personal data control, homomorphic encryption, and similar technologies incredibly important.

nightcracker 8y ago
> This makes decentralized personal data control, homomorphic encryption, and similar technologies incredibly important. Call me a cynic, but we can't even get companies to do the bare minimum to protect data that is centralized, and you expect them to do all that? I'm not kidding when I say that your last sentence is a bigger, harder and more important task than the machine learning it is supposed to facilitate.
- ngcc_hk 8y ago
  Yes. Found that sentence very odd if one accept the idea of the more data needed meant sacrify all those values ... otherwise just ask around ... hence may be china and its firms will win as they do not need to.
jonathankoren 8y ago
They way you compete with Google et al, is to find a niche they think they’re serving, but in reality completely suck at.
- hayksaakian 8y ago
  If someone would like an example here, check out this animated video filters (see snapchat, tiktok etc). Just because Google et al have the underlying technology, doesn't mean they're the best at creating a product that uses that tech. Certainly gives them a leg-up to get started, but it's not the end-all be-all.

Xcelerate 8y ago

It's a data race because we've run up against another wall on the algorithms side. Find a technique that works better than GBDT for the same type of problem. Other than some minor tweaks described in the academic literature, it's been a while since something really advanced the state of the art. Small datasets still have massive predictive potential; we just need better algorithms. (As an extreme example, suppose I give you the first 30 digits of pi or e and ask you to predict what comes next. Despite being a small amount of data of low algorithmic complexity, machine learning cannot currently handle this type of problem.)

ydj 8y ago
The pi and e example seems more complicated than it looks. If you ask a human who doesn’t know about pi or e, how much effort would it take for them to figure out the next digits? Seems like they’d have to rediscover the math first (or I suppose, perform a google search)
- montenegrohugo 8y ago
  Yes, it would be a hugely complicated undertaking and probably impossible for most humans with little academic mathematical knowledge. But the point is that it would be possible, which indicates that the problem does not necessarily lie in the amount of data but in the algorithmic approach itself. ML is a great tool that is creating very real and tangible value, but it still has ways to go. Just adding more computational capabilities and more data will only bring marginal improvements.
2sk21 8y ago
I completely agree with this. I subscribe to the Gary Marcus' view that what we need is the marriage of symbolic AI and ML.

nuclx 8y ago

"Machines need a lot more data than humans do in order to get smart" - is that true?

zozbot123 8y ago
The general answer is yes - it's the "one shot learning" problem. But humans also have access to a large (if not very well defined) amount of data via background knowledge, which plausibly enables them to apply stronger, more effective priors to any one task.
- lordnacho 8y ago
  This is the thing. Machines start on a blank slate each time, humans never. So how do you compare the amounts of data needed?
  - ars 8y ago
    So try it: Teach a computer about animals in general. Then show it a single example of a new animal it has never seen, and see how well it would do at telling you if challenge images are or are not that animal. It's not even going to be close to what a human would manage. It's not because of the data, it's because a human understands what he's seeing.
    - lordnacho 8y ago
      Then you're comparing a machine that's never seen anything but animals against a human that has no separation between different kinds of data and can't start on a blank slate. I may have all sorts of images in my mind, allowing me to make the distinction?
    - dmurray 8y ago
      This doesn't sound too different from face recognition systems where you train a network to be able to tell if two faces are the same or different. I don't know what the state of the art in animal recognition is, though.

lordnacho 8y ago

It kinda can't be any other way though. What are models? They're a way to describe the data. So this is where a load of philosophical stuff like Occam's Razor comes in and favours things like having fewer degrees of freedom, lower errors, etc. What is data? It's what makes one model a more likely explanation than another. You can make up any number of ways to describe some phenomenon, but without data there's no way to tell which of them is better. Or rather, you will fall back on some model with fewer specifics, because of those considerations we mentioned earlier. So getting smarter (ie more complex) with models can't help on its own.

darawk 8y ago
I'm not sure i'm understanding you, or i'm not sure I agree. Humans, for instance, can learn from substantially less data than (most) machine learning models, if the data is presented in a way that we're optimized for (e.g. human facial recognition). If we can devise ML algorithms capable of generalizing from substantially smaller datasets, while data will still be relevant, it will be a less important bottleneck in the process than it is now.
joe_the_user 8y ago
Human beings can be smart in using a fairly small set of data. Humans have an enormous store of data to work with but when confronted with a bit of new data on a new subject, they can sometimes do very well. This is a result of the ML paradigm essentially starting new with whatever data set it is trying to "solve" but overall, the paradigm doesn't have to be that.
- PeterisP 8y ago
  IMHO pretty much every example I've seen for how humans can get good results with using a fairly small set of data actually involves successful generalization / transfer of "generic life experience" or "generic audiovisual processing" to that new problem instead of actual learning from limited data. We know how to do that in ML, in general - we can do transfer learning from few examples quite well if the underlying generic data is good enough. However, we currently don't have good enough underlying generic data to match the years of generic life data that any human kid has accumulated. A human learning to play an Atari game that involves an agent jumping over a pit has to learn only the mechanics of that game, and that can be done in minutes. Learning that game from scratch, on the other hand, requires also learning interpreting vision and the whole concept that the world has objects that may move around - which takes months of learning for human brain. So comparing sample efficiency is an apples to elephants comparison if we disregard the ability to reuse/transfer knowledge from related tasks that all humans learn during childhood.

miguelrochefort 8y ago

Don't they have more than enough data by now? What are they waiting for?

miguelrochefort 8y ago

What will happen is that we will move toward an agent-centric system like Holochain, and put all data about our environment there. Only when people will realize that transparency is better than privacy, will they start putting all their quantified self data on there. This will become a Decentralized Artificial Intelligence network, from which consciousness and AGI will emerge.

synaesthesisx 8y ago

Mark my words - organizations like Amazon, Google, etc will soon start offering a Data Marketplace (intended for both buyers and sellers of all sorts of "alternative" data - everything from small business metrics to enterprise/b2b and anything in-between. The next logical step would be to offer insight/models as a service as a layer built on top of this.. I've been studying their moves carefully and have no doubt this is where they're headed. While I don't work for Amazon I think AWS in particular is uniquely positioned for these next couple of moves to a level that may make it challenging for others to compete. Interesting times...

joe_the_user 8y ago
Indeed, the clever thing would be to have an API flexible enough to allow a user to utilize their own ML architecture and training process but it still keep that user from accessing the raw data. The ideal approach would somehow allow this anonymized access to multiple large databases simultaneously. I don't know how you'd do that but if you claim Etherium would help, you'd arrive at buzzword nirvana even if you were wrong.
AznHisoka 8y ago
“I've been studying their moves carefully” What moves specifically?

mslate 8y ago

No, it's a process race. The data is just the most visible "asset" in play. The entities that own the best data mining process will win. This includes data collection + ETL/storage + model training/deployment. Obtaining a vertical monopoly on this process is the goal.

yonkshi 8y ago

The sample efficiency of ML systems are increasing rapidly in the theoretical realm, though the available data are growing magnitudes faster than the progress in ML sample efficiency. The article has a valid point: It's far cheaper to hoard data than to invent a more efficient ML system, thus we will see people race towards data rather than technical complexity.

fooker 8y ago
>The sample efficiency of NN are increasing rapidly in the theoretical realm. Can you provide some references for this?
- yonkshi 8y ago
  Sure. I don't have any holistic survey to prove my point, but an example of recent progress in terms of sample efficiency is this paper[0]. Derivatives of this paper have been used to solve Sudoku[1], Starcraft II[2] and more [3]. This paper enabled more efficient use of data by creating a probabilistic graphical model between logical sets. [0]https://arxiv.org/abs/1706.01427 https://arxiv.org/abs/1706.01427 [1]https://arxiv.org/abs/1711.08028 https://arxiv.org/abs/1711.08028 [2]https://arxiv.org/abs/1806.01830 https://arxiv.org/abs/1806.01830 [3]https://arxiv.org/abs/1806.01822 https://arxiv.org/abs/1806.01822
  - state_less 8y ago
    I like the 3D datasets in these papers since, like a scientist in a lab, you can setup the experiment and explore the domain. Adding time in would be cool too (e.g. Are the blue and red ball going to collide in 10 seconds?) It also helps to be able to show you can answer some of these questions in principle with your model. It gives you hope that it might be able to cover real world images.

gesman 8y ago

THIS. I was just saying to our partner (as well as to my wife!) how lucky we are to work on Healthcare solutions. We have access to data about medications, opioids, patients and physicians behaviors that so very little of others (who can have any clue about data analytics) has. The realization of sitting on a goldmine of impossible-to-access-data + capability of developing cutting edge analytics solutions that could change the world is the best place to be.

tonyhb 8y ago
Google partners with the NHS to get medical data.

karmasimida 8y ago

Model is cheap, data is HARD. Plus investing is data is much more predictable: the outcome is always going to get better, though the margin will be diminishing, but better is better. While modeling is not, hiring 100 machine learning 'experts' will not solve the problem 100 times better than 10 of them. On the other side, 100 labelers are surely going to provide 10 times of throughput provided the same upscale.

aqme28 8y ago
> though the margin will be diminishing So by the metric of model performance (the only thing that matters here), what you're saying is that hiring more labelers is actually not linear.
- karmasimida 8y ago
  True, of course. We can only go as high as 100% for accuracy, right? However the labellers are scalable in terms of their throughput and coverage of the data, you can always find bad examples or holes in your current data plane, and that is the time when labelers, not the scientists, are going to rescue you

makewavesnotwar 8y ago

It's incredible. The last company I worked for before going it alone (I was the front-end engineer and moved away from ML based business models) was trying to automate statistical analysis. I came from an academic background in economics and I tried to propose modelling to them in simple terms and they jumped to, "it sounds like you're talking about forecasting, let's go with that." And they started trying to implement python models from academic papers with highly limited training data and my reaction was generally, WTF?! You can't even start to forecast without a reliable base model. But they went ahead trying to sell stuff like churn prediction to companies with 0 understanding of how these models work at the basest levels. And yeah, Google started to throw their hat in the game with Analytics 360 and an enormously larger training base. Amazon's another major player. Weirdly enough though, people do still blindly pay my previous employers to figure stuff out because easy answers are always actionable, even if they're wrong. It's just crazy because the CEO explained to me that lying about the service to potential customers and investors was necessary because "Faking it til you make it" was a sound business principle in his mind like 1980's Michael J Fox was his primary sources of business info. Long story short, don't waste your time with these little companies purporting ML holy grails. They're probably just lying to you, whether intentionally or not. ML is a game for the big boys with access to market level aggregates. The models that last company came up with were wildly inaccurate.

ryebreadistasty 8y ago
That's obvious to anyone with any sort of basic idea as to how machine learning works. Feed bots data and test the bots predictions - the more data you have then obviously it'll be better. If you have 1 picture of a bee to test you'd only ever get one very specific shape for a bee from any AI, if you have thousands, you get a much better representation. It's pretty simple.
- sgt101 8y ago
  This is mostly true at the minute, but not completely and probably not for the long term (10 years +) See http://science.sciencemag.org/content/350/6266/1332.full http://science.sciencemag.org/content/350/6266/1332.full Also you can have problems where more and more data doesn't make an impact - the stock market being a clear example.

craftinator 8y ago

I don't understand how this is a race at all. There is no way to finish it, and "going fast" is likely to cause major issues in developing sound technologies. Maybe changing the attitude of this article to a more scientifically sound position would increase it's relevance.

mark_l_watson 8y ago

I have just started working with Ocean Protocol (https://oceanprotocol.com/ https://oceanprotocol.com/) and I have set up a local meetup in January for local entrepreneurs looking for access to machine learning data.