9 ms·

More Data Is Not Better and Machine Learning Is a Grind

202 points by sonabinu 8y ago

judge2020 8y ago
See the "Law of large numbers" - https://en.wikipedia.org/wiki/Law_of_large_numbers https://en.wikipedia.org/wiki/Law_of_large_numbers Saying you need more data is like saying you need to flip a quarter 500 million times to get a better percentage estimate of heads vs. tails compares to 1 million coin flips. After a certain point, having more data only helps with identifying outliers and changes in behavior over time (when dealing with human/natural data).
- bunderbunder 8y ago
  I think, more to the point, is that more data really is not better data. Oftentimes it's worse -- the larger the volume of data you collect, the harder (and more expensive) it can be to manage sources of systematic bias in your data collection. One of the more extreme examples of this situation is Big Data, where everything is a convenience sample, and therefore nothing is particularly trustworthy. Combine it with standard statistical hypothesis testing, whose p-values, being designed for the size of experiment that one could realistically manage in a lab setting circa 1910, are as much a reflection of your sample size as of anything else, and you're at great risk of leading yourself into a tar pit.
- mr_toad 8y ago
  Depends what you’re looking for. If you’re trying to measure a very slight bias then more observations can help. Now imagine that you want to measure that bias across different types of coins and year of minting. Does the bias between coins of the same type vary more than across types? Curse of dimensionality starts to raise its ugly head.
- Scene_Cast2 8y ago
  You're using low-dimensional intuition. If your data has a non-linear relationship between 1000 features (or more), then 500 million samples is quantifiably better than 1 million samples. If these are boolean features, you have 2^1000 possible data points. Your samples are extremely sparse. The current state of the art of ML assumes nonlinear relationships between all parameters. It can't assume simpler & reasonable models, and therefore it can't extrapolate easily with reduced data.
  - throwawayjava 8y ago
    > The current state of the art of ML assumes nonlinear relationships between all parameters. It can't assume simpler & reasonable models, and therefore it can't extrapolate easily with reduced data. I'm not really sure what "low-dimensional intuition" means, but I pretty regularly build models that do not "assume nonlinear relationships between all parameters".
    - jacques_chester 8y ago
      My understanding is: It's a fancy way of saying "lots of variables", because a lot of these problems are converted into linear algebra representations (which can have geometric analogies, hence: dimensions).
    - toufiqbarhamov 8y ago
      I think the Wikipedia article on the Curse covers it pretty well. https://en.m.wikipedia.org/wiki/Curse_of_dimensionality https://en.m.wikipedia.org/wiki/Curse_of_dimensionality The common theme of these problems is that when the dimensionality increases, the volume of the space increases so fast that the available data become sparse. This sparsity is problematic for any method that requires statistical significance. In order to obtain a statistically sound and reliable result, the amount of data needed to support the result often grows exponentially with the dimensionality. Also, organizing and searching data often relies on detecting areas where objects form groups with similar properties; in high dimensional data, however, all objects appear to be sparse and dissimilar in many ways, which prevents common data organization strategies from being efficient.
    - tomnipotent 8y ago
      Meaning that you can run PCA and the first few components will explain a vast majority of the variance. Some things may have so many moving pieces that it's just not possible to grok how all the variables interact to get the results we see.
    - gnulinux 8y ago
      As the dimension d goes to infinity, if you sample integers from d-multivariate normal distribution, your samples will collect near the unit d-sphere; whereas in low dimensions they'd collect near the origin. This is because as d goes to infinity, your samples become more and more far apart. This is a topological intuition of why higher dimensions don't work like lower dimensions.
- gnulinux 8y ago
  I think this intuition is fundamentally wrong and misleading. This intuition really only works in low dimensions. It is not necessarily true that more data is better but you cannot blindly invoke the law of large numbers here. As someone here pointed out in high dim data, diversity is more important than numbers.
ausbah 8y ago
Parts of this piece appear to me as evidence for the importance of basic-mid level classes / foundations in statistics, probability, and econometrics for machine learning. Learning to build a model doesn’t mean you can always use it well.
- bunderbunder 8y ago
  Indeed. The first one, for example, reads to me as a dead ringer for a case where someone didn't bother to think about ecological validity until after they started having problems in production.
myWindoonn 8y ago
Title was editorialized from "More Data is Not Better and Machine Learning is a Grind…. Just Ask Amazon" Was it the ellipsis or the reference to the employer of the lecturers? Either way, please don't change titles needlessly. Thanks!
- ggggtez 8y ago
  Disagree. "...Just ask Amazon" is needless clickbaity extra words. Ultimately the opinion is that of the writer.
  - killjoywashere 8y ago
    He was writing a summary of a talk from an Amazon exec. I suppose "per Amazon exec" would have been more exact, but are we just not allowed to have tone anymore? Style is out?
  - Wordball 8y ago
    "More Data is Not Better" is also clickbaity and needless
tedd4u 8y ago
In image recognition, Google asserted in 2017 that they were unable to find decreasing returns -- "Performance increases logarithmically based on volume of training data." Maybe for supply-chain and regression models there is a limit but it seems in deep neural nets maybe there's a different answer. Blog: https://ai.googleblog.com/2017/07/revisiting-unreasonable-effectiveness.html https://ai.googleblog.com/2017/07/revisiting-unreasonable-ef... Paper: https://arxiv.org/abs/1707.02968 https://arxiv.org/abs/1707.02968
- FPGAhacker 8y ago
  Isn’t that the very definition of diminishing returns?
  - mactrey 8y ago
    Perhaps it would be more accurate to say that marginal return is materially positive for all feasible n
    - JoshuaDavid 8y ago
      A logarithmic increase in performance is positive for all n, but not necessarily _materially_ positive for all n. In fact there will be an n where the marginal gain from adding more data will _not_ be worth it, if the cost of adding more data scales linearly but the benefit scale logarithmically.
      - throwawaymath 8y ago
        Right. If the cost of adding more training data remains flat or increases, a logarithmically increasing n will eventually reach a point such that it's more expensive to continue increasing training volume. In fact, unless the cost of adding and using training data exponentially decreases over time, it's a mathematical certainty that a logarithmically increasing n will quickly incur expensive, diminishing returns for using more data. So in the context of this Google paper, you could conceive of a situation where training data actually becomes easier to load (albeit subexponentially) and still becomes too expensive to use relatively quickly.
csours 8y ago
My limited opinion: The Right Data is better than Big Data, if you can get it. There was a push to get data out of app dbs and into big data repositories. But then no one could use the big data because it made no sense. So then ML? But if you already know what it means in the app db, just make it available in a sensible format.
- erikb 8y ago
  I think you can have both situations. I still remember a talk from a Google person maybe around 2010 where he showed clearly that if you try to make the data "better" but in general have way too little data points, then it can't be useful either. It's just now that we had almost a decade of pushing towards more IoT and more Big Data, that many companies have huge data lakes that they don't know how to make use off. So instead of applying one of these lessons it's probably best to see where one is lacking (quantity or quality) and work on resolving that specific problem accordingly.
- jrumbut 8y ago
  One of the most interest HN comments I saw last year claimee many programming trends that burn out make false promises about not needing to worry about database design anymore. The insights you're looking for could be cutting research problems or near trivial depending on the condition of the app db. I have become fond of saying that the opportunities of ML and data science are just more rewards for getting your database schema right.
- nestorD 8y ago
  There is a very good talk on "The Unreasonable Effectiveness of Structure" that detail how just adding this structural information instead of droppping it can give great, low-hanging, rewards : https://www.youtube.com/watch?v=t4k5LKCpboc https://www.youtube.com/watch?v=t4k5LKCpboc
- jillesvangurp 8y ago
  ML definitely works better on good data. If you have bad data, putting some effort into turning it into good data can get you a lot of decent results very quickly. Whenever you hear a ML experts talk about 'feature extraction', they are talking about turning unstructured data into more structured data. With unsupervised learning, you rely on this happening automagically. I've seen some naive approaches struggle to get good results when all that was needed was beating some sense into the data. This typically requires domain knowledge. IMHO most of the engineering effort in ML is not tuning the algorithms but moving data around. A little practical example: consider a dataset containing POIs and a ML based search ranking algorithm that is struggling with correctly ranking airports. Do you 1) spend a year trying to get your algorithm to work better using examples of 'good' results, or 2) figure out a better source for the limited number of airports that exist in the world with better metadata. Turns out a ~day of sorting this out with a few good open data sources gets you a lot further than months of trying out different ways to extract features that aren't there. We had a whole team of ML PhDs. on this and they couldn't get it done. A single engineering intern came up with the obvious analysis: this data is shit, fixing that is easy, lets fix that. Problem solved.
superconformist 8y ago
After I collect a dataset I save duplicate entries with the text reversed. More data is good data. Except now my machine learning robot is dyslexic.
- why_only_15 8y ago
  Of course you're joking here, but for images people do reverse the image to enhance the output of their models and this has shown to be beneficial. There are other augmentation you can do as well. In total they're called "Synthetic Data".
  - superconformist 8y ago
    You see, I'm speaking only the truth, merely referring to synthetic data. And to think that they're downvoting me, poor fools.
  - gnulinux 8y ago
    This is not surprising because if image A is an apple, its mirror image A' is also an apple. Adding A' to your dataset is just plain ol' regularization. It is meant to lower bias, to prevent overfitting.
    - Topolomancer 8y ago
      It's not surprising, but highly necessary because you might otherwise learn too little about structural features, and too much about unnecessary things such as the background colour of some objects.
    - jzwinck 8y ago
      It will also make the flag of Côte d’Ivoire be perceived as that of Ireland.
  - paradoxparalax 8y ago
    The other day I was thinking that if I create a black cube in a simulation with white background, and generate a number of PNG images of the cube in different positions, The adjusted weights I get in my network when it classifies with success in further simulations(applying the learned weights to classify now, instead of learning and adjusting the weights), Can also be used to classify a real life black dice on a real life white table, if the camera is well positioned and the light is right. In this case, the learning would "generalize", from the simulation to the real life. Maybe this simulated data used to adjust the weights could also be called "synthetic data" or "simulated data" or just "artificially generated data".
SemiTom 8y ago
How and where data gets scrubbed will have significant consequences. While clean data is more valuable than dirty data (the mass of raw data collected by sensors), there is hidden value in those large masses of data. They can show broad trends and patterns that are not obvious in clean data, which is the modern equivalent of not seeing the forest for the trees https://semiengineering.com/data-vs-physics/ https://semiengineering.com/data-vs-physics/
- jmatthews 8y ago
  This may be a bit elementary for this crowd but regarding the balance of data cost vs capturing the most significant features. We use a simple decision tree as a significance cluster and optimize data munging around these clusters. On some levels it is anti-diversity but given real world constraints it has yielded the best results. Any thoughts or links regarding this topic would be appreciated.
known 8y ago
Diversity in data is better.
rv-de 8y ago
> The results were consistent with asymptotic theory (central limit theorem) that predicts that more data has diminishing returns. That's not asymptotic theory or central limit theorem. It's a meek proposition that you learn to prove in first semester when studying math. A bounded and monotonous function will always converge. > Higher velocity data does not improve percentage accuracy and makes accuracy levels worse! What on earth is this supposed to mean? An ML algorithm doesn't care about how fast data "arrives". "Higher velocity" is marketing lingo related to their Kinesis Data Streams service. But I assume that he refers to the usual loss of performance observed with all online-learning algos. Feeding knowledge one by one will mostly just overwrite learned generalizations. Or maybe he is indicating that they are compromising algo quality for faster execution. > It is important to pick a single metric to improve, even if it is not perfect, but to use it as the basis for measuring performance improvement. I think that is wrong. Single metrics never capture complex behavior well and will lead to distortions if fed back to the system. I mean it's normal to use one metric for the error. But saying this is the real deal sounds ridiculous to me as this is being done most of the time anyway and probably is going to change in the future. Backprop only works with one metric at the moment - that's just the fact. > Pat noted that improvement and learning is often very slow – sort of like a slow weight loss program, where you lose weight very slowly. Processes may only be improving by 20 basis points a quarter, or 80 basis points a year. That isn’t a lot, but over a decade, it really makes a difference. Now he's contradicting himself as he gives a reason for why big data is beneficial. > His final word of advice – students should be broad in their knowledge of a lot of things, but need to be very deep in one area. And again he is contradicting himself. Because if you make an analogy from student to a learning algorithm he now gives TWO orthogonal metrics to optimize for.
- darawk 8y ago
  > What on earth is this supposed to mean? An ML algorithm doesn't care about how fast data "arrives". My guess is that he's referring to the resolution of a time series. E.g. Going from months to weeks is 'more' data, but makes your models worse.
  - throwawaymath 8y ago
digitalzombie 8y ago
More data is better. You can reduce it via PCA one of the many techniques in multivariate statistic. You can do anova to select your predictors. In general you can use a subset of it using the tools that statistic have provided. Complaining about messy data... welcome to the real world. As for complaining about non-reproducible models , choose a reproducible ones. I've only done mostly statistical models and forest base algorithms and they're all reproducible. All I see in this post is complaints and no real solutions. The solution that's given is what? Have less data? > The results were consistent with asymptotic theory (central limit theorem) that predicts that more data has diminishing returns CLT talks about sampling from the population infinitely. It doesn't say anything about diminishing returns. I don't get how you go from sampling infinitely to diminishing returns.
- iagovar 8y ago
  Thank you, Im not crazy. I was reading HN very confused.
- spongepoc 8y ago
  >CLT talks about sampling from the population infinitely. It doesn't say anything about diminishing returns. I don't get how you go from sampling infinitely to diminishing returns. Yes it does. It even implies it in the name 'limit'. In the limit of infinitely many samples, we approximate a normal distribution. This approximation has diminishing returns. >All I see in this post is complaints and no real solutions. The solution that's given is what? Have less data? It's fine to point out problems without giving solutions. You seem very aggravated.
- apercu 8y ago
  > Complaining about messy data... welcome to the real world. I mean, that's the crux is if you have bad data you will have bad results. Data cleanup/transformation is key for anything (reporting, etc...) and not just limited to ML because it's sexy these days.
- b_tterc_p 8y ago
  PCA has specific use cases. It’s not a catch all dimensionality reduction technique. You can’t use it effectively, for example, if things are not linearly correlated. There of course many tools for addressing many problems, but as the title states, this is often a grind. For any practical problem, exclusive of huge black box neural nets where you don’t need to understand the model, you are probably better off starting with a smaller set of reasonable sounding features and then slowly growing out your model to incorporate others. Also if you meant random forest by forests... those aren’t especially reproducible. Understanding what’s going on is not always easy, and most people seem to misinterpret the idea of “variable importance” when you have a mix of categorical and numeric features. Decision trees and linear regressions are nice and reproducible.
sgt101 8y ago
Statistically they must be right - most of the statistical improvement by an ML system is provided by the first "large" "good" data set, after that gains are sure to be marginal. But, a lot of problems aren't really statistical in nature; they are scientific in that if you can discover a new, better, theory and use that then you can catch a few cases that you may often do the right thing on without knowing why, and sort those out. For example - you may be learning to diagnose a disease with one common driver, there may be a rare disease that responds to the same treatment but has a slightly different presentation, you often treat these patients because of false positives, but if you learn to properly classify them then you can almost always catch them; the statistical gain is negligible, the value is tangible.
- sgt101 8y ago
  I should say that above is an example, I am very cautious about using ML for medical diagnosis.
__coaxialcabal 8y ago
As it turns out, more data is a far second to doing anything at all.
YeGoblynQueenne 8y ago
>> A model is now running in production but not producing the same results (or at the same level of accuracy) as what was demonstrated during experimentation… no one knows why The simplest explanation is that the training sample (meaning the entire dataset; not just the training partition in cross-validation) was not drawn from the same distribution as the distribution of the data that is being processed in production. Machine learning is guaranteed some performance bounds under PAC learning, assuming that the training dataset and unseen data, on which the training model will be used, come from the same distribution. Absent this, performance cannot be predicted. You might as well classify stuff by throwing a bunch of dice. Unfortunately, this assumption, that we're representing the real-world distribution in our training dataset, cannot be justified as long as we don't know the ground truth in the real world. Which is most of the time. Essentially, there's no way to know for sure that a model that has performed very well in experiments will continue to do so once it's deployed in production.
- Scene_Cast2 8y ago
  I agree with the intuition, but not sure that I agree with the takeaways (in particular the lack of knowledge of the ground truth). I mean it is a problem, but there are bigger, more practical mechanisms at work tool. In terms of sample distribution - let's say that you have an online model serving traffic (and the outcomes of that traffic are logged as your training & holdout data). Then, when you train a wildly different model, it may start picking other things to serve that the original model never served - and therefore, there was no training data. This is a pretty fundamental and hard problem. Second, funny thing, but sometimes you can't use the same metric for training as you do for evaluating your model online. I don't want to (can't) get into details, but it's also a pretty fundamental and hard problem. Last, the production traffic always shifts. Using historical data, given how much current models are "reactive" and "compression-like", as opposed to true generalization - they perform worse given fresh traffic that changed slightly from e.g. 5 days ago. If your training data is "day -10 to day -3", your holdout is "day -2 to day 0" - models will likely perform worse on holdout than pure overfit theories would have you assume (mind you, still plenty enough to have a ton of value), but when you launch it on day 1, it will perform worse still - as day 1 is different from days -10 to day -3. I haven't done the analysis, but I'd assume that for non-historical models where you don't need to structure your holdout data to be from the "future" - that they'd perform better when you first launch them online.
paradoxparalax 8y ago
Speak not untrue generalizations is better. Not that the opposite is true, by no means. Generalizations are extremely rarely found to be true.

202 points by sonabinu 8y ago

judge2020 8y ago

See the "Law of large numbers" - https://en.wikipedia.org/wiki/Law_of_large_numbers https://en.wikipedia.org/wiki/Law_of_large_numbers Saying you need more data is like saying you need to flip a quarter 500 million times to get a better percentage estimate of heads vs. tails compares to 1 million coin flips. After a certain point, having more data only helps with identifying outliers and changes in behavior over time (when dealing with human/natural data).

bunderbunder 8y ago
I think, more to the point, is that more data really is not better data. Oftentimes it's worse -- the larger the volume of data you collect, the harder (and more expensive) it can be to manage sources of systematic bias in your data collection. One of the more extreme examples of this situation is Big Data, where everything is a convenience sample, and therefore nothing is particularly trustworthy. Combine it with standard statistical hypothesis testing, whose p-values, being designed for the size of experiment that one could realistically manage in a lab setting circa 1910, are as much a reflection of your sample size as of anything else, and you're at great risk of leading yourself into a tar pit.
mr_toad 8y ago
Depends what you’re looking for. If you’re trying to measure a very slight bias then more observations can help. Now imagine that you want to measure that bias across different types of coins and year of minting. Does the bias between coins of the same type vary more than across types? Curse of dimensionality starts to raise its ugly head.
Scene_Cast2 8y ago
You're using low-dimensional intuition. If your data has a non-linear relationship between 1000 features (or more), then 500 million samples is quantifiably better than 1 million samples. If these are boolean features, you have 2^1000 possible data points. Your samples are extremely sparse. The current state of the art of ML assumes nonlinear relationships between all parameters. It can't assume simpler & reasonable models, and therefore it can't extrapolate easily with reduced data.
- throwawayjava 8y ago
  > The current state of the art of ML assumes nonlinear relationships between all parameters. It can't assume simpler & reasonable models, and therefore it can't extrapolate easily with reduced data. I'm not really sure what "low-dimensional intuition" means, but I pretty regularly build models that do not "assume nonlinear relationships between all parameters".
  - jacques_chester 8y ago
    My understanding is: It's a fancy way of saying "lots of variables", because a lot of these problems are converted into linear algebra representations (which can have geometric analogies, hence: dimensions).
  - toufiqbarhamov 8y ago
    I think the Wikipedia article on the Curse covers it pretty well. https://en.m.wikipedia.org/wiki/Curse_of_dimensionality https://en.m.wikipedia.org/wiki/Curse_of_dimensionality The common theme of these problems is that when the dimensionality increases, the volume of the space increases so fast that the available data become sparse. This sparsity is problematic for any method that requires statistical significance. In order to obtain a statistically sound and reliable result, the amount of data needed to support the result often grows exponentially with the dimensionality. Also, organizing and searching data often relies on detecting areas where objects form groups with similar properties; in high dimensional data, however, all objects appear to be sparse and dissimilar in many ways, which prevents common data organization strategies from being efficient.
  - tomnipotent 8y ago
    Meaning that you can run PCA and the first few components will explain a vast majority of the variance. Some things may have so many moving pieces that it's just not possible to grok how all the variables interact to get the results we see.
  - gnulinux 8y ago
    As the dimension d goes to infinity, if you sample integers from d-multivariate normal distribution, your samples will collect near the unit d-sphere; whereas in low dimensions they'd collect near the origin. This is because as d goes to infinity, your samples become more and more far apart. This is a topological intuition of why higher dimensions don't work like lower dimensions.
gnulinux 8y ago
I think this intuition is fundamentally wrong and misleading. This intuition really only works in low dimensions. It is not necessarily true that more data is better but you cannot blindly invoke the law of large numbers here. As someone here pointed out in high dim data, diversity is more important than numbers.

ausbah 8y ago

Parts of this piece appear to me as evidence for the importance of basic-mid level classes / foundations in statistics, probability, and econometrics for machine learning. Learning to build a model doesn’t mean you can always use it well.

bunderbunder 8y ago
Indeed. The first one, for example, reads to me as a dead ringer for a case where someone didn't bother to think about ecological validity until after they started having problems in production.

myWindoonn 8y ago

Title was editorialized from "More Data is Not Better and Machine Learning is a Grind…. Just Ask Amazon" Was it the ellipsis or the reference to the employer of the lecturers? Either way, please don't change titles needlessly. Thanks!

ggggtez 8y ago
Disagree. "...Just ask Amazon" is needless clickbaity extra words. Ultimately the opinion is that of the writer.
- killjoywashere 8y ago
  He was writing a summary of a talk from an Amazon exec. I suppose "per Amazon exec" would have been more exact, but are we just not allowed to have tone anymore? Style is out?
- Wordball 8y ago
  "More Data is Not Better" is also clickbaity and needless

tedd4u 8y ago

In image recognition, Google asserted in 2017 that they were unable to find decreasing returns -- "Performance increases logarithmically based on volume of training data." Maybe for supply-chain and regression models there is a limit but it seems in deep neural nets maybe there's a different answer. Blog: https://ai.googleblog.com/2017/07/revisiting-unreasonable-effectiveness.html https://ai.googleblog.com/2017/07/revisiting-unreasonable-ef... Paper: https://arxiv.org/abs/1707.02968 https://arxiv.org/abs/1707.02968

FPGAhacker 8y ago
Isn’t that the very definition of diminishing returns?
- mactrey 8y ago
  Perhaps it would be more accurate to say that marginal return is materially positive for all feasible n
  - JoshuaDavid 8y ago
    A logarithmic increase in performance is positive for all n, but not necessarily _materially_ positive for all n. In fact there will be an n where the marginal gain from adding more data will _not_ be worth it, if the cost of adding more data scales linearly but the benefit scale logarithmically.
    - throwawaymath 8y ago
      Right. If the cost of adding more training data remains flat or increases, a logarithmically increasing n will eventually reach a point such that it's more expensive to continue increasing training volume. In fact, unless the cost of adding and using training data exponentially decreases over time, it's a mathematical certainty that a logarithmically increasing n will quickly incur expensive, diminishing returns for using more data. So in the context of this Google paper, you could conceive of a situation where training data actually becomes easier to load (albeit subexponentially) and still becomes too expensive to use relatively quickly.

csours 8y ago

My limited opinion: The Right Data is better than Big Data, if you can get it. There was a push to get data out of app dbs and into big data repositories. But then no one could use the big data because it made no sense. So then ML? But if you already know what it means in the app db, just make it available in a sensible format.

erikb 8y ago
I think you can have both situations. I still remember a talk from a Google person maybe around 2010 where he showed clearly that if you try to make the data "better" but in general have way too little data points, then it can't be useful either. It's just now that we had almost a decade of pushing towards more IoT and more Big Data, that many companies have huge data lakes that they don't know how to make use off. So instead of applying one of these lessons it's probably best to see where one is lacking (quantity or quality) and work on resolving that specific problem accordingly.
jrumbut 8y ago
One of the most interest HN comments I saw last year claimee many programming trends that burn out make false promises about not needing to worry about database design anymore. The insights you're looking for could be cutting research problems or near trivial depending on the condition of the app db. I have become fond of saying that the opportunities of ML and data science are just more rewards for getting your database schema right.
nestorD 8y ago
There is a very good talk on "The Unreasonable Effectiveness of Structure" that detail how just adding this structural information instead of droppping it can give great, low-hanging, rewards : https://www.youtube.com/watch?v=t4k5LKCpboc https://www.youtube.com/watch?v=t4k5LKCpboc
jillesvangurp 8y ago
ML definitely works better on good data. If you have bad data, putting some effort into turning it into good data can get you a lot of decent results very quickly. Whenever you hear a ML experts talk about 'feature extraction', they are talking about turning unstructured data into more structured data. With unsupervised learning, you rely on this happening automagically. I've seen some naive approaches struggle to get good results when all that was needed was beating some sense into the data. This typically requires domain knowledge. IMHO most of the engineering effort in ML is not tuning the algorithms but moving data around. A little practical example: consider a dataset containing POIs and a ML based search ranking algorithm that is struggling with correctly ranking airports. Do you 1) spend a year trying to get your algorithm to work better using examples of 'good' results, or 2) figure out a better source for the limited number of airports that exist in the world with better metadata. Turns out a ~day of sorting this out with a few good open data sources gets you a lot further than months of trying out different ways to extract features that aren't there. We had a whole team of ML PhDs. on this and they couldn't get it done. A single engineering intern came up with the obvious analysis: this data is shit, fixing that is easy, lets fix that. Problem solved.

superconformist 8y ago

After I collect a dataset I save duplicate entries with the text reversed. More data is good data. Except now my machine learning robot is dyslexic.

why_only_15 8y ago
Of course you're joking here, but for images people do reverse the image to enhance the output of their models and this has shown to be beneficial. There are other augmentation you can do as well. In total they're called "Synthetic Data".
- superconformist 8y ago
  You see, I'm speaking only the truth, merely referring to synthetic data. And to think that they're downvoting me, poor fools.
- gnulinux 8y ago
  This is not surprising because if image A is an apple, its mirror image A' is also an apple. Adding A' to your dataset is just plain ol' regularization. It is meant to lower bias, to prevent overfitting.
  - Topolomancer 8y ago
    It's not surprising, but highly necessary because you might otherwise learn too little about structural features, and too much about unnecessary things such as the background colour of some objects.
  - jzwinck 8y ago
    It will also make the flag of Côte d’Ivoire be perceived as that of Ireland.
- paradoxparalax 8y ago
  The other day I was thinking that if I create a black cube in a simulation with white background, and generate a number of PNG images of the cube in different positions, The adjusted weights I get in my network when it classifies with success in further simulations(applying the learned weights to classify now, instead of learning and adjusting the weights), Can also be used to classify a real life black dice on a real life white table, if the camera is well positioned and the light is right. In this case, the learning would "generalize", from the simulation to the real life. Maybe this simulated data used to adjust the weights could also be called "synthetic data" or "simulated data" or just "artificially generated data".

SemiTom 8y ago

How and where data gets scrubbed will have significant consequences. While clean data is more valuable than dirty data (the mass of raw data collected by sensors), there is hidden value in those large masses of data. They can show broad trends and patterns that are not obvious in clean data, which is the modern equivalent of not seeing the forest for the trees https://semiengineering.com/data-vs-physics/ https://semiengineering.com/data-vs-physics/

jmatthews 8y ago
This may be a bit elementary for this crowd but regarding the balance of data cost vs capturing the most significant features. We use a simple decision tree as a significance cluster and optimize data munging around these clusters. On some levels it is anti-diversity but given real world constraints it has yielded the best results. Any thoughts or links regarding this topic would be appreciated.

known 8y ago

Diversity in data is better.

rv-de 8y ago

> The results were consistent with asymptotic theory (central limit theorem) that predicts that more data has diminishing returns. That's not asymptotic theory or central limit theorem. It's a meek proposition that you learn to prove in first semester when studying math. A bounded and monotonous function will always converge. > Higher velocity data does not improve percentage accuracy and makes accuracy levels worse! What on earth is this supposed to mean? An ML algorithm doesn't care about how fast data "arrives". "Higher velocity" is marketing lingo related to their Kinesis Data Streams service. But I assume that he refers to the usual loss of performance observed with all online-learning algos. Feeding knowledge one by one will mostly just overwrite learned generalizations. Or maybe he is indicating that they are compromising algo quality for faster execution. > It is important to pick a single metric to improve, even if it is not perfect, but to use it as the basis for measuring performance improvement. I think that is wrong. Single metrics never capture complex behavior well and will lead to distortions if fed back to the system. I mean it's normal to use one metric for the error. But saying this is the real deal sounds ridiculous to me as this is being done most of the time anyway and probably is going to change in the future. Backprop only works with one metric at the moment - that's just the fact. > Pat noted that improvement and learning is often very slow – sort of like a slow weight loss program, where you lose weight very slowly. Processes may only be improving by 20 basis points a quarter, or 80 basis points a year. That isn’t a lot, but over a decade, it really makes a difference. Now he's contradicting himself as he gives a reason for why big data is beneficial. > His final word of advice – students should be broad in their knowledge of a lot of things, but need to be very deep in one area. And again he is contradicting himself. Because if you make an analogy from student to a learning algorithm he now gives TWO orthogonal metrics to optimize for.

darawk 8y ago
> What on earth is this supposed to mean? An ML algorithm doesn't care about how fast data "arrives". My guess is that he's referring to the resolution of a time series. E.g. Going from months to weeks is 'more' data, but makes your models worse.
- throwawaymath 8y ago

digitalzombie 8y ago

More data is better. You can reduce it via PCA one of the many techniques in multivariate statistic. You can do anova to select your predictors. In general you can use a subset of it using the tools that statistic have provided. Complaining about messy data... welcome to the real world. As for complaining about non-reproducible models , choose a reproducible ones. I've only done mostly statistical models and forest base algorithms and they're all reproducible. All I see in this post is complaints and no real solutions. The solution that's given is what? Have less data? > The results were consistent with asymptotic theory (central limit theorem) that predicts that more data has diminishing returns CLT talks about sampling from the population infinitely. It doesn't say anything about diminishing returns. I don't get how you go from sampling infinitely to diminishing returns.

iagovar 8y ago
Thank you, Im not crazy. I was reading HN very confused.
spongepoc 8y ago
>CLT talks about sampling from the population infinitely. It doesn't say anything about diminishing returns. I don't get how you go from sampling infinitely to diminishing returns. Yes it does. It even implies it in the name 'limit'. In the limit of infinitely many samples, we approximate a normal distribution. This approximation has diminishing returns. >All I see in this post is complaints and no real solutions. The solution that's given is what? Have less data? It's fine to point out problems without giving solutions. You seem very aggravated.
apercu 8y ago
> Complaining about messy data... welcome to the real world. I mean, that's the crux is if you have bad data you will have bad results. Data cleanup/transformation is key for anything (reporting, etc...) and not just limited to ML because it's sexy these days.
b_tterc_p 8y ago
PCA has specific use cases. It’s not a catch all dimensionality reduction technique. You can’t use it effectively, for example, if things are not linearly correlated. There of course many tools for addressing many problems, but as the title states, this is often a grind. For any practical problem, exclusive of huge black box neural nets where you don’t need to understand the model, you are probably better off starting with a smaller set of reasonable sounding features and then slowly growing out your model to incorporate others. Also if you meant random forest by forests... those aren’t especially reproducible. Understanding what’s going on is not always easy, and most people seem to misinterpret the idea of “variable importance” when you have a mix of categorical and numeric features. Decision trees and linear regressions are nice and reproducible.

sgt101 8y ago

Statistically they must be right - most of the statistical improvement by an ML system is provided by the first "large" "good" data set, after that gains are sure to be marginal. But, a lot of problems aren't really statistical in nature; they are scientific in that if you can discover a new, better, theory and use that then you can catch a few cases that you may often do the right thing on without knowing why, and sort those out. For example - you may be learning to diagnose a disease with one common driver, there may be a rare disease that responds to the same treatment but has a slightly different presentation, you often treat these patients because of false positives, but if you learn to properly classify them then you can almost always catch them; the statistical gain is negligible, the value is tangible.

sgt101 8y ago
I should say that above is an example, I am very cautious about using ML for medical diagnosis.

__coaxialcabal 8y ago

As it turns out, more data is a far second to doing anything at all.

YeGoblynQueenne 8y ago

>> A model is now running in production but not producing the same results (or at the same level of accuracy) as what was demonstrated during experimentation… no one knows why The simplest explanation is that the training sample (meaning the entire dataset; not just the training partition in cross-validation) was not drawn from the same distribution as the distribution of the data that is being processed in production. Machine learning is guaranteed some performance bounds under PAC learning, assuming that the training dataset and unseen data, on which the training model will be used, come from the same distribution. Absent this, performance cannot be predicted. You might as well classify stuff by throwing a bunch of dice. Unfortunately, this assumption, that we're representing the real-world distribution in our training dataset, cannot be justified as long as we don't know the ground truth in the real world. Which is most of the time. Essentially, there's no way to know for sure that a model that has performed very well in experiments will continue to do so once it's deployed in production.

Scene_Cast2 8y ago
I agree with the intuition, but not sure that I agree with the takeaways (in particular the lack of knowledge of the ground truth). I mean it is a problem, but there are bigger, more practical mechanisms at work tool. In terms of sample distribution - let's say that you have an online model serving traffic (and the outcomes of that traffic are logged as your training & holdout data). Then, when you train a wildly different model, it may start picking other things to serve that the original model never served - and therefore, there was no training data. This is a pretty fundamental and hard problem. Second, funny thing, but sometimes you can't use the same metric for training as you do for evaluating your model online. I don't want to (can't) get into details, but it's also a pretty fundamental and hard problem. Last, the production traffic always shifts. Using historical data, given how much current models are "reactive" and "compression-like", as opposed to true generalization - they perform worse given fresh traffic that changed slightly from e.g. 5 days ago. If your training data is "day -10 to day -3", your holdout is "day -2 to day 0" - models will likely perform worse on holdout than pure overfit theories would have you assume (mind you, still plenty enough to have a ton of value), but when you launch it on day 1, it will perform worse still - as day 1 is different from days -10 to day -3. I haven't done the analysis, but I'd assume that for non-historical models where you don't need to structure your holdout data to be from the "future" - that they'd perform better when you first launch them online.

paradoxparalax 8y ago

Speak not untrue generalizations is better. Not that the opposite is true, by no means. Generalizations are extremely rarely found to be true.