6 ms·

> “More data” is not inherently good. Without a framework of information for interpretation of that data, it will do far more harm than good. I'm glad people a

> “More data” is not inherently good. Without a framework of information for interpretation of that data, it will do far more harm than good. I'm glad people are saying this. I'm in a constant fight at work with people who want "more data." What ends up happening most of the time is it's ignored or people do not interpret it correctly because they don't have the ability or context to do it (doing stats wrong for instance).

davemel37 9y ago
Have them read Chapter 5 of this book. https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/books-and-monographs/psychology-of-intelligence-analysis/PsychofIntelNew.pdf https://www.cia.gov/library/center-for-the-study-of-intellig... "Using experts in a variety of fields as test subjects, experimental psychologists have examined the relationship between the amount of information available to the experts, the accuracy of judgments they make based on this information, and the experts’ confidence in the accuracy of these judgments." "Key findings from this research are: • Once an experienced analyst has the minimum information necessary to make an informed judgment, obtaining additional information generally does not improve the accuracy of his or her estimates. Additional information does, however, lead the analyst to become more confident in the judgment, to the point of overconfidence. • Experienced analysts have an imperfect understanding of what information they actually use in making judgments. They are unaware of the extent to which their judgments are determined by a few dominant factors, rather than by the systematic integration of all available information. Analysts actually use much less of the available information than they think they do."
- 0xCMP 9y ago
  But, the main issue is "once an experienced analyst has the minimum information necessary to make an informed judgement" is followed by "they are unaware of the extent to which their judgements are determined by a few dominant factors." Which implies they do not know which part of the data is most important. No? disclaimer: didn't click link/read chapter
  - davemel37 9y ago
    The authors ultimate solution is to use the scientific method to compare competing hypothesis...otherwise you'll be blinded by all sorts of biases. If you are running an experiment and using competing hypothesis...in theory, you can identify which metrics are best for that test... but if you had much more data, you'll probably end up using every data point past the first 5 as confirmation bias. (I'm no expert though and haven't read the book in about 3 years.)
- mynegation 9y ago
  Depends on the way of consuming that data. I agree it may be counter-productive to feed more data to humans, who are able to hold on only to 7-9 object in immediate memory at a time, and driven by subconscious reflexes hardwired into us by billions of evolution. AI is not going to "think" like humans. It is not going to be better or worse, just "different". And all experience so far show that the more data the better are the decisions made by AI.
  - arkades 9y ago
    Cart, horse. People are proposing smart whatever now, and handeaving its value proposition as -eventually- feeding into ML which will -eventually- have benefits to patients. I don’t think there is disagreement that data + ML is useful, but the value for increasing data because of AI assistance isn’t meaningful in the absence of that AI. Hospitals and insurers have huge data sets available: build a halfway decent AI first, and then argue for the value-add of the data.
    - vasilipupkin 9y ago
      Those data sets are not public, hipaa and other issues make it difficult to build any sort of AI for this.
      - arkades 9y ago
        That is technically correct, the best kind of correct! Except any department in any hospital or insurer that welcomes you to do research can make that de-identified data available to you with a simple contract. How do you think college freshmen get in the door to work on research with physicians? No, it’s not public, but it’s trivially accessible.
        vasilipupkin 9y ago
        It's not trivially accessible. It's not impossible, but far from trivial. Startups like Tempus are doing it but I doubt it's trivial for just any startup to get that data
chiefalchemist 9y ago
Yup. More data is not the same as better information. Put another way, at the extreme, you can have more and more data but unless you have the right data and are able to interpret it correctly, more data will only blind and confuse.

davemel37 9y ago

Have them read Chapter 5 of this book. https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/books-and-monographs/psychology-of-intelligence-analysis/PsychofIntelNew.pdf https://www.cia.gov/library/center-for-the-study-of-intellig... "Using experts in a variety of fields as test subjects, experimental psychologists have examined the relationship between the amount of information available to the experts, the accuracy of judgments they make based on this information, and the experts’ confidence in the accuracy of these judgments." "Key findings from this research are: • Once an experienced analyst has the minimum information necessary to make an informed judgment, obtaining additional information generally does not improve the accuracy of his or her estimates. Additional information does, however, lead the analyst to become more confident in the judgment, to the point of overconfidence. • Experienced analysts have an imperfect understanding of what information they actually use in making judgments. They are unaware of the extent to which their judgments are determined by a few dominant factors, rather than by the systematic integration of all available information. Analysts actually use much less of the available information than they think they do."

0xCMP 9y ago
But, the main issue is "once an experienced analyst has the minimum information necessary to make an informed judgement" is followed by "they are unaware of the extent to which their judgements are determined by a few dominant factors." Which implies they do not know which part of the data is most important. No? disclaimer: didn't click link/read chapter
- davemel37 9y ago
  The authors ultimate solution is to use the scientific method to compare competing hypothesis...otherwise you'll be blinded by all sorts of biases. If you are running an experiment and using competing hypothesis...in theory, you can identify which metrics are best for that test... but if you had much more data, you'll probably end up using every data point past the first 5 as confirmation bias. (I'm no expert though and haven't read the book in about 3 years.)
mynegation 9y ago
Depends on the way of consuming that data. I agree it may be counter-productive to feed more data to humans, who are able to hold on only to 7-9 object in immediate memory at a time, and driven by subconscious reflexes hardwired into us by billions of evolution. AI is not going to "think" like humans. It is not going to be better or worse, just "different". And all experience so far show that the more data the better are the decisions made by AI.
- arkades 9y ago
  Cart, horse. People are proposing smart whatever now, and handeaving its value proposition as -eventually- feeding into ML which will -eventually- have benefits to patients. I don’t think there is disagreement that data + ML is useful, but the value for increasing data because of AI assistance isn’t meaningful in the absence of that AI. Hospitals and insurers have huge data sets available: build a halfway decent AI first, and then argue for the value-add of the data.
  - vasilipupkin 9y ago
    Those data sets are not public, hipaa and other issues make it difficult to build any sort of AI for this.
    - arkades 9y ago
      That is technically correct, the best kind of correct! Except any department in any hospital or insurer that welcomes you to do research can make that de-identified data available to you with a simple contract. How do you think college freshmen get in the door to work on research with physicians? No, it’s not public, but it’s trivially accessible.
      - vasilipupkin 9y ago
        It's not trivially accessible. It's not impossible, but far from trivial. Startups like Tempus are doing it but I doubt it's trivial for just any startup to get that data

chiefalchemist 9y ago

Yup. More data is not the same as better information. Put another way, at the extreme, you can have more and more data but unless you have the right data and are able to interpret it correctly, more data will only blind and confuse.