5 ms·

> However, the researchers found that several of the most predictive factors (such as accidental injury, a benign breast lump, or colonoscopy) don’t make sense

> However, the researchers found that several of the most predictive factors (such as accidental injury, a benign breast lump, or colonoscopy) don’t make sense as risk factors for stroke. So, just what is going on? It turned out that the model was just identifying people who utilize health care a lot. A good analogous example of this is PCA. If your first component has a dominating effect, then this will drown everything out (you can compensate by looking at components 2 and 3). (Examples: [monetary] inflation, year or month effects.) It's a cool exercise to do PCA on datasets and to see whether things like this pop out. This is also why PCA (which does maximisation) is an explorative analysis and should not be used as an authoritative "result" or "metric". You could, but you have to be careful. Even in picking your components you introduce curation and bias. Imagine a scenario where we regress back to the dark ages and the eugenics inclined doctor says: "Sorry Stan, your second component value is an outlier, no kids for you; your genes are not considered adequate."

vlz 7y ago
PCA probably means Principal Component Analysis, if anybody wonders. https://en.m.wikipedia.org/wiki/Principal_component_analysis https://en.m.wikipedia.org/wiki/Principal_component_analysis
- EdwardDiego 7y ago
  Thanks :)
msds 7y ago
Just to piggyback off this, one should be very careful combining techniques with PCA. One error I see a lot in biostatistics is people running statistical significance testing to select variables before running PCA. It's a super cool exercise to do that on datasets, because something WILL show up, and it's very often misleading.
thinkingkong 7y ago
In lots of organizations sets of metrics aren't "held in tension" and that causes all sorts of wild problems. As a trivial example say you want to stop people from quitting your app. Easy! Just lower the price by 50%. So we often have to set metrics while satisfying all the other constraints. This is the #1 thing I've seen go wrong when teams try to use OKRs or some other metrics based activity; they fail to figure out the related components and either hold those in place or improve them along side their initial goals.
Spooky23 7y ago
This is a great illustration of the limits of AI — the scope of available data is limited. The data available in medical records is both limited and often garbage. Without the context of behavioral data and other factors, it’s dangerous to draw conclusions. In other scenarios, like say predicting the likelihood that a mole is cancerous, AI will eventually be better and more consistent than a human.
- mlthoughts2018 7y ago
  What decision making process if not a statistical one would be better when it faces the same data limitations? Or what if we know the best decision making process for a given context, but it’s too expensive to apply? For example, we may know that evaluating loan applicants is best done with in-person interviews, but it’s simply too costly and delays getting loans to people too long, so we agree to use an automated decision process knowing the cost of it making some mistakes will be part of what is paid for the benefit of being able to process many applications quickly.
  - Spooky23 7y ago
    If a predictive process discards risk for humans who lack access to healthcare, it’s creating an obvious ethical problem. There’s a big difference between writing a loan and missing a stroke.
    - mlthoughts2018 7y ago
      That seems more like an argument to make more data available. Presumably more people at risk of a stroke can be notified by an algorithmic assessment than could be possible trying to fit them all through actual physician screenings, no? In a public health sense, it would do more good in terms of lives saved, even if algorithmic errors cost lives in some specific cases.
      - Spooky23 7y ago
        I’m familiar with programs that at scale that have been demonstrated to predict certain negative health/behavior patterns. From a medical point of view, you can assume a 10-20% probability of a negative outcome. If you layer on other factors or behaviors in the persons life, the probability of a bad outcome approaches 90%. Problem is, you need deeply intrusive data at scale. What are the consequences of a organization with Google/Facebook capability who also is your AI doctor? It’s easy to foresee complications.

vlz 7y ago

PCA probably means Principal Component Analysis, if anybody wonders. https://en.m.wikipedia.org/wiki/Principal_component_analysis https://en.m.wikipedia.org/wiki/Principal_component_analysis

EdwardDiego 7y ago
Thanks :)

msds 7y ago

Just to piggyback off this, one should be very careful combining techniques with PCA. One error I see a lot in biostatistics is people running statistical significance testing to select variables before running PCA. It's a super cool exercise to do that on datasets, because something WILL show up, and it's very often misleading.

thinkingkong 7y ago

In lots of organizations sets of metrics aren't "held in tension" and that causes all sorts of wild problems. As a trivial example say you want to stop people from quitting your app. Easy! Just lower the price by 50%. So we often have to set metrics while satisfying all the other constraints. This is the #1 thing I've seen go wrong when teams try to use OKRs or some other metrics based activity; they fail to figure out the related components and either hold those in place or improve them along side their initial goals.

Spooky23 7y ago

This is a great illustration of the limits of AI — the scope of available data is limited. The data available in medical records is both limited and often garbage. Without the context of behavioral data and other factors, it’s dangerous to draw conclusions. In other scenarios, like say predicting the likelihood that a mole is cancerous, AI will eventually be better and more consistent than a human.

mlthoughts2018 7y ago
What decision making process if not a statistical one would be better when it faces the same data limitations? Or what if we know the best decision making process for a given context, but it’s too expensive to apply? For example, we may know that evaluating loan applicants is best done with in-person interviews, but it’s simply too costly and delays getting loans to people too long, so we agree to use an automated decision process knowing the cost of it making some mistakes will be part of what is paid for the benefit of being able to process many applications quickly.
- Spooky23 7y ago
  If a predictive process discards risk for humans who lack access to healthcare, it’s creating an obvious ethical problem. There’s a big difference between writing a loan and missing a stroke.
  - mlthoughts2018 7y ago
    That seems more like an argument to make more data available. Presumably more people at risk of a stroke can be notified by an algorithmic assessment than could be possible trying to fit them all through actual physician screenings, no? In a public health sense, it would do more good in terms of lives saved, even if algorithmic errors cost lives in some specific cases.
    - Spooky23 7y ago
      I’m familiar with programs that at scale that have been demonstrated to predict certain negative health/behavior patterns. From a medical point of view, you can assume a 10-20% probability of a negative outcome. If you layer on other factors or behaviors in the persons life, the probability of a bad outcome approaches 90%. Problem is, you need deeply intrusive data at scale. What are the consequences of a organization with Google/Facebook capability who also is your AI doctor? It’s easy to foresee complications.