3 ms·

As someone with a bioinformatics degree and who worked in industry doing liquid biopsy for cancer diagnostics for half a decade, much of what they claimed they

As someone with a bioinformatics degree and who worked in industry doing liquid biopsy for cancer diagnostics for half a decade, much of what they claimed they could do is possible- just, not with the technology of Theranos' day, and not with the volumes of blood they claimed to be able to use. Example: for a decade now it's been possible to take a standard blood draw from a stage 4 cancer patient, sequence all DNA circulating in the plasma, distinguish the cancer DNA from non-cancer, and use the cancer sequences to inform treatment. In 2022, this is readily available to most stage 4 American cancer patients. The obvious logical future extension of this, is draw blood from any person, and see if they have any stage of cancer. This is something that is being actively worked on, and is currently in the experimental/cutting edge state; expect to see it become somewhat common/affordable in perhaps another 4 years. Theranos claimed to be able to do the above, a decade ago, and with an order of magnitude less biological input material than current technology requires. One reason they garnered so much attention, is that people generally familiar with the field knew that much of what they claimed, were things that were possible but were 15-30 years out based on the trajectory of technology at the time. So it was plausible that a genuine breakthrough had occurred that massively accelerated that timeline. Sadly, as we all now know, that was not the case.

kstrauser 4y ago
Thanks for that extra context!
pclmulqdq 4y ago
The mindset around tests that you just suggested, and clearly Holmes had, is based on a serious misconception about statistics and testing. Every test has false positives and false negatives. A lot of tests are very bad in this regard, with things like 25% false negative rates on one test and 50% false positive rates on another. When you administer those tests to someone who is a likely candidate for a disease, and combine them with a few other tests, you get useful information. If you just scattershot the tests to everyone, you are going to get a lot of mis-diagnoses. COVID has de-sensitized us to the idea of over-testing for the hell of it (in the case of COVID, this is because society is okay with a few false positives being told to quarantine and the test had the sensitivity turned way up so that false negatives were nearly impossible), but it is still a really good way to get really bad information. It is the real-life version of "p-hacking" that occurs in some academic settings. For a test to be useful, you need a hypothesis before you administer a test.
- s1artibartfast 4y ago
  That is a problem readily solved by serial testing. False positive rate is reduced exponentially by the number of tests. >For a test to be useful, you need a hypothesis before you administer a test. Hypothesis: some people get cancer
  - deleted 4y ago
    [deleted]
  - pclmulqdq 4y ago
    Serial testing only solves the problem if the tests are independent. With complicated medical tests, they often are not independent - for example, you can have a genetic factor that makes you reliably test positive for certain cancers. Note that COVID tests were generally independent (although not completely). If you want to flood hospitals with false positives that sounds like a good hypothesis. Then the people who actually do have cancer will not be taken seriously. Here's an example: Suppose you test for a cancer that is very common, and 1 in 10000 people have it. Also, suppose you have a very accurate test with a 5% false positive rate and a 5% false negative rate, and those false positives are sticky. If you 10000 people the test every week, 501 of them will reliably report positive. Every one of them will then report to the hospital with the cancer. Now the hospital has to deal with finding the one true positive before injecting people with literal poison to get rid of that cancer (chemotherapy). Since the hypothesis is "some people get cancer," you will find all of them! And you will find a shit load more of them who do not. Compare that to today, when your doctor says "you have a small lump in your body, let's test you for cancer" - that 5% false positive rate will mean very few false positives, so a positive cancer test means that you are ready for invasive treatment. Data is not information. You need information to make decisions. "Serial testing" only gets you data. That's why we only do it with "informational" tests like the Chem 20 and the CBC.
  - thaumasiotes 4y ago
    > That is a problem readily solved by serial testing. False positive rate is reduced exponentially by the number of tests. This depends. There are two ways a test might deliver the wrong result: (A) It should have delivered the right result, but somewhere along the line something happened the wrong way. (B) The test correctly assessed the instrumental variable, but -- in this subject -- the instrumental variable did not reflect the variable of interest in the manner that it usually does. That is, many medical tests are not actually testing for the outcome we care about. They're testing for something that is usually related to the outcome we care about, because we don't know how to test for the real thing, or we do know how but the reliable test is far more invasive, or some such. There is a very accessible example of this kind of thing right now - you can take a sample from someone and test for the presence of covid. Or you can test for the presence of covid antibodies. Those variables are related, but different, and we care about the presence of the virus a lot more than we care about the presence of antibodies. If the test is failing in way (B), retests will not reduce the false positive rate, because it's actually a true positive that is being misinterpreted. Retesting will only solve failures of type (A). For type (B), you'd have to apply a different kind of test, and that might not be worth it if the first test is unreliable enough. Imagine an unreliable test where the reliable followup requires a bone marrow sample.

kstrauser 4y ago

Thanks for that extra context!

pclmulqdq 4y ago

The mindset around tests that you just suggested, and clearly Holmes had, is based on a serious misconception about statistics and testing. Every test has false positives and false negatives. A lot of tests are very bad in this regard, with things like 25% false negative rates on one test and 50% false positive rates on another. When you administer those tests to someone who is a likely candidate for a disease, and combine them with a few other tests, you get useful information. If you just scattershot the tests to everyone, you are going to get a lot of mis-diagnoses. COVID has de-sensitized us to the idea of over-testing for the hell of it (in the case of COVID, this is because society is okay with a few false positives being told to quarantine and the test had the sensitivity turned way up so that false negatives were nearly impossible), but it is still a really good way to get really bad information. It is the real-life version of "p-hacking" that occurs in some academic settings. For a test to be useful, you need a hypothesis before you administer a test.

s1artibartfast 4y ago
That is a problem readily solved by serial testing. False positive rate is reduced exponentially by the number of tests. >For a test to be useful, you need a hypothesis before you administer a test. Hypothesis: some people get cancer
- deleted 4y ago
  [deleted]
- pclmulqdq 4y ago
  Serial testing only solves the problem if the tests are independent. With complicated medical tests, they often are not independent - for example, you can have a genetic factor that makes you reliably test positive for certain cancers. Note that COVID tests were generally independent (although not completely). If you want to flood hospitals with false positives that sounds like a good hypothesis. Then the people who actually do have cancer will not be taken seriously. Here's an example: Suppose you test for a cancer that is very common, and 1 in 10000 people have it. Also, suppose you have a very accurate test with a 5% false positive rate and a 5% false negative rate, and those false positives are sticky. If you 10000 people the test every week, 501 of them will reliably report positive. Every one of them will then report to the hospital with the cancer. Now the hospital has to deal with finding the one true positive before injecting people with literal poison to get rid of that cancer (chemotherapy). Since the hypothesis is "some people get cancer," you will find all of them! And you will find a shit load more of them who do not. Compare that to today, when your doctor says "you have a small lump in your body, let's test you for cancer" - that 5% false positive rate will mean very few false positives, so a positive cancer test means that you are ready for invasive treatment. Data is not information. You need information to make decisions. "Serial testing" only gets you data. That's why we only do it with "informational" tests like the Chem 20 and the CBC.
- thaumasiotes 4y ago
  > That is a problem readily solved by serial testing. False positive rate is reduced exponentially by the number of tests. This depends. There are two ways a test might deliver the wrong result: (A) It should have delivered the right result, but somewhere along the line something happened the wrong way. (B) The test correctly assessed the instrumental variable, but -- in this subject -- the instrumental variable did not reflect the variable of interest in the manner that it usually does. That is, many medical tests are not actually testing for the outcome we care about. They're testing for something that is usually related to the outcome we care about, because we don't know how to test for the real thing, or we do know how but the reliable test is far more invasive, or some such. There is a very accessible example of this kind of thing right now - you can take a sample from someone and test for the presence of covid. Or you can test for the presence of covid antibodies. Those variables are related, but different, and we care about the presence of the virus a lot more than we care about the presence of antibodies. If the test is failing in way (B), retests will not reduce the false positive rate, because it's actually a true positive that is being misinterpreted. Retesting will only solve failures of type (A). For type (B), you'd have to apply a different kind of test, and that might not be worth it if the first test is unreliable enough. Imagine an unreliable test where the reliable followup requires a bone marrow sample.