4 ms·

Flipping 12 coins and having them all land on heads is a 0.02% probability event. Cancer spontaneously going into remission in a patient is pretty far away from

Flipping 12 coins and having them all land on heads is a 0.02% probability event. Cancer spontaneously going into remission in a patient is pretty far away from a coinflip.

dekhn 3y ago
That's true but this is a clinical trial where the patients are being treated- the patients were selected according to strict criteria, and received other treatments in cancer centers. It's not really just "coinflips", it's heavily biased coinflips. Maybe I should have put this in a more productive way: I prefer studies that have a large enough n that you can actually measure the failure rate. There's a classic statistics paper where an operation was done on 3 patients and none died- the paper's title is "If nobody dies, is everything OK?" and goes into whether you can reliably conclude causal effects with very small n. Further- let's say the drug caused remission in 50% of the patients, and the other 50% had spontaneous remission (or remissionc aused by other aspects of the treatment). Now it's not 1/2 * 12, it's 1/2 * 6 (or whatever base probability you want to choose). If you compare that probability to the number of people in clinical trials in the US, you would expect that after 30-40 years, you'd see at least one trial with 12 patients with complete remission. (I don't actually doubt the drug is effective. We already know that. And we already know that studies this small really stretch the concepts of significance and causality).
- superb_dev 3y ago
  I think most people would prefer a study with more participants, but you can’t jump to a large human trial right away
  - dekhn 3y ago
    Patient recruitment is hard, but this is a phase II on an already approved drug. Typical Phase II has at least 100 patients.
- vikramkr 3y ago
  Edit - also, you should go and look up the rates of spontaneous remission for colorectal cancer (e.g. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10273780/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10273780/). We're talking about 12 patients with stage 2/3 colorectal cancer, in remission with the treatment alone. What's really cool is that papers tend to have sections where they discuss the statistics so you can use real numbers. Their null hypothesis was a 25% response rate. "This decision rule would result in a type I error rate of 6% if 25% of the patients had an overall response and would provide the study with 84% power if 50% had an overall response. The null hypothesis was established on the basis of a study by Seligmann et al., in which the observed response to chemotherapy among patients with mismatch repair–deficient rectal cancers was 7% (8 of 115 patients)."
readthenotes1 3y ago
As other people have mentioned, this is just a variation on a theme of other successful drugs. It may well be that the odds are greater than 50% the way they selected the patients. There's also publication and hacker News front page bias to consider in the odds making as well...
- dekhn 3y ago
  I end up seeing a lot of these discussions and you can always tell the difference between a person who doesn't know much math but works in the field, to somebody who knows a lot of math but doesn't work in the field. Experience and wisdom matter a lot wrt drug discovery and cancer treatment.
  - vikramkr 3y ago
    What's funny is that the paper has the statistics. The original point is sort of valid in that narrowly speaking, there's probably been other cancer trials with similar response rates. Sticking with that specific argument then yeah that claim is probably exaggerated for clicks. But that's a claim about this study compared to other studies. Doesn't have anything to do with the actual effectiveness of the drug or the statistics of the study! I don't think it's just experience and wisdom - it's also the math lol this is plenty enough to get a strong insight into causality. Colorectal cancer doesn't exactly tend to go into spontaneous remission that often...
    - dekhn 3y ago
      When a press release says "all 12 patients went into full remission", you can be sure that the PR authors truly want you to believe that the drug had the effect of curing all the patients.
      - vikramkr 3y ago
        The press release is obviously overselling (also remission != cure), but that's separate from the statistics of the study and it's design and the significance of the results. It's almost always a safe bet to just ignore press releases entirely and jump straight to the paper.
bawolff 3y ago
> Flipping 12 coins and having them all land on heads is a 0.02% probability event. Cancer spontaneously going into remission in a patient is pretty far away from a coinflip. You don't actually have to run that many trials before a 0.02% liklihood event happens by chance.
- pie420 3y ago
  only 5000 or so trials, easy
  - Retric 3y ago
    5,000 isn’t a guarantee, but your already at 10% odds of at least one hit with 527 trials. Considering how many of these get conducted and the odds are often better than 50/50, I expect multiple trials have cured everyone.
  - bawolff 3y ago
    In the history of the world, i am pretty sure there have been more than 5000 cancer trials.
    - inglor_cz 3y ago
      This entire chain of comments rests on the presumption of a "coin toss". If colorectal cancers in 2nd or 3rd stage actually had a fifty-fifty chance of spontaneous remission in the real world, the disease would be a lot less scary than it is. They don't. Spontaneous remission does happen, but isn't common. The chance that a 12-patient trial produces spontaneous remission in all 12 patients in a truly random fashion, is much smaller than 1:5000. And the GP actually mentioned it. But everyone reacting to him still addresses the 1:5000 value. It is a nice example of anchoring as described by Daniel Kahneman - here is a number, let's cling to it.
      - dekhn 3y ago
        You totally misunderstood my comment. I never meant that all 12 patients had spontaneous remission. I understand why people think that- my statement was ambiguous. What I did mean was that it's possible that some of the patients cancers went into remission for a reason that was not specific to this drug. It could be 11 people got better because of the drug- and 1 got better because they were being treated in a world-class hospital with a number of other factors that affect disease progression. THat's how it works in clinical trials. I didn't make my statement out of ignorance- I worked adjacent to trials for decades and have seen all sorts of claims of results that didn't hold up to scrutiny.

Flipping 12 coins and having them all land on heads is a 0.02% probability event. Cancer spontaneously going into remission in a patient is pretty far away from a coinflip.

dekhn 3y ago

That's true but this is a clinical trial where the patients are being treated- the patients were selected according to strict criteria, and received other treatments in cancer centers. It's not really just "coinflips", it's heavily biased coinflips. Maybe I should have put this in a more productive way: I prefer studies that have a large enough n that you can actually measure the failure rate. There's a classic statistics paper where an operation was done on 3 patients and none died- the paper's title is "If nobody dies, is everything OK?" and goes into whether you can reliably conclude causal effects with very small n. Further- let's say the drug caused remission in 50% of the patients, and the other 50% had spontaneous remission (or remissionc aused by other aspects of the treatment). Now it's not 1/2 * 12, it's 1/2 * 6 (or whatever base probability you want to choose). If you compare that probability to the number of people in clinical trials in the US, you would expect that after 30-40 years, you'd see at least one trial with 12 patients with complete remission. (I don't actually doubt the drug is effective. We already know that. And we already know that studies this small really stretch the concepts of significance and causality).

superb_dev 3y ago
I think most people would prefer a study with more participants, but you can’t jump to a large human trial right away
- dekhn 3y ago
  Patient recruitment is hard, but this is a phase II on an already approved drug. Typical Phase II has at least 100 patients.
vikramkr 3y ago
Edit - also, you should go and look up the rates of spontaneous remission for colorectal cancer (e.g. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10273780/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10273780/). We're talking about 12 patients with stage 2/3 colorectal cancer, in remission with the treatment alone. What's really cool is that papers tend to have sections where they discuss the statistics so you can use real numbers. Their null hypothesis was a 25% response rate. "This decision rule would result in a type I error rate of 6% if 25% of the patients had an overall response and would provide the study with 84% power if 50% had an overall response. The null hypothesis was established on the basis of a study by Seligmann et al., in which the observed response to chemotherapy among patients with mismatch repair–deficient rectal cancers was 7% (8 of 115 patients)."

readthenotes1 3y ago

As other people have mentioned, this is just a variation on a theme of other successful drugs. It may well be that the odds are greater than 50% the way they selected the patients. There's also publication and hacker News front page bias to consider in the odds making as well...

dekhn 3y ago
I end up seeing a lot of these discussions and you can always tell the difference between a person who doesn't know much math but works in the field, to somebody who knows a lot of math but doesn't work in the field. Experience and wisdom matter a lot wrt drug discovery and cancer treatment.
- vikramkr 3y ago
  What's funny is that the paper has the statistics. The original point is sort of valid in that narrowly speaking, there's probably been other cancer trials with similar response rates. Sticking with that specific argument then yeah that claim is probably exaggerated for clicks. But that's a claim about this study compared to other studies. Doesn't have anything to do with the actual effectiveness of the drug or the statistics of the study! I don't think it's just experience and wisdom - it's also the math lol this is plenty enough to get a strong insight into causality. Colorectal cancer doesn't exactly tend to go into spontaneous remission that often...
  - dekhn 3y ago
    When a press release says "all 12 patients went into full remission", you can be sure that the PR authors truly want you to believe that the drug had the effect of curing all the patients.
    - vikramkr 3y ago
      The press release is obviously overselling (also remission != cure), but that's separate from the statistics of the study and it's design and the significance of the results. It's almost always a safe bet to just ignore press releases entirely and jump straight to the paper.

bawolff 3y ago

> Flipping 12 coins and having them all land on heads is a 0.02% probability event. Cancer spontaneously going into remission in a patient is pretty far away from a coinflip. You don't actually have to run that many trials before a 0.02% liklihood event happens by chance.

pie420 3y ago
only 5000 or so trials, easy
- Retric 3y ago
  5,000 isn’t a guarantee, but your already at 10% odds of at least one hit with 527 trials. Considering how many of these get conducted and the odds are often better than 50/50, I expect multiple trials have cured everyone.
- bawolff 3y ago
  In the history of the world, i am pretty sure there have been more than 5000 cancer trials.
  - inglor_cz 3y ago
    This entire chain of comments rests on the presumption of a "coin toss". If colorectal cancers in 2nd or 3rd stage actually had a fifty-fifty chance of spontaneous remission in the real world, the disease would be a lot less scary than it is. They don't. Spontaneous remission does happen, but isn't common. The chance that a 12-patient trial produces spontaneous remission in all 12 patients in a truly random fashion, is much smaller than 1:5000. And the GP actually mentioned it. But everyone reacting to him still addresses the 1:5000 value. It is a nice example of anchoring as described by Daniel Kahneman - here is a number, let's cling to it.
    - dekhn 3y ago
      You totally misunderstood my comment. I never meant that all 12 patients had spontaneous remission. I understand why people think that- my statement was ambiguous. What I did mean was that it's possible that some of the patients cancers went into remission for a reason that was not specific to this drug. It could be 11 people got better because of the drug- and 1 got better because they were being treated in a world-class hospital with a number of other factors that affect disease progression. THat's how it works in clinical trials. I didn't make my statement out of ignorance- I worked adjacent to trials for decades and have seen all sorts of claims of results that didn't hold up to scrutiny.