5 ms·

It's only a "great way to create statistical fraud" if you assume that the statisticians, and the FDA, are either stupid or corrupt. And contrary to popular bel

It's only a "great way to create statistical fraud" if you assume that the statisticians, and the FDA, are either stupid or corrupt. And contrary to popular belief, they tend to be competent enough to at least cope with the sort of hair-brained schemes HN believes would fool them.[0] Stopping a trial early and it's statistical implications should, by the way, be somewhat familiar with web developers: it's commonly done with A/B tests, and the related problem is the "One-armed bandit" (https://en.wikipedia.org/wiki/Multi-armed_bandit#Empirical_motivation https://en.wikipedia.org/wiki/Multi-armed_bandit#Empirical_m...) [0]: Legend says they are even familiar with correlation != causation, and some have mastered the advanced skill of knowing that trial size matters, which is why they started, a few years back, to test drugs on more than one person.

Simon_says 7y ago
The word is harebrained unless you think hair has a brain.
- sjcsjc 7y ago
  Thanks - I didn't know this. http://www.word-detective.com/2012/08/harebrained/ http://www.word-detective.com/2012/08/harebrained/
- the_seraphim 7y ago
  surely it means to have a brain made of hair?
WhitneyLand 7y ago
>>the sort of hair-brained schemes HN believes would fool them Is there much value in an ad-hominem reply? Or in generalizing it to an entire community? The irony seems hard to avoid when such a generalization is made without mentioning any statistical foundation, while pontificating about statistics.
- eitland 7y ago
  > Is there much value in an ad-hominem reply? > Or in generalizing it to an entire community? I don't like this style either but sometimes I think we deserve it for being a smug echo chamber. Basically, much the same thing could have been said in a different way and I would have upvoted it.
  - nindalf 7y ago
    > we deserve it for being a smug echo chamber. This and more. Almost every comment here is made with absolute confidence and authority, much more than is warranted. The tone of the person who said that "it's a great way to commit statistical fraud" without a caveat sounded like a know-it-all who knows better than the FDA and the scientists who made this cure in the first place or that they're outright unethical. Most times such comments get upvoted. Sometimes they get downvoted when they're called out on their BS. So there's some balance.
- benj111 7y ago
  Have you read any thread that involves statistics? I havent really detected that low an opinion for the FDA in particular but I recognised the broad outline painted. I could add quite a few more generalisations myself while I'm at it, HN isn't solely populated by hyper intelligent, well reasoned, logical doctors, who form their ideas completely independently of everyone else. There's going to be group think, there's going to be like minded individuals attracted to one another, that's human nature, deal with it. I don't even think it's an ad-hominem. HNers are generally argumentative gits (generalisation), who like finding an exception/hack/new way of looking at something (generalisation). I have no problem with some of those schemes being described as 'hairbrained' (sic although actually... [1]), but that's why I come here, because if people didn't think that way they wouldn't be Hackers. [1] https://www.thefreedictionary.com/harebrained https://www.thefreedictionary.com/harebrained
  - friendly_chap 7y ago
    > sic although actually... Not sure if that was for comedic effect but I laughed out loud! A brilliant demonstration of the behaviour you describe.
mlevental 7y ago
rule #1 of hn: anyone on hn is smarter than everyone not on hn no matter the domain. we could solve all the world's problems by just getting everyone into hn!
- powerbroker 7y ago
  rule #2 of hn: High signal to noise is preferred over low signal to noise. HN > twitter.
  - skosuri 7y ago
    To be fair, and when it comes to health and biology, Twitter >> HN. The comments on most bio-related posts here make me cringe.
    - Fomite 7y ago
      This. I read HN regularly, but for health and biology related information, I'd much rather turn to the various professional communities on Twitter. HN is at best "Lies my Bio 101 Professor Told Me" most of the time, and at worst a worked example of physicist/engineer's syndrome.
- excalibur 7y ago
  We have the best brains!
hanniabu 7y ago
> It's only a "great way to create statistical fraud" if you assume that the statisticians, and the FDA, are either stupid or corrupt. Considering how much money is in pharmaceuticals, I'd definitely go with corrupt, especially considering this explains how a majority of the world works now and also since there's currently an outbreak and likely want to capitalize on it. I'm not at all saying this is the case, but that if something like that did come out then I really wouldn't br surprised considering it seems to be a weekly occurrence that you hear about corrupt behavior now.
- TheSpiceIsLife 7y ago
  Are you familiar with what PJ O'Rouke says about politicians? Have a listen to this https://www.youtube.com/watch?v=JbIqKqojOZU https://www.youtube.com/watch?v=JbIqKqojOZU He is the author of Parliament of Whores Parliament of Whores is an international best-selling political humor book by P. J. O'Rourke published by Atlantic Monthly Press in 1991. Subtitled "A Lone Humorist Attempts to Explain the Entire US Government"
- excalibur 7y ago
  I feel like Hanlon's razor is applicable here.
- parasubvert 7y ago
  The issue with cynicism and corruption conspiracy theories is that they become a self-fulfilling prophecy. If everyone is corrupt, then it’s OK if you are, etc. There is no fair application of the rule of law, there’s no reason for rule of law, only the rule of men. Better hope they’re on your side! My point is that clear corruption as we think of it (fraud, bibery, self dealing) in most Western countries is NOT considered normal. Ie. It happens but we’ve built institutions to counteract it. The FDA is notoriously bureaucratic in order to detect and prevent corruption, and for the few cases to slip through, to correct it. When you hear about corruption stories, that’s about the system WORKING against those cynics that are testing it for their own benefit. Cynicism is self destructive. If our institutions are broken, we must work to repair them.
TeMPOraL 7y ago
> Stopping a trial early and it's statistical implications should, by the way, be somewhat familiar with web developers: it's commonly done with A/B tests AFAIK stopping an A/B test early is the way to go if you want to convince yourself (or your customer) that something has an effect, even though it hasn't.
- laughinghan 7y ago
  Yes, if you "peek" at the results of an A/B test before it's done in order to decide whether to stop early, the numbers you "peeked" at have much lower statistical power than if you foreswore stopping early. Obviously, failing to take that into account when drawing conclusions about the effectiveness of the treatment is a colossal mistake. However, the decrease in statistical power is still quantifiable, and with the right math you can still calculate an accurate 95% confidence interval on the effect size (which will be much wider than the wrong math where you naively don't account for the "peeking"). And of course, it's totally possible that the treatment is so effective that even the accurate calculation shows that the lower bound on the effect size is so much higher than the control group that the responsible thing to do is to stop the trial early. Here's some the math on how much lower the statistical power is if you "peek": https://www.evanmiller.org/bayesian-ab-testing.html https://www.evanmiller.org/bayesian-ab-testing.html
panopticon 7y ago
> It's only a "great way to create statistical fraud" if you assume that the statisticians, and the FDA, are either stupid or corrupt. This is a legitimate issue in the scientific community. The FDA made it a requirement to publish drug trial criteria beforehand to prevent trials from stopping prematurely or adding more participants. This resulted in a massive drop in drugs with positive results: "...before the registry was created, more than half of the published studies of heart disease showed positive results. After the registry was created, only 8 percent had positive results". [0] The FDA and scientific community isn't as "pure" as your flippant dismissal would suggest. The FDA can only do so much, and trial runners have misaligned incentives. [0] https://www.npr.org/templates/transcript/transcript.php?storyId=463237871 https://www.npr.org/templates/transcript/transcript.php?stor...
slowhand09 7y ago
I was a contractor working on an IS at the FDA in the early 2000's. Shortly before the VIOXX scandal (https://www.drugwatch.com/vioxx/ https://www.drugwatch.com/vioxx/). The number of cheats they were specifying to go around safeguards was scary. Fortunately, they fired the company I worked for and awarded the contract to the company employing the husband of one of the FDA managers. ISYN. A relief to leave.

Simon_says 7y ago

The word is harebrained unless you think hair has a brain.

sjcsjc 7y ago
Thanks - I didn't know this. http://www.word-detective.com/2012/08/harebrained/ http://www.word-detective.com/2012/08/harebrained/
the_seraphim 7y ago
surely it means to have a brain made of hair?

WhitneyLand 7y ago

>>the sort of hair-brained schemes HN believes would fool them Is there much value in an ad-hominem reply? Or in generalizing it to an entire community? The irony seems hard to avoid when such a generalization is made without mentioning any statistical foundation, while pontificating about statistics.

eitland 7y ago
> Is there much value in an ad-hominem reply? > Or in generalizing it to an entire community? I don't like this style either but sometimes I think we deserve it for being a smug echo chamber. Basically, much the same thing could have been said in a different way and I would have upvoted it.
- nindalf 7y ago
  > we deserve it for being a smug echo chamber. This and more. Almost every comment here is made with absolute confidence and authority, much more than is warranted. The tone of the person who said that "it's a great way to commit statistical fraud" without a caveat sounded like a know-it-all who knows better than the FDA and the scientists who made this cure in the first place or that they're outright unethical. Most times such comments get upvoted. Sometimes they get downvoted when they're called out on their BS. So there's some balance.
benj111 7y ago
Have you read any thread that involves statistics? I havent really detected that low an opinion for the FDA in particular but I recognised the broad outline painted. I could add quite a few more generalisations myself while I'm at it, HN isn't solely populated by hyper intelligent, well reasoned, logical doctors, who form their ideas completely independently of everyone else. There's going to be group think, there's going to be like minded individuals attracted to one another, that's human nature, deal with it. I don't even think it's an ad-hominem. HNers are generally argumentative gits (generalisation), who like finding an exception/hack/new way of looking at something (generalisation). I have no problem with some of those schemes being described as 'hairbrained' (sic although actually... [1]), but that's why I come here, because if people didn't think that way they wouldn't be Hackers. [1] https://www.thefreedictionary.com/harebrained https://www.thefreedictionary.com/harebrained
- friendly_chap 7y ago
  > sic although actually... Not sure if that was for comedic effect but I laughed out loud! A brilliant demonstration of the behaviour you describe.

mlevental 7y ago

rule #1 of hn: anyone on hn is smarter than everyone not on hn no matter the domain. we could solve all the world's problems by just getting everyone into hn!

powerbroker 7y ago
rule #2 of hn: High signal to noise is preferred over low signal to noise. HN > twitter.
- skosuri 7y ago
  To be fair, and when it comes to health and biology, Twitter >> HN. The comments on most bio-related posts here make me cringe.
  - Fomite 7y ago
    This. I read HN regularly, but for health and biology related information, I'd much rather turn to the various professional communities on Twitter. HN is at best "Lies my Bio 101 Professor Told Me" most of the time, and at worst a worked example of physicist/engineer's syndrome.
excalibur 7y ago
We have the best brains!

hanniabu 7y ago

> It's only a "great way to create statistical fraud" if you assume that the statisticians, and the FDA, are either stupid or corrupt. Considering how much money is in pharmaceuticals, I'd definitely go with corrupt, especially considering this explains how a majority of the world works now and also since there's currently an outbreak and likely want to capitalize on it. I'm not at all saying this is the case, but that if something like that did come out then I really wouldn't br surprised considering it seems to be a weekly occurrence that you hear about corrupt behavior now.

TheSpiceIsLife 7y ago
Are you familiar with what PJ O'Rouke says about politicians? Have a listen to this https://www.youtube.com/watch?v=JbIqKqojOZU https://www.youtube.com/watch?v=JbIqKqojOZU He is the author of Parliament of Whores Parliament of Whores is an international best-selling political humor book by P. J. O'Rourke published by Atlantic Monthly Press in 1991. Subtitled "A Lone Humorist Attempts to Explain the Entire US Government"
excalibur 7y ago
I feel like Hanlon's razor is applicable here.
parasubvert 7y ago
The issue with cynicism and corruption conspiracy theories is that they become a self-fulfilling prophecy. If everyone is corrupt, then it’s OK if you are, etc. There is no fair application of the rule of law, there’s no reason for rule of law, only the rule of men. Better hope they’re on your side! My point is that clear corruption as we think of it (fraud, bibery, self dealing) in most Western countries is NOT considered normal. Ie. It happens but we’ve built institutions to counteract it. The FDA is notoriously bureaucratic in order to detect and prevent corruption, and for the few cases to slip through, to correct it. When you hear about corruption stories, that’s about the system WORKING against those cynics that are testing it for their own benefit. Cynicism is self destructive. If our institutions are broken, we must work to repair them.

TeMPOraL 7y ago

> Stopping a trial early and it's statistical implications should, by the way, be somewhat familiar with web developers: it's commonly done with A/B tests AFAIK stopping an A/B test early is the way to go if you want to convince yourself (or your customer) that something has an effect, even though it hasn't.

laughinghan 7y ago
Yes, if you "peek" at the results of an A/B test before it's done in order to decide whether to stop early, the numbers you "peeked" at have much lower statistical power than if you foreswore stopping early. Obviously, failing to take that into account when drawing conclusions about the effectiveness of the treatment is a colossal mistake. However, the decrease in statistical power is still quantifiable, and with the right math you can still calculate an accurate 95% confidence interval on the effect size (which will be much wider than the wrong math where you naively don't account for the "peeking"). And of course, it's totally possible that the treatment is so effective that even the accurate calculation shows that the lower bound on the effect size is so much higher than the control group that the responsible thing to do is to stop the trial early. Here's some the math on how much lower the statistical power is if you "peek": https://www.evanmiller.org/bayesian-ab-testing.html https://www.evanmiller.org/bayesian-ab-testing.html

panopticon 7y ago

> It's only a "great way to create statistical fraud" if you assume that the statisticians, and the FDA, are either stupid or corrupt. This is a legitimate issue in the scientific community. The FDA made it a requirement to publish drug trial criteria beforehand to prevent trials from stopping prematurely or adding more participants. This resulted in a massive drop in drugs with positive results: "...before the registry was created, more than half of the published studies of heart disease showed positive results. After the registry was created, only 8 percent had positive results". [0] The FDA and scientific community isn't as "pure" as your flippant dismissal would suggest. The FDA can only do so much, and trial runners have misaligned incentives. [0] https://www.npr.org/templates/transcript/transcript.php?storyId=463237871 https://www.npr.org/templates/transcript/transcript.php?stor...

slowhand09 7y ago

I was a contractor working on an IS at the FDA in the early 2000's. Shortly before the VIOXX scandal (https://www.drugwatch.com/vioxx/ https://www.drugwatch.com/vioxx/). The number of cheats they were specifying to go around safeguards was scary. Fortunately, they fired the company I worked for and awarded the contract to the company employing the husband of one of the FDA managers. ISYN. A relief to leave.