4 ms·

"extensive data" is 44K patients!? Wait until I tell you about the extremely extensive data that tech companies at scale deal with every day. I run a large data

"extensive data" is 44K patients!? Wait until I tell you about the extremely extensive data that tech companies at scale deal with every day. I run a large data science organization at a major tech company. If someone told me that it was going to take 3wks to analyze an experiment of this type, I would probably have to start firing people.

6gvONxR4sf7o 6y ago
If someone wanted to take three weeks to analyze an experiment with these kinds of stakes, you’d fire them?! I have to assume you’re being hyperbolic. And are you suggesting that it’s easier because the data isn’t as extensive as what the rest of us deal with? That only makes it harder. Sure if it’s some random A/B test with data from your own systems from an experiment you designed, but this is a vaccine the whole fucking world has been waiting for for most of the year. God forbid people treat it a little more carefully.
- creddit 6y ago
  > If someone wanted to take three weeks to analyze an experiment with these kinds of stakes, you’d fire them?! I would absolutely need to start a performance improvement process. They are definitionally 1/3 as productive as they should be. Also in terms of stakes, please show me even a modicum of work that suggests the downside risk of approval is greater than the delayed vaccinations. What reason have I to believe there is great risk in approval? Again, the UK approved this over a week ago. > And are you suggesting that it’s easier because the data isn’t as extensive as what the rest of us deal with? That only makes it harder. No, I'm mocking the idea of it being extensive data. It's not. > but this is a vaccine the whole fucking world has been waiting for for most of the year. And that's reason to go slower!? There's no need for this delay!!! https://thedispatch.com/p/fda-career-staff-are-delaying-the https://thedispatch.com/p/fda-career-staff-are-delaying-the
  - 6gvONxR4sf7o 6y ago
    > Also in terms of stakes, please show me even a modicum of work that suggests the downside risk of approval is greater than the delayed vaccinations. What reason have I to believe there is great risk in approval? As I understand it, medical ethics isn’t about utilitarianism, or else we’d skip the rats and monkeys and just go straight to testing on the poor for money. They don’t seem to prioritize minimizing the opportunity cost of avoiding bad outcomes. Their guiding principle seems to be avoiding giving harmful treatment. Meanwhile some healthcare workers at my partner’s facility are skeptical about whether the vaccine was rushed (I think they’re crazy, for the record). But there’s real risk if enough of the public doesn’t trust the vaccine. I’d bet that kind of angle is part of why they focus on “do no harm” instead of “minimize cumulative regret.”
    - creddit 6y ago
      But like even all of that being the case, that’s still no justification for taking 3wks to validate the data. The actual testing process is no different, the only difference is in how quickly approval comes. The same evidence is considered regardless. So what is the ethical justification for taking longer to consider the exact same evidence? Now if your point were that they should consider more evidence, then that’s fine and I can understand that but that’s not what’s at question here. The question is the speed at which the evidence is considered. What is the medical ethics justification that the UK was able to use to get this out over a week ago that is invalid in the US?
      - 6gvONxR4sf7o 6y ago
        I bring up the ethics question because a strict utilitarian analysis would be primarily answer the question of whether the drug is effective, looking for any red flags secondarily. A do no harm analysis (in the ethics sense, not the experimental design sense) has to look for any harms as a primary goal, maybe even equally to the question of efficacy. Answering the question of whether the experiment harmed anything important is always much harder, as I’m sure you know. You see some metric is way down, but after a multiple comparisons correction it might just as well be noise, so then you have to start digging, doing a bespoke analysis for the fact that these non iid metrics point in these particular directions. You can’t pre-automate it because with N metrics that are practically significantly up/flat/down, that’s at least 3^N possible investigations to convince yourself whether it’s enough to stop rollout. That’s the part that takes time, in any analysis I’ve ever done. Maybe literally zero time to check efficacy because that part is automated, then sometimes days to check whether there’s anything else worrisome in the probably-just-noise on other metrics, spending time proportional to the cost of getting the harm part wrong. Some of the hard questions aren’t statistical. You see some KPIs definitely up and some KPIs definitely down and deciding whether to roll out becomes a bitter fight between competing interests. Maybe you don’t roll it out yet for client segment X (like this trial and kids and pregnant women, as I understand it). Re US vs UK, no idea. I’m not saying 3 weeks is necessarily the right amount of time here. I’m just so taken aback at the idea that 3 weeks is necessarily too long to the point you “would probably have to start firing people” and would “absolutely” have to start putting them on PIPs.
raverbashing 6y ago
I don't remember the last time I saw such a completely clueless comment like this one I'm glad we have actual specialists checking the data and not people thinking it's only putting stuff on excel and writing a function
- creddit 6y ago
  So where am I wrong? Where is this MD, MPH from Johns Hopkins wrong? https://thedispatch.com/p/fda-career-staff-are-delaying-the https://thedispatch.com/p/fda-career-staff-are-delaying-the Why is the UK able to approve it over a week ago but the FDA needs longer?
  - raverbashing 6y ago
    > So where am I wrong? Where you think processing the data just means loading up an excel table (or running a CSV through R) and that's the only data relating to the studies. Patient data is not the only thing being analyzed and it includes things like manufacturing, transport, etc > Where is this MD, MPH from Johns Hopkins wrong? He's not wrong but his worries as a clinical trial expert are a subset of the worries of an agency like the FDA. And I'm sure his workflow as clinical study expert is more fine tuned for that. Or as in a developer analogy: making a code change is a 5 minute job. Actually making the change, testing, deploying, etc may take a whole day. > Why is the UK able to approve it over a week ago but the FDA needs longer? Pfizer sent the last info to the FDA on 22th of November. The MHRA had started a rolling review and received the last batch of info they needed on the 23rd of November. I am certainly frustrated with the speed as well, but I don't think it's too much of a difference. Yes, maybe it could be faster.
    - creddit 6y ago
      > Where you think processing the data just means loading up an excel table (or running a CSV through R) Is it possible you could construct an even more ridiculous straw man? > I am certainly frustrated with the speed as well... Yes, maybe it could be faster. Oh, so you agree with what I'm saying and your problem is the straw man argument you created.

6gvONxR4sf7o 6y ago

If someone wanted to take three weeks to analyze an experiment with these kinds of stakes, you’d fire them?! I have to assume you’re being hyperbolic. And are you suggesting that it’s easier because the data isn’t as extensive as what the rest of us deal with? That only makes it harder. Sure if it’s some random A/B test with data from your own systems from an experiment you designed, but this is a vaccine the whole fucking world has been waiting for for most of the year. God forbid people treat it a little more carefully.

creddit 6y ago
> If someone wanted to take three weeks to analyze an experiment with these kinds of stakes, you’d fire them?! I would absolutely need to start a performance improvement process. They are definitionally 1/3 as productive as they should be. Also in terms of stakes, please show me even a modicum of work that suggests the downside risk of approval is greater than the delayed vaccinations. What reason have I to believe there is great risk in approval? Again, the UK approved this over a week ago. > And are you suggesting that it’s easier because the data isn’t as extensive as what the rest of us deal with? That only makes it harder. No, I'm mocking the idea of it being extensive data. It's not. > but this is a vaccine the whole fucking world has been waiting for for most of the year. And that's reason to go slower!? There's no need for this delay!!! https://thedispatch.com/p/fda-career-staff-are-delaying-the https://thedispatch.com/p/fda-career-staff-are-delaying-the
- 6gvONxR4sf7o 6y ago
  > Also in terms of stakes, please show me even a modicum of work that suggests the downside risk of approval is greater than the delayed vaccinations. What reason have I to believe there is great risk in approval? As I understand it, medical ethics isn’t about utilitarianism, or else we’d skip the rats and monkeys and just go straight to testing on the poor for money. They don’t seem to prioritize minimizing the opportunity cost of avoiding bad outcomes. Their guiding principle seems to be avoiding giving harmful treatment. Meanwhile some healthcare workers at my partner’s facility are skeptical about whether the vaccine was rushed (I think they’re crazy, for the record). But there’s real risk if enough of the public doesn’t trust the vaccine. I’d bet that kind of angle is part of why they focus on “do no harm” instead of “minimize cumulative regret.”
  - creddit 6y ago
    But like even all of that being the case, that’s still no justification for taking 3wks to validate the data. The actual testing process is no different, the only difference is in how quickly approval comes. The same evidence is considered regardless. So what is the ethical justification for taking longer to consider the exact same evidence? Now if your point were that they should consider more evidence, then that’s fine and I can understand that but that’s not what’s at question here. The question is the speed at which the evidence is considered. What is the medical ethics justification that the UK was able to use to get this out over a week ago that is invalid in the US?
    - 6gvONxR4sf7o 6y ago
      I bring up the ethics question because a strict utilitarian analysis would be primarily answer the question of whether the drug is effective, looking for any red flags secondarily. A do no harm analysis (in the ethics sense, not the experimental design sense) has to look for any harms as a primary goal, maybe even equally to the question of efficacy. Answering the question of whether the experiment harmed anything important is always much harder, as I’m sure you know. You see some metric is way down, but after a multiple comparisons correction it might just as well be noise, so then you have to start digging, doing a bespoke analysis for the fact that these non iid metrics point in these particular directions. You can’t pre-automate it because with N metrics that are practically significantly up/flat/down, that’s at least 3^N possible investigations to convince yourself whether it’s enough to stop rollout. That’s the part that takes time, in any analysis I’ve ever done. Maybe literally zero time to check efficacy because that part is automated, then sometimes days to check whether there’s anything else worrisome in the probably-just-noise on other metrics, spending time proportional to the cost of getting the harm part wrong. Some of the hard questions aren’t statistical. You see some KPIs definitely up and some KPIs definitely down and deciding whether to roll out becomes a bitter fight between competing interests. Maybe you don’t roll it out yet for client segment X (like this trial and kids and pregnant women, as I understand it). Re US vs UK, no idea. I’m not saying 3 weeks is necessarily the right amount of time here. I’m just so taken aback at the idea that 3 weeks is necessarily too long to the point you “would probably have to start firing people” and would “absolutely” have to start putting them on PIPs.

raverbashing 6y ago

I don't remember the last time I saw such a completely clueless comment like this one I'm glad we have actual specialists checking the data and not people thinking it's only putting stuff on excel and writing a function

creddit 6y ago
So where am I wrong? Where is this MD, MPH from Johns Hopkins wrong? https://thedispatch.com/p/fda-career-staff-are-delaying-the https://thedispatch.com/p/fda-career-staff-are-delaying-the Why is the UK able to approve it over a week ago but the FDA needs longer?
- raverbashing 6y ago
  > So where am I wrong? Where you think processing the data just means loading up an excel table (or running a CSV through R) and that's the only data relating to the studies. Patient data is not the only thing being analyzed and it includes things like manufacturing, transport, etc > Where is this MD, MPH from Johns Hopkins wrong? He's not wrong but his worries as a clinical trial expert are a subset of the worries of an agency like the FDA. And I'm sure his workflow as clinical study expert is more fine tuned for that. Or as in a developer analogy: making a code change is a 5 minute job. Actually making the change, testing, deploying, etc may take a whole day. > Why is the UK able to approve it over a week ago but the FDA needs longer? Pfizer sent the last info to the FDA on 22th of November. The MHRA had started a rolling review and received the last batch of info they needed on the 23rd of November. I am certainly frustrated with the speed as well, but I don't think it's too much of a difference. Yes, maybe it could be faster.
  - creddit 6y ago
    > Where you think processing the data just means loading up an excel table (or running a CSV through R) Is it possible you could construct an even more ridiculous straw man? > I am certainly frustrated with the speed as well... Yes, maybe it could be faster. Oh, so you agree with what I'm saying and your problem is the straw man argument you created.