5 ms·

The FDA reran, in a matter of weeks, the entire analysis undertaken by Pfizer’s pharmacologists and statisticians over a series of months.

zaroth 6y ago
To be fair, this was an extremely basic analysis on a relatively minuscule amount of data, where the form and format of data and the method of analysis has been prescribed months ago. There was no decision that shouldn’t have been made ahead of time. Either it met the predefined criteria or it didn’t.
- Retric 6y ago
  Regulators need to independently verify more than just raw calculations. The incentives to hide or falsify data can be enormous.
  - zaroth 6y ago
    The only thing that couldn’t be verified until the last minute was the unblinded results. Everything else should have been a continuous process of pre-verification up to the point the 128 positive confirmed COVID cases were unblinded. Assume the unblinded results will be good enough, and verify everything else you would normally verify ahead of time. The only thing that remains is to verify the unblinded cases the day they are announced. And not just that but your approval thresholds should already be preset and preregistered, the doses already distributed to point-of-care, and nurses start administering shots the very next day. Yes it’s a big waste of money if it turns out not to work. That’s the whole point. Turns out the vaccine is highly effective 10 days after the first dose. We could have been doing at least a million doses a day since November 19th. Realistically even long before then.
    - refurb 6y ago
      Ahhh... the old HN post of “I work in tech so I know more about how to approve an experimental vaccine than the FDA which is comprised of thousands of scientists and has been approving drugs for over 70 years.”
      - zaroth 6y ago
        Remember when the CDC was considered the world’s absolutely top infectious disease center? Remember when they couldn’t even ship COVID test kits with uncontaminated distilled water? Remember when the FDA wouldn’t let public or private labs run their own COVID tests and everything was being shipped to CDC headquarters to PCR? The CDC and the FDA essentially guaranteed through extraordinary ineptitude that the US would not fend off COVID back in January/February. They deserve zero benefit of the doubt. The FDA’s 70 years of process likely worked against them here. What I can say for certain is that there was a t-minus one, the day before the results were unblinded, and there was a t-zero, the day results were unblinded. On that day t-zero they uncovered which of the patient IDs got the vaccine versus which got placebo. No expense should have been spared in preloading the analysis, verification, whatever could have been done, it should have been done up until t-minus-1. On day t-zero you get 15,000 patient IDs in column A and 15,000 in column B, that’s the only new thing you learn that day. There should be nothing new to talk about or discuss, the approval criteria should already be set in stone. Check the IDs and launch.
        loopz 6y ago
        Don't forget who is the Chief in command responsible for this outcome.
        brian_cloutier 6y ago
        I'm sorry you're being downvoted. The testing disaster killed far more than this vaccine approval delay has killed and it's infuriating that nobody seems to care. They shut down testing back when we needed it most, allowing us to pretend there was no spread in the US for far too long: https://www.statnews.com/2020/05/27/coronavirus-testing-seattle-bill-gates-fda/ https://www.statnews.com/2020/05/27/coronavirus-testing-seat... They spent far too long before approving any kind of at-home testing and to this day have not approved most rapid testing. Three weeks ago the EUA was submitted, so three weeks ago there was an entire team of people at pfizer who had reviewed all the data and were confident FDA would approve the application, but somehow nobody at FDA was part of that set of people. In any reasonable process that application would have contained no new information, if it contained any new information they were not communicating closely enough. I understand spending a few days doing some final checks on the data before approving. Three weeks, however, is suspiciously long. It's possible they were in fact doing important work which could not possibly have been done before the application was submitted, but nobody seems able to articulate what that important work actually was.
    - 6gvONxR4sf7o 6y ago
      I’m not in the drug industry, but in mine you can never turn the full decision process into a push button. Its trivial to turn the tests on your main metrics (is it effective) into push button, but you also have to look to make sure nothing else concerning is present, and it’s muuuch harder to formalize “nothing else here to make you worry” independent of the data.
  - aphextron 6y ago
    >The incentives to hide or falsify data can be enormous. Highly doubtful. Pushing out an unsafe vaccine would be disastrous to Pfizer. They have every incentive in the world to be as absolutely certain as possible that the data is solid.
    - loopz 6y ago
      That is a scientifically weak argument, and medically dangerous one.
      - aphextron 6y ago
        > That is a scientifically weak argument, and medically dangerous one. And so is the argument that Pfizer would have any incentive to fudge the data.
        loopz 6y ago
        Please refrain from using strawman argument.
  - wbl 6y ago
    Only the FDA does things this way.
    - ummonk 6y ago
      Perhaps e.g. the FAA should have done things the same way with the Boeing 737 Max certification.
      - cyberlurker 6y ago
        Excellent comparison. These agencies shouldn’t blindly accept what the private sector puts out.
creddit 6y ago
What kind of computation do you think it is they are doing on this data? Seriously, it takes at MOST a week to do an extremely deep code review of this. It's not like this is some brand new testing methodology developed ad hoc for this. How is it you think the UK was able to approve it already??? https://marginalrevolution.com/marginalrevolution/2020/12/how-badly-has-the-fda-been-lagging.html https://marginalrevolution.com/marginalrevolution/2020/12/ho... EDIT: Multiple downvotes already. Anyone care to refute what's presented in my post? Did you read any of the sources cited in the linked blogpost?
- loopz 6y ago
  The link provides some arguments and maybe FDA should've/could've rolled with the process. Long times do not provide safety, only double blind tests and verifications do. Are Gov jobs granted overtime work by ruling party, and how was this important work prioritized and coordinated by top leader?
winston_smith 6y ago
This Johns Hopkins doc claims they took off four days for Thanksgiving: https://thedispatch.com/p/fda-career-staff-are-delaying-the https://thedispatch.com/p/fda-career-staff-are-delaying-the

zaroth 6y ago

To be fair, this was an extremely basic analysis on a relatively minuscule amount of data, where the form and format of data and the method of analysis has been prescribed months ago. There was no decision that shouldn’t have been made ahead of time. Either it met the predefined criteria or it didn’t.

Retric 6y ago
Regulators need to independently verify more than just raw calculations. The incentives to hide or falsify data can be enormous.
- zaroth 6y ago
  The only thing that couldn’t be verified until the last minute was the unblinded results. Everything else should have been a continuous process of pre-verification up to the point the 128 positive confirmed COVID cases were unblinded. Assume the unblinded results will be good enough, and verify everything else you would normally verify ahead of time. The only thing that remains is to verify the unblinded cases the day they are announced. And not just that but your approval thresholds should already be preset and preregistered, the doses already distributed to point-of-care, and nurses start administering shots the very next day. Yes it’s a big waste of money if it turns out not to work. That’s the whole point. Turns out the vaccine is highly effective 10 days after the first dose. We could have been doing at least a million doses a day since November 19th. Realistically even long before then.
  - refurb 6y ago
    Ahhh... the old HN post of “I work in tech so I know more about how to approve an experimental vaccine than the FDA which is comprised of thousands of scientists and has been approving drugs for over 70 years.”
    - zaroth 6y ago
      Remember when the CDC was considered the world’s absolutely top infectious disease center? Remember when they couldn’t even ship COVID test kits with uncontaminated distilled water? Remember when the FDA wouldn’t let public or private labs run their own COVID tests and everything was being shipped to CDC headquarters to PCR? The CDC and the FDA essentially guaranteed through extraordinary ineptitude that the US would not fend off COVID back in January/February. They deserve zero benefit of the doubt. The FDA’s 70 years of process likely worked against them here. What I can say for certain is that there was a t-minus one, the day before the results were unblinded, and there was a t-zero, the day results were unblinded. On that day t-zero they uncovered which of the patient IDs got the vaccine versus which got placebo. No expense should have been spared in preloading the analysis, verification, whatever could have been done, it should have been done up until t-minus-1. On day t-zero you get 15,000 patient IDs in column A and 15,000 in column B, that’s the only new thing you learn that day. There should be nothing new to talk about or discuss, the approval criteria should already be set in stone. Check the IDs and launch.
      - loopz 6y ago
        Don't forget who is the Chief in command responsible for this outcome.
      - brian_cloutier 6y ago
        I'm sorry you're being downvoted. The testing disaster killed far more than this vaccine approval delay has killed and it's infuriating that nobody seems to care. They shut down testing back when we needed it most, allowing us to pretend there was no spread in the US for far too long: https://www.statnews.com/2020/05/27/coronavirus-testing-seattle-bill-gates-fda/ https://www.statnews.com/2020/05/27/coronavirus-testing-seat... They spent far too long before approving any kind of at-home testing and to this day have not approved most rapid testing. Three weeks ago the EUA was submitted, so three weeks ago there was an entire team of people at pfizer who had reviewed all the data and were confident FDA would approve the application, but somehow nobody at FDA was part of that set of people. In any reasonable process that application would have contained no new information, if it contained any new information they were not communicating closely enough. I understand spending a few days doing some final checks on the data before approving. Three weeks, however, is suspiciously long. It's possible they were in fact doing important work which could not possibly have been done before the application was submitted, but nobody seems able to articulate what that important work actually was.
  - 6gvONxR4sf7o 6y ago
    I’m not in the drug industry, but in mine you can never turn the full decision process into a push button. Its trivial to turn the tests on your main metrics (is it effective) into push button, but you also have to look to make sure nothing else concerning is present, and it’s muuuch harder to formalize “nothing else here to make you worry” independent of the data.
- aphextron 6y ago
  >The incentives to hide or falsify data can be enormous. Highly doubtful. Pushing out an unsafe vaccine would be disastrous to Pfizer. They have every incentive in the world to be as absolutely certain as possible that the data is solid.
  - loopz 6y ago
    That is a scientifically weak argument, and medically dangerous one.
    - aphextron 6y ago
      > That is a scientifically weak argument, and medically dangerous one. And so is the argument that Pfizer would have any incentive to fudge the data.
      - loopz 6y ago
        Please refrain from using strawman argument.
- wbl 6y ago
  Only the FDA does things this way.
  - ummonk 6y ago
    Perhaps e.g. the FAA should have done things the same way with the Boeing 737 Max certification.
    - cyberlurker 6y ago
      Excellent comparison. These agencies shouldn’t blindly accept what the private sector puts out.

creddit 6y ago

What kind of computation do you think it is they are doing on this data? Seriously, it takes at MOST a week to do an extremely deep code review of this. It's not like this is some brand new testing methodology developed ad hoc for this. How is it you think the UK was able to approve it already??? https://marginalrevolution.com/marginalrevolution/2020/12/how-badly-has-the-fda-been-lagging.html https://marginalrevolution.com/marginalrevolution/2020/12/ho... EDIT: Multiple downvotes already. Anyone care to refute what's presented in my post? Did you read any of the sources cited in the linked blogpost?

loopz 6y ago
The link provides some arguments and maybe FDA should've/could've rolled with the process. Long times do not provide safety, only double blind tests and verifications do. Are Gov jobs granted overtime work by ruling party, and how was this important work prioritized and coordinated by top leader?

winston_smith 6y ago

This Johns Hopkins doc claims they took off four days for Thanksgiving: https://thedispatch.com/p/fda-career-staff-are-delaying-the https://thedispatch.com/p/fda-career-staff-are-delaying-the