6 ms·

How do you make 15 million of anything without realizing there's an issue, let alone the most scrutinized thing on the planet right now?

endisneigh 6y ago
> Context: Workers at the plant, which had been producing Johnson & Johnson and AstraZeneca vaccines, conflated the ingredients between the two different types of vaccines, destroying 15 million doses of Johnson & Johnson vaccine, according to the Times.
- markdown 6y ago
  Jesus Christ, this shouldn't happen in a certified cGMP (food) facility let alone a pharmaceutical plant. HACCP would prevent this in any number of ways.
  - eric-hu 6y ago
    Can you describe in more detail how that process would prevent such an error?
    - markdown 6y ago
      Hard to do without knowing anything about this facility, but maybe just not store the ingredients for two different vaccines in the same place. Basically restrict the set of ingredients available to the person making the batch. Or have the employee log what he is doing in real time BEFORE he does it. I am adding sugar, I am adding eggs, I am adding wheat flour. Then an alert is triggered when the system (which knows what product is being made) recognises that the ingredient doesn't belong. This is a multi-billion dollar company... they could have the process live-streamed to 30 independent chemists each one watching the "chef's" every move (this is not a serious proposal). This stuff is all hashed out when one writes up a HACCP process.
  - epistasis 6y ago
    Errors like this happen in every single GMP and clinical facility. What makes them different is 1) there are ample checks in place to prevent mistakes from ever leaving the facility, and 2) there is process that reduces the chances of having it happen in the first place. But all of these places will have incident logs of mistakes like this, along with corrective actions that change the procedures and make it less likely to happen a second time.
    - markdown 6y ago
      > Errors like this happen in every single GMP and clinical facility. They shouldn't happen in a 150yr old, multi-billion dollar pharmaceutical company. How hard is it to restrict the ingredients available to the "chef" to the ones needed for the current product? Not hard at all unless you want to save a few dollars cutting corners.
      - Kim_Bruning 6y ago
        Sounds like what epistasis is describing is a healthy QA environment with rapid feedback. Some other comments have already pointed out that 15M doses is probably only one or a few batches. Combined with the fact that the procedure documents for this particular line are probably still warm from the laser printer, I'd say let them iterate a bit more to get the kinks out. [ Procedures are not (should not be) engraved on stone tablets handed down from on high. If you try it that way, you are just as guaranteed to fail as when you had no procedures at all. ] As to what went wrong at this facility (besides overpromising?); we'll have to see. I wish there was such a thing as open source procedures!
      - epistasis 6y ago
        Being GMP isn't about eliminating all errors everywhere, it's about acknowledging the inevitability of error, minimizing error through conscious processes that stop those errors, and testing to assure that you made what you made. In a situation where maximizing vaccine production sooner saves lives, pinning this on saving a few dollars doesn't make sense. Whatever they were doing cost them quite a few dollars in addition to the lives lost from slower vaccination. I don't know their process, but suppose the "restricting the ingredients" phase happens. What if the ingredients were swapped at the time of restriction? Same problem happens.
  - Johnny555 6y ago
    I think you mean it shouldn't happen undetected... these kinds of mistakes can and do happen. Everything from human error (oops, Billy put baking powder in the baking soda tank) to equipment error (the tank valve stuck open, adding 10X more of ingredient X than expected)
senectus1 6y ago
It'll be one vat with the potential for 15 million doses... the corrupted mix never reached the individual dose stage.
- taeric 6y ago
  I find it sad how quick the narrative jumps to the more absurd 15 million number. Obviously, the mistake is bad. But this is something with a very high fan out. And the smaller number just doesn't get the clicks. :(
  - maxerickson 6y ago
    The batch time seems to be weeks/months, so there is some relevance to the dose count (it's wasted time in addition to the wasted ingredients). And it's put other production from the facility into question.
    - mc32 6y ago
      And there will be an FDA investigation which will put further delays into new production.
    - taeric 6y ago
      But this is a natural consequence of mistakes in processes with very high fan out. Obviously, it should be investigated such that were can reduce the chance is mistakes in the future. But piling on with blame will just convince folks not to try.
      - maxerickson 6y ago
        Given the $10 a dose revenue, I'd call it a failure in the process rather than a mistake. That it was possible is the bigger problem than it happening. And then there's the much greater value of the doses once they have been administered.
  - Igelau 6y ago
    And it's J+J's name that goes in the headline. It wasn't even their fault. At least they didn't somehow cram the AZ connection in there too, though I'm sure other outlets are working on it.
deleted 6y ago
[deleted]
maxerickson 6y ago
Dose is 0.5 ml. Say the batch is equivalent in volume to the final doses (who knows, but it should be somewhere vaguely close). That's 7,500 liters. Microbrewers often use fermenters larger than that.
- manquer 6y ago
  The question should be more why would there have to be such large batches which can be SPOF for so many doses. Wouldn't it make more sense to make smaller seperate batches to derisk the process. This is akin to me using a large 64 core server for an app, instead of say 4x16 . Sure, the larger server can easily handle the load, however the 4x16 setup is more redundant and avoids many scenarios where all users will be out of service Just because there are high capacity machines capable of doing that much does not mean it should be used especially a With a new process trying to scale up
  - tomaskafka 6y ago
    You might have smaller batches and smaller throughput, but some pesky people keep shouting that we need all the vaccines yesterday :).
  - maxerickson 6y ago
    The production goal is hundreds of millions of doses this year, so I guess I wouldn't want to speculate too much about what batch size is sensible.
    - manquer 6y ago
      eventual goal maybe, Not for the month of March. 15M out of planned 20M ? . The scale out for any process starts small to try avoid this kind of issues. Yes fuck ups happen at plant and in planning, and yes we are not experts in the field to comment in detail on what is best approach. However if you are looking at it as an admin, clearly risk was not well managed.
  - fragmede 6y ago
    To use the same alarmism and a software analogy: sw developer destroys thousands of hours of CPU time with bad code!1 Yet every single time I push, the CI pipeline runs an extensive suite of automated tests, and I'd not have it any other way. Wait but CPUs aren't being destroyed by running tests, you say. Yeah but niche reporting is always atrocious and a mainstream writer can't be trusted to get things entirely right. (If you're using a 4x16 server, whatever that is, to provide isolation in your system, you may want to talk to additional devops folks for other options.)
dcolkitt 6y ago
I'm guessing because you're not really making "15 million of something". The process is more like making one or two (or ten) batches, which is then sub-divided into 15 million doses. It'd be like the difference between burning dinner 7 nights in a row, versus making a single mistake during Meal Prep Sunday and ruining a weeks worth of dinners all at once. The latter is much more understandable.
- justwalt 6y ago
  It’s rare that an analogy is so easy to understand while remaining highly accurate. Nice.
HWR_14 6y ago
It's probably analogous to making a mistake in a single line that gets pushed to production - that then gets patched downstream into all 15 million customer's systems. Large systems moving in giant batches.
ergocoder 6y ago
It's not the most scrutinized thing on the planet with the emergence of pro-vaxxers (the anti anti-vaxxers) who refuses to acknowledge anything bad about vaccines.

endisneigh 6y ago

> Context: Workers at the plant, which had been producing Johnson & Johnson and AstraZeneca vaccines, conflated the ingredients between the two different types of vaccines, destroying 15 million doses of Johnson & Johnson vaccine, according to the Times.

markdown 6y ago
Jesus Christ, this shouldn't happen in a certified cGMP (food) facility let alone a pharmaceutical plant. HACCP would prevent this in any number of ways.
- eric-hu 6y ago
  Can you describe in more detail how that process would prevent such an error?
  - markdown 6y ago
    Hard to do without knowing anything about this facility, but maybe just not store the ingredients for two different vaccines in the same place. Basically restrict the set of ingredients available to the person making the batch. Or have the employee log what he is doing in real time BEFORE he does it. I am adding sugar, I am adding eggs, I am adding wheat flour. Then an alert is triggered when the system (which knows what product is being made) recognises that the ingredient doesn't belong. This is a multi-billion dollar company... they could have the process live-streamed to 30 independent chemists each one watching the "chef's" every move (this is not a serious proposal). This stuff is all hashed out when one writes up a HACCP process.
- epistasis 6y ago
  Errors like this happen in every single GMP and clinical facility. What makes them different is 1) there are ample checks in place to prevent mistakes from ever leaving the facility, and 2) there is process that reduces the chances of having it happen in the first place. But all of these places will have incident logs of mistakes like this, along with corrective actions that change the procedures and make it less likely to happen a second time.
  - markdown 6y ago
    > Errors like this happen in every single GMP and clinical facility. They shouldn't happen in a 150yr old, multi-billion dollar pharmaceutical company. How hard is it to restrict the ingredients available to the "chef" to the ones needed for the current product? Not hard at all unless you want to save a few dollars cutting corners.
    - Kim_Bruning 6y ago
      Sounds like what epistasis is describing is a healthy QA environment with rapid feedback. Some other comments have already pointed out that 15M doses is probably only one or a few batches. Combined with the fact that the procedure documents for this particular line are probably still warm from the laser printer, I'd say let them iterate a bit more to get the kinks out. [ Procedures are not (should not be) engraved on stone tablets handed down from on high. If you try it that way, you are just as guaranteed to fail as when you had no procedures at all. ] As to what went wrong at this facility (besides overpromising?); we'll have to see. I wish there was such a thing as open source procedures!
    - epistasis 6y ago
      Being GMP isn't about eliminating all errors everywhere, it's about acknowledging the inevitability of error, minimizing error through conscious processes that stop those errors, and testing to assure that you made what you made. In a situation where maximizing vaccine production sooner saves lives, pinning this on saving a few dollars doesn't make sense. Whatever they were doing cost them quite a few dollars in addition to the lives lost from slower vaccination. I don't know their process, but suppose the "restricting the ingredients" phase happens. What if the ingredients were swapped at the time of restriction? Same problem happens.
- Johnny555 6y ago
  I think you mean it shouldn't happen undetected... these kinds of mistakes can and do happen. Everything from human error (oops, Billy put baking powder in the baking soda tank) to equipment error (the tank valve stuck open, adding 10X more of ingredient X than expected)

senectus1 6y ago

It'll be one vat with the potential for 15 million doses... the corrupted mix never reached the individual dose stage.

taeric 6y ago
I find it sad how quick the narrative jumps to the more absurd 15 million number. Obviously, the mistake is bad. But this is something with a very high fan out. And the smaller number just doesn't get the clicks. :(
- maxerickson 6y ago
  The batch time seems to be weeks/months, so there is some relevance to the dose count (it's wasted time in addition to the wasted ingredients). And it's put other production from the facility into question.
  - mc32 6y ago
    And there will be an FDA investigation which will put further delays into new production.
  - taeric 6y ago
    But this is a natural consequence of mistakes in processes with very high fan out. Obviously, it should be investigated such that were can reduce the chance is mistakes in the future. But piling on with blame will just convince folks not to try.
    - maxerickson 6y ago
      Given the $10 a dose revenue, I'd call it a failure in the process rather than a mistake. That it was possible is the bigger problem than it happening. And then there's the much greater value of the doses once they have been administered.
- Igelau 6y ago
  And it's J+J's name that goes in the headline. It wasn't even their fault. At least they didn't somehow cram the AZ connection in there too, though I'm sure other outlets are working on it.

deleted 6y ago

[deleted]

maxerickson 6y ago

Dose is 0.5 ml. Say the batch is equivalent in volume to the final doses (who knows, but it should be somewhere vaguely close). That's 7,500 liters. Microbrewers often use fermenters larger than that.

manquer 6y ago
The question should be more why would there have to be such large batches which can be SPOF for so many doses. Wouldn't it make more sense to make smaller seperate batches to derisk the process. This is akin to me using a large 64 core server for an app, instead of say 4x16 . Sure, the larger server can easily handle the load, however the 4x16 setup is more redundant and avoids many scenarios where all users will be out of service Just because there are high capacity machines capable of doing that much does not mean it should be used especially a With a new process trying to scale up
- tomaskafka 6y ago
  You might have smaller batches and smaller throughput, but some pesky people keep shouting that we need all the vaccines yesterday :).
- maxerickson 6y ago
  The production goal is hundreds of millions of doses this year, so I guess I wouldn't want to speculate too much about what batch size is sensible.
  - manquer 6y ago
    eventual goal maybe, Not for the month of March. 15M out of planned 20M ? . The scale out for any process starts small to try avoid this kind of issues. Yes fuck ups happen at plant and in planning, and yes we are not experts in the field to comment in detail on what is best approach. However if you are looking at it as an admin, clearly risk was not well managed.
- fragmede 6y ago
  To use the same alarmism and a software analogy: sw developer destroys thousands of hours of CPU time with bad code!1 Yet every single time I push, the CI pipeline runs an extensive suite of automated tests, and I'd not have it any other way. Wait but CPUs aren't being destroyed by running tests, you say. Yeah but niche reporting is always atrocious and a mainstream writer can't be trusted to get things entirely right. (If you're using a 4x16 server, whatever that is, to provide isolation in your system, you may want to talk to additional devops folks for other options.)

dcolkitt 6y ago

I'm guessing because you're not really making "15 million of something". The process is more like making one or two (or ten) batches, which is then sub-divided into 15 million doses. It'd be like the difference between burning dinner 7 nights in a row, versus making a single mistake during Meal Prep Sunday and ruining a weeks worth of dinners all at once. The latter is much more understandable.

justwalt 6y ago
It’s rare that an analogy is so easy to understand while remaining highly accurate. Nice.

HWR_14 6y ago

It's probably analogous to making a mistake in a single line that gets pushed to production - that then gets patched downstream into all 15 million customer's systems. Large systems moving in giant batches.

ergocoder 6y ago

It's not the most scrutinized thing on the planet with the emergence of pro-vaxxers (the anti anti-vaxxers) who refuses to acknowledge anything bad about vaccines.