6 ms·

Book Review: From Oversight To Overkill

149 points by apsec112 4y ago

ogennadi 4y ago
"Doctors are told to weigh the benefits vs. costs of every treatment. So what are the benefits and costs of IRBs? "Whitney can find five people who unexpectedly died from research in the past twenty-five years. ... "What are the costs? ...the monetary costs are around the order of $1.6 billion. "What about non-monetary costs? ... Low confidence estimate, but somewhere between 10,000 and 100,000 Americans probably die each year from IRB-related research delays. "So the cost-benefit calculation looks like - save a tiny handful of people per year, while killing 10,000 to 100,000 more, for a price tag of $1.6 billion. If this were a medication, I would not prescribe it."
scotty79 4y ago
I love how the word oversight has two opposite meanings.
- eru 4y ago
  You'll like https://en.wikipedia.org/wiki/Auto-antonym https://en.wikipedia.org/wiki/Auto-antonym then.
  - epgui 4y ago
    Side note: the French example “apprendre” listed in the article seems like it might not really belong in this category. On it’s own it always means “to learn”, and it is only when you use it in specific grammatical constructions, for example following indicative personal pronouns, that it takes the opposite meaning: “lui apprendre” or “leur apprendre” means “to teach him/her/them”. The meaning is not merely inferred from context, it is determined by grammar.
civilized 4y ago
> ... ethicist Jerry Menikoff [...] had decided maybe it was unethical to do RCTs on ventilator settings at all. He asked whether they might be able to give every patient the right setting while still doing the study. The study team tried to explain to him that they didn’t know which was the right setting, that was why they had to do the study. He wouldn’t budge. > Finally, after a year, another panel of experts ruled in favor of the investigators and gave them permission to restart the study right away. They did, but the delay was responsible for thousands of deaths, and produced a chain effect on ventilator research that made us less prepared for the surge in ventilator demand around COVID. People who do not understand or accept the role of risk-taking in the expansion of scientific knowledge and the betterment of the human condition should not be in positions of authority over the scientific process. I have to ask: what went wrong here? Why would such a person be in this position? Surely a judgment like this is exceptionally bad...? Or has the rot spread much farther, to the point where incompetence this severe is normal?
- AbrahamParangi 4y ago
  That science requires risk taking means that the more institutionalized it becomes, the less scientific it becomes as institutions are naturally allergic to risk.
  - biomcgary 4y ago
    Institutions are allergic to short-term, acute risks that result from their actions. They are insensitive to long-term, chronic risks that results from their inaction. Meta-risk, the risk that comes from not taking any risk, needs to be drilled into everyone's head: the public, journalists, politicians, IRBs, etc. I think the concept that not taking risks is risky should have its own label, but if someone doesn't like "meta-risk", maybe they will accept "risk tradeoffs".
    - HDMI_Cable 4y ago
      I think to some extent, we're just not naturally wired to think of meta-risk. We have innate biases against negative outcomes, but since risks—or surefire outcomes, like in Scott's post—are more abstract, we just don't naturally think about them. I think you're right though, we have to start teaching people that not taking a small risk can have a much larger impact.
      - makerdiety 4y ago
        To have a labor market that demands people who can perceive abstract issues and interconnected system failures requires, in the first place, a prerequisite population that has the cognitive ability to conceive big picture models which include risk contemplation as a decisive factor in organizational success. These intelligent representatives will not be working for free without indoctrination projects like the lifelong instilling of patriotism. So, realistically, any executive officer who can meet the requirements for a demanding large scale management labor will be selective in what administration he serves. Leaving those unfortunate enterprises that can't satisfy natural requirements left to fend for themselves with whatever unsustainable capabilities they can acquire or start up with. All this is broad, globally occurring competition for talent, labor, and resources and it provides an explanation for why things are the way they are. This comprehension on business management can lead to the conclusion that socialist visions will always have capitalist characteristics associated with their attempted executions. So it seems there's an extensive and influential capitalism that exists somewhere in the world, right now. The only way for socialist goals to prosper is to remove that wide-reaching capitalism. So just educating people about risk might actually, ironically and funnily, be an unproductive decision and indicative of a much more interesting problem.
armchairhacker 4y ago
> The IRB demanded that he give his patients consent forms warning that they could get smallpox. Dr. Knight tried to explain that smallpox had been extinct in the wild since the 1970s, the only remaining samples in US and Russian biosecurity labs... > ...the IRB demanded patients sign consent forms in pen (not pencil) but the psychiatric ward would only allow patients to have pencils (not pen)...
hirundo 4y ago
In '84 I had a college class in experimental psychology, and the culmination of that was to perform our own experiment, which had to clear the IRB. Mine involved measuring the student-subject's galvanic skin response and showing them photos of art. To calibrate that device the protocol was for the experimenter to slap their hands together a few feet away from the subject. That was too much for the IRB even 40 years ago. They did not allow the slap. Too traumatic I guess. It didn't really make a difference because it was a toy study for a class. But that was a great lesson in IRBs.
HDMI_Cable 4y ago
> Ezra Klein calls this “vetocracy”, rule by safety-focused bureaucrats whose mandate is to stop anything that might cause harm, with no consideration of the harm of stopping too many things. It’s worst in medicine, but everywhere else is catching up. This really scares me. It seems like every important facet of American life that can be touched by increased, predatory, and superfluous litigation and regulation has. If we take Scott's word for it, this 'vetocracy' costs 10 000 lives a year due to IRB over-diligence, makes Bay-area housing completely unaffordable, reduces the quality of education, and so much more. It really seems like our first priority as a nation should be to dismantle this 'vetocracy' and let researchers perform studies, and teachers teach.
- eru 4y ago
  > It really seems like our first priority as a nation should be to dismantle this 'vetocracy' and let researchers perform studies, and teachers teach. The housing is also really, really important: the Silicon Valley, San Francisco and New York etc are some of the most productive parts of your country. If more people were allowed to live there [0], your economy could grow a lot. A bigger economy directly translates to lives saved. To give a simple example: your highway engineering people work with a value of life of something like x million USD. Any improvement to safety that costs less than x million USD per life saved gets implemented. The value of x gets set in vague relation to how rich society is. [0] No any one individual is directly barred from moving to SF, if they pony up enough money for rent; but the NIMBY climate effectively sets a limit on the total number of people allowed to dwell there.
  - GauntletWizard 4y ago
    Location has almost nothing to do with the productivity of workers in Silicon Valley.
    - charliea0 4y ago
      Okay, but then why do folks move there? My impression is that it’s mostly for work.
jasonhansel 4y ago
[dead]
deleted 4y ago
[deleted]
bawolff 4y ago
Interesting. The thing that seems missing here is deeper comparison to other countries (there is only a brief mention of britian). Its presented as an american problem, so how does the american system compare to other systems? Do they have the same problems? Did everyone copy america?
niccl 4y ago
I think there are two things at work. First, if you pay someone to find problems with a study, they're going to want to find problems with the study to justify their existence. Associated with that is that many people don't consider the level of risk when they are considering whether something is or is not a problem. And both things don't just apply to IRBs. A friend told me about when his workplace had a security assessment (for Reasons). the assessor's only finding: that a houseplant by the door had sharp leaves and could be used by a malicious intruder as a weapon. Yes, I suppose so. Job done by security assesso. But, just perhaps, a malicious intruder might not choose to rely on being able to slash someone with the houseplant, in which case worse things might possibly happen.
- TedDoesntTalk 4y ago
  > if you pay someone to find problems with a study, they're going to want to find problems with the study to justify their existence. This is the same reason a home inspection report during a real estate sale ALWAYS finds problems with the property for sale
groby_b 4y ago
If Scott's wondering whom specifically to blame: Accretion of safeguards. It's inevitable, it seems. Something goes wrong, the process is amended to avoid that failure case. Something else goes wrong, add another amendment. All individual additions are likely net positive - positive in 99% of the cases, negative in the rest. Assuming events are mostly independent, you'll sooner or later are in a world were net positive outcomes are becoming less likely than net negative ones. You can sometimes sidestep that by combining/rewriting rules to some extent, but in the limit, you can only avoid all additional risk if you prevent any changes. That's not a fault of IRBs. That's a fault of trying to minimize risks. I'm not sure there is a good answer here outside of "here's the maximum amount of risk we're willing to take, and here are the few clear ethical boundaries we'll never overstep". There's a reason any place that's dealing with a lot of constraints is concerned with risk budgets, not risk elimination. And I'm not sure there's a lot of thinking on probabilistic ethics. (Or maybe I just can't find it)
TedDoesntTalk 4y ago
Balancing safety against freedom and responsibility. Let’s hope the pendulum swings like free-range parenting (freedom, responsibility) is having a moment over helicopter parenting (safety): https://www.freerangekids.com/ https://www.freerangekids.com/ https://en.m.wikipedia.org/wiki/Free-range_parenting https://en.m.wikipedia.org/wiki/Free-range_parenting
archon1410 4y ago
> Lawyers sue institutions every time they harm someone (but not when they fail to benefit someone) > pull the culture towards celebrating harm-avoidance as the greatest good, and cast suspicion on anyone who tries to add benefit-getting to the calculation Minor nitpick, but it is more accurate to say that they prefer prevention of new harms over the prevention of old harms. The current wording (plus the commentary over Gnosticism) kinda makes it sound like negative utilitarianism, or harm aversion (preferring the reduction of suffering over the addition of "positive pleasures") which would be a legitimate position. But the post is something more like "new risks aversion", i.e. preferring current harms over possible new harms regardless of the lower intensity of the "possible new harms", which sounds pretty irrational.
hoseja 4y ago
This is a big trolley problem, isn't it? Do studies that risk 2 deaths per decade, or do no studies and let tens of thousands die. That Hans Jonas quote is insane. I hate insane americans, why do they have such outsize influence on culture there (and through cultural imperialism everywhere else)?
- deleted 4y ago
  [deleted]
dsign 4y ago
Let’s engage in some speculative thinking: Somebody, let’s call them Dr. X, invents a computer algorithm that cures cancer. It works by running simulations of the patient’s cells. Its output is a list of diverse interventions, for example, pairs of RNA fragments and custom delivery mechanisms for each fragment. For a given patient, the algorithm outputs between 100 and 10000 interventions, all of them different and tailored to the patient. The simulation also takes into account possible adverse effects, it’s really a cure. With the current regulatory framework, does Dr. X has any hope of his invention being used at hospitals in less than a decade, or all he can do is to use it on himself and/or make the algorithm available in the dark web?
walkhour 4y ago
The fact that bureaucrats err on side of minimizing deaths caused by experiments is just a consequence of deaths being caused by experiments being unequivocally trivial to prove. Meanwhile finding out the number of deaths caused by extra caution and delaying the experiment requires a calculation, and then a very lengthy blog post to be communicated. Bureaucrats don't care about the second kind of deaths because they can hardly be linked to their actions. Now the first kind of deaths are directly their responsibility. And guess what, they can be avoided with a flick of a pen.
x-complexity 4y ago
(Internal anger: 150/100) Add another tally mark to "bureaucracies can only kill, never create". Try saying "accept what we cannot avoid" to my skewed feet, OHRP. I WISH I had the finances to fix them, but because of people like you, any advancements on cheap treatments for this issue will be met with hundreds of miles of red tape just to form a simple test. This quote sums up how I feel about the OHRP, and all bureaucracies similar to it: > A survey of how researchers feel about IRBs did include one person who said “I hope all those at OHRP [the bureaucracy in charge of IRBs] and the ethicists die of diseases that we could have made significant progress on if we had [the research materials IRBs are banning us from using]”.
ZeroGravitas 4y ago
It seems a common human failing to not be able to cope with rules that affect them. One you regularly see here is the complaint about Wikipedia: "I tried to edit the page about me/my work/my family and they wouldn't let me. But I'm the expert on me/my work/my family and it's currently wrong". Which sounds outrageous!. But at the same time "people can freely edit their own pages" has obvious failure modes too. Similarly, getting mental patients to sign in pen can be presented as pointless nonsense. Or it can be seen as part of a strategy to stop defenceless patients being taken advantage of by malicious doctors and stop vulnerable doctors being sued by malicious patients.

149 points by apsec112 4y ago

"Doctors are told to weigh the benefits vs. costs of every treatment. So what are the benefits and costs of IRBs? "Whitney can find five people who unexpectedly died from research in the past twenty-five years. ... "What are the costs? ...the monetary costs are around the order of $1.6 billion. "What about non-monetary costs? ... Low confidence estimate, but somewhere between 10,000 and 100,000 Americans probably die each year from IRB-related research delays. "So the cost-benefit calculation looks like - save a tiny handful of people per year, while killing 10,000 to 100,000 more, for a price tag of $1.6 billion. If this were a medication, I would not prescribe it."

scotty79 4y ago

I love how the word oversight has two opposite meanings.

eru 4y ago
You'll like https://en.wikipedia.org/wiki/Auto-antonym https://en.wikipedia.org/wiki/Auto-antonym then.
- epgui 4y ago
  Side note: the French example “apprendre” listed in the article seems like it might not really belong in this category. On it’s own it always means “to learn”, and it is only when you use it in specific grammatical constructions, for example following indicative personal pronouns, that it takes the opposite meaning: “lui apprendre” or “leur apprendre” means “to teach him/her/them”. The meaning is not merely inferred from context, it is determined by grammar.

civilized 4y ago

> ... ethicist Jerry Menikoff [...] had decided maybe it was unethical to do RCTs on ventilator settings at all. He asked whether they might be able to give every patient the right setting while still doing the study. The study team tried to explain to him that they didn’t know which was the right setting, that was why they had to do the study. He wouldn’t budge. > Finally, after a year, another panel of experts ruled in favor of the investigators and gave them permission to restart the study right away. They did, but the delay was responsible for thousands of deaths, and produced a chain effect on ventilator research that made us less prepared for the surge in ventilator demand around COVID. People who do not understand or accept the role of risk-taking in the expansion of scientific knowledge and the betterment of the human condition should not be in positions of authority over the scientific process. I have to ask: what went wrong here? Why would such a person be in this position? Surely a judgment like this is exceptionally bad...? Or has the rot spread much farther, to the point where incompetence this severe is normal?

AbrahamParangi 4y ago
That science requires risk taking means that the more institutionalized it becomes, the less scientific it becomes as institutions are naturally allergic to risk.
- biomcgary 4y ago
  Institutions are allergic to short-term, acute risks that result from their actions. They are insensitive to long-term, chronic risks that results from their inaction. Meta-risk, the risk that comes from not taking any risk, needs to be drilled into everyone's head: the public, journalists, politicians, IRBs, etc. I think the concept that not taking risks is risky should have its own label, but if someone doesn't like "meta-risk", maybe they will accept "risk tradeoffs".
  - HDMI_Cable 4y ago
    I think to some extent, we're just not naturally wired to think of meta-risk. We have innate biases against negative outcomes, but since risks—or surefire outcomes, like in Scott's post—are more abstract, we just don't naturally think about them. I think you're right though, we have to start teaching people that not taking a small risk can have a much larger impact.
    - makerdiety 4y ago
      To have a labor market that demands people who can perceive abstract issues and interconnected system failures requires, in the first place, a prerequisite population that has the cognitive ability to conceive big picture models which include risk contemplation as a decisive factor in organizational success. These intelligent representatives will not be working for free without indoctrination projects like the lifelong instilling of patriotism. So, realistically, any executive officer who can meet the requirements for a demanding large scale management labor will be selective in what administration he serves. Leaving those unfortunate enterprises that can't satisfy natural requirements left to fend for themselves with whatever unsustainable capabilities they can acquire or start up with. All this is broad, globally occurring competition for talent, labor, and resources and it provides an explanation for why things are the way they are. This comprehension on business management can lead to the conclusion that socialist visions will always have capitalist characteristics associated with their attempted executions. So it seems there's an extensive and influential capitalism that exists somewhere in the world, right now. The only way for socialist goals to prosper is to remove that wide-reaching capitalism. So just educating people about risk might actually, ironically and funnily, be an unproductive decision and indicative of a much more interesting problem.

armchairhacker 4y ago

> The IRB demanded that he give his patients consent forms warning that they could get smallpox. Dr. Knight tried to explain that smallpox had been extinct in the wild since the 1970s, the only remaining samples in US and Russian biosecurity labs... > ...the IRB demanded patients sign consent forms in pen (not pencil) but the psychiatric ward would only allow patients to have pencils (not pen)...

hirundo 4y ago

In '84 I had a college class in experimental psychology, and the culmination of that was to perform our own experiment, which had to clear the IRB. Mine involved measuring the student-subject's galvanic skin response and showing them photos of art. To calibrate that device the protocol was for the experimenter to slap their hands together a few feet away from the subject. That was too much for the IRB even 40 years ago. They did not allow the slap. Too traumatic I guess. It didn't really make a difference because it was a toy study for a class. But that was a great lesson in IRBs.

HDMI_Cable 4y ago

> Ezra Klein calls this “vetocracy”, rule by safety-focused bureaucrats whose mandate is to stop anything that might cause harm, with no consideration of the harm of stopping too many things. It’s worst in medicine, but everywhere else is catching up. This really scares me. It seems like every important facet of American life that can be touched by increased, predatory, and superfluous litigation and regulation has. If we take Scott's word for it, this 'vetocracy' costs 10 000 lives a year due to IRB over-diligence, makes Bay-area housing completely unaffordable, reduces the quality of education, and so much more. It really seems like our first priority as a nation should be to dismantle this 'vetocracy' and let researchers perform studies, and teachers teach.

eru 4y ago
> It really seems like our first priority as a nation should be to dismantle this 'vetocracy' and let researchers perform studies, and teachers teach. The housing is also really, really important: the Silicon Valley, San Francisco and New York etc are some of the most productive parts of your country. If more people were allowed to live there [0], your economy could grow a lot. A bigger economy directly translates to lives saved. To give a simple example: your highway engineering people work with a value of life of something like x million USD. Any improvement to safety that costs less than x million USD per life saved gets implemented. The value of x gets set in vague relation to how rich society is. [0] No any one individual is directly barred from moving to SF, if they pony up enough money for rent; but the NIMBY climate effectively sets a limit on the total number of people allowed to dwell there.
- GauntletWizard 4y ago
  Location has almost nothing to do with the productivity of workers in Silicon Valley.
  - charliea0 4y ago
    Okay, but then why do folks move there? My impression is that it’s mostly for work.

jasonhansel 4y ago

[dead]

deleted 4y ago

[deleted]

bawolff 4y ago

Interesting. The thing that seems missing here is deeper comparison to other countries (there is only a brief mention of britian). Its presented as an american problem, so how does the american system compare to other systems? Do they have the same problems? Did everyone copy america?

niccl 4y ago

I think there are two things at work. First, if you pay someone to find problems with a study, they're going to want to find problems with the study to justify their existence. Associated with that is that many people don't consider the level of risk when they are considering whether something is or is not a problem. And both things don't just apply to IRBs. A friend told me about when his workplace had a security assessment (for Reasons). the assessor's only finding: that a houseplant by the door had sharp leaves and could be used by a malicious intruder as a weapon. Yes, I suppose so. Job done by security assesso. But, just perhaps, a malicious intruder might not choose to rely on being able to slash someone with the houseplant, in which case worse things might possibly happen.

TedDoesntTalk 4y ago
> if you pay someone to find problems with a study, they're going to want to find problems with the study to justify their existence. This is the same reason a home inspection report during a real estate sale ALWAYS finds problems with the property for sale

groby_b 4y ago

If Scott's wondering whom specifically to blame: Accretion of safeguards. It's inevitable, it seems. Something goes wrong, the process is amended to avoid that failure case. Something else goes wrong, add another amendment. All individual additions are likely net positive - positive in 99% of the cases, negative in the rest. Assuming events are mostly independent, you'll sooner or later are in a world were net positive outcomes are becoming less likely than net negative ones. You can sometimes sidestep that by combining/rewriting rules to some extent, but in the limit, you can only avoid all additional risk if you prevent any changes. That's not a fault of IRBs. That's a fault of trying to minimize risks. I'm not sure there is a good answer here outside of "here's the maximum amount of risk we're willing to take, and here are the few clear ethical boundaries we'll never overstep". There's a reason any place that's dealing with a lot of constraints is concerned with risk budgets, not risk elimination. And I'm not sure there's a lot of thinking on probabilistic ethics. (Or maybe I just can't find it)

TedDoesntTalk 4y ago

Balancing safety against freedom and responsibility. Let’s hope the pendulum swings like free-range parenting (freedom, responsibility) is having a moment over helicopter parenting (safety): https://www.freerangekids.com/ https://www.freerangekids.com/ https://en.m.wikipedia.org/wiki/Free-range_parenting https://en.m.wikipedia.org/wiki/Free-range_parenting

archon1410 4y ago

> Lawyers sue institutions every time they harm someone (but not when they fail to benefit someone) > pull the culture towards celebrating harm-avoidance as the greatest good, and cast suspicion on anyone who tries to add benefit-getting to the calculation Minor nitpick, but it is more accurate to say that they prefer prevention of new harms over the prevention of old harms. The current wording (plus the commentary over Gnosticism) kinda makes it sound like negative utilitarianism, or harm aversion (preferring the reduction of suffering over the addition of "positive pleasures") which would be a legitimate position. But the post is something more like "new risks aversion", i.e. preferring current harms over possible new harms regardless of the lower intensity of the "possible new harms", which sounds pretty irrational.

hoseja 4y ago

This is a big trolley problem, isn't it? Do studies that risk 2 deaths per decade, or do no studies and let tens of thousands die. That Hans Jonas quote is insane. I hate insane americans, why do they have such outsize influence on culture there (and through cultural imperialism everywhere else)?

deleted 4y ago
[deleted]

dsign 4y ago

Let’s engage in some speculative thinking: Somebody, let’s call them Dr. X, invents a computer algorithm that cures cancer. It works by running simulations of the patient’s cells. Its output is a list of diverse interventions, for example, pairs of RNA fragments and custom delivery mechanisms for each fragment. For a given patient, the algorithm outputs between 100 and 10000 interventions, all of them different and tailored to the patient. The simulation also takes into account possible adverse effects, it’s really a cure. With the current regulatory framework, does Dr. X has any hope of his invention being used at hospitals in less than a decade, or all he can do is to use it on himself and/or make the algorithm available in the dark web?

walkhour 4y ago

The fact that bureaucrats err on side of minimizing deaths caused by experiments is just a consequence of deaths being caused by experiments being unequivocally trivial to prove. Meanwhile finding out the number of deaths caused by extra caution and delaying the experiment requires a calculation, and then a very lengthy blog post to be communicated. Bureaucrats don't care about the second kind of deaths because they can hardly be linked to their actions. Now the first kind of deaths are directly their responsibility. And guess what, they can be avoided with a flick of a pen.

x-complexity 4y ago

(Internal anger: 150/100) Add another tally mark to "bureaucracies can only kill, never create". Try saying "accept what we cannot avoid" to my skewed feet, OHRP. I WISH I had the finances to fix them, but because of people like you, any advancements on cheap treatments for this issue will be met with hundreds of miles of red tape just to form a simple test. This quote sums up how I feel about the OHRP, and all bureaucracies similar to it: > A survey of how researchers feel about IRBs did include one person who said “I hope all those at OHRP [the bureaucracy in charge of IRBs] and the ethicists die of diseases that we could have made significant progress on if we had [the research materials IRBs are banning us from using]”.

ZeroGravitas 4y ago

It seems a common human failing to not be able to cope with rules that affect them. One you regularly see here is the complaint about Wikipedia: "I tried to edit the page about me/my work/my family and they wouldn't let me. But I'm the expert on me/my work/my family and it's currently wrong". Which sounds outrageous!. But at the same time "people can freely edit their own pages" has obvious failure modes too. Similarly, getting mental patients to sign in pen can be presented as pointless nonsense. Or it can be seen as part of a strategy to stop defenceless patients being taken advantage of by malicious doctors and stop vulnerable doctors being sued by malicious patients.