17 ms·

My Institutional Review Board Nightmare

304 points by apsec112 9y ago

maxxxxx 9y ago
Mandatory IRB story: http://www.npr.org/templates/story/story.php?storyId=102393652 http://www.npr.org/templates/story/story.php?storyId=1023936...
waisbrot 9y ago
It sounds like the study was poorly designed. Having an actual PI would have helped. The IRB probably would have seemed a lot less painful with a better-designed study and an engaged PI.
- civilian 9y ago
  You're probably getting downvoted because you're making a casual armchair judgement and this is Scott fucking Alexander we're talking about. //edit: I only just got to this section of the article, but it feels good to include as a counter-example. > During that year, Dr. W and I worked together on two less ambitious studies, carefully designed not to require any contact with the IRB. One was a case report, the other used publicly available data. > They won 1st and 2nd prize at a regional research competition. I got some nice certificates for my wall and a little prize money. I went on to present one of them at the national meeting of the American Psychiatric Association, a friend helped me write it up formally, and it was recently accepted for publication by a medium-tier journal. > I say this not to boast, but to protest that I’m not as much of a loser as my story probably makes me sound. I’m capable of doing research, I think I have something to contribute to Science. I still think the bipolar screening test sucks, and I still think that patients are being harmed by people’s reliance on it. I still think somebody should look into it and publish the results.
  - umanwizard 9y ago
    Scott "fucking" Alexander is a blogger, not some sort of god immune to criticism. He's also not an expert authority on, as far as I can tell, anything but psychiatry.
  - tallanvor 9y ago
    Sorry, but this is a real question: Why would we know who Scott Alexander is?
    - civilian 9y ago
      He's written a lot of great and influential things, and I feel like he's entering the "nerd/intellectual" common knowledge. The biggest example for me is Meditations on Moloch, which is the only thing that's ever made me deeply question my libertarianism. http://slatestarcodex.com/2014/07/30/meditations-on-moloch/ http://slatestarcodex.com/2014/07/30/meditations-on-moloch/ "I can tolerate anything except the outgroup" is a depressing piece on American politics. http://slatestarcodex.com/2014/09/30/i-can-tolerate-anything-except-the-outgroup/ http://slatestarcodex.com/2014/09/30/i-can-tolerate-anything... My original comment is not the best, it was a kneejerk reaction. I'm not representing the fandom well. :-/
- SubiculumCode 9y ago
  IRB stuff can be painful, but in my experience submitting these through the process is a lot less problematic than it was described in the article. An experienced PI knows how to navigate these issues. Practically, there are two issues with which the IRB is concerned. 1) Protection of particpants 2) Protection of the Institution from litigation by participants. Realizing the second oft unspoken goal can help you manage the IRB process
deleted 9y ago
[deleted]
karmajunkie 9y ago
I thought this was going to be about Ruby...
- civilian 9y ago
  Hah, I think your joke got stomped by a mod re-writing the headline. :]
  - karmajunkie 9y ago
    yeah, they probably saw that coming a mile away... :)
jlarocco 9y ago
Poor choice of headline for this site, IMO. I can't be the only one who clicked expecting something about the Ruby REPL. For the benefit of anybody else confused by the title, IRB in this article refers to an "Institutional Review Board" https://en.wikipedia.org/wiki/Institutional_review_board https://en.wikipedia.org/wiki/Institutional_review_board
hirsin 9y ago
My freshman year of college I proposed a study to our hospitals IRB to strap small lasers to three week old infants in an effort to measure concentrations of a chemical in their blood. The most frustrating part was not the arcane insistence on ink and bolded study names, but the hardline insistence that it was impossible (illegal) to test the device before getting IRB approval - even on ourselves. Meaning that without any calibration or testing, our initial study would likely come back with poor results or be a dud, but we couldn't find out until we filled out all the paperwork. Incidentally, I recall it being much less painful than this, at least to get to the approval step. I would expect that each IRB varies greatly from group to group.
- jpatokal 9y ago
  What stopped you from testing on yourselves anyway outside the study?
  - hirsin 9y ago
    Our PI was perhaps stricter than was useful. I believe it was intended as a teaching tool to force us to deal with the irb
omegaworks 9y ago
A take from the opposite end of the spectrum (ethically and logistically): https://arstechnica.com/science/2017/08/bucking-fda-peter-thiel-funds-patently-unethical-herpes-vaccine-trial/ https://arstechnica.com/science/2017/08/bucking-fda-peter-th... I wish the author took more seriously the warnings around patient health and safety. Yes, they may not apply to his particular project, but that doesn't make asking about them useless. To take past abuses lightly undermines his overall point. It's done precisely to avoid a situation like the way he delegated responsibility to his PI: "Dr. W had vaguely hoped that I was taking care of it. I had vaguely hoped that Dr. W was taking care of it." The review board shouldn't make vague assumptions about something as critical and necessary as ethical study design.
- apsec112 9y ago
  There are very serious costs to excessive safety regulation. When it makes drug development more expensive, marginal drugs don't get put through trials, and those drugs then never reach patients. Rational analysis means weighing the cost of each additional safety rule against the benefit. In many cases, there's a lot of evidence that the benefits of the current IRB process are trivial or nonexistent, and the costs are large.
- ScottAlexander 9y ago
  I take the possibility of serious ethical violations...well, seriously. I'm not sure whether the global cost of IRBs outweighs the benefits. But I'm pretty sure that the sort of regulatory changes OHRP proposed (mentioned in the linked NYT article), where studies like mine with minimal possible risk are exempted from IRB requirements, seem good and important. I also wonder whether it's possible to monitor even the scary studies with lots of potential risks - like the herpes vaccine trial you link to - with a little more common sense and a little bit less insistence on everything being signed in pen.
  - zaroth 9y ago
    BTW, welcome [back] to HN. And just wanted to say, I am immensely enjoying very many of your articles and glad to see them making the front page!
paultopia 9y ago
It's kinda amazing any research gets done at all, with all the bureaucracy. My university requires official approval and a full process even to apply for a small non-institutional grant. I was going to apply for that tiny little AI Grant that was posted on here a while back, then I concluded that the rules probably would require me to get approval to apply for that. So I said "fuck it."
terminado 9y ago
“YOU LET PEOPLE SIGN CONSENT FORMS IN PENCIL, HOW CAN YOU JUSTIFY THAT?!” The funny thing is, contracts are legally binding agreements, whether they are verbal, scribbled on a napkin, signed in pencil, pen or blood, and whether the signature is an easily forged 'X' or elaborate calligraphy. Consent forms are a slightly different use-case, but I think the premise is the same: to show a meeting of the minds. Knowing this, the pencil signature actually protects the patient more than the institution, since the patient could theoretically forge a revocation of consent by erasing their mark. Meanwhile, ink isn't as binding, without some sort of objective third-party official (e.g. a notary public, beholden neither to patient nor to institution) serving as formal witness, further validating a non-forgery, in the case of a simple 'X' or erasure thereof. In practical terms, one solution to render the pencil signature binding would be to create a tamper-evident photocopy, with fraud-marks like a UUID moire pattern sprayed by an ordinary desktop printer, for each serialized document, readily detectable in the photocopy but permanently printed into the background of the form (e.g. yes, this is a photocopy of the same piece of paper, and the forger failed to notice the lightly tinted pattern, when erasing their signature), and retain the pencil copy, to ensure a match. In the event of an erasure, the paper might show an abrasion, and the photocopy would act as a snapshot/backup. Meanwhile, it should be possible to access existing, archived, historical questionnaires and actual clinical diagnoses from normal records, in a de-identified fashion, and compare results, no? Am I misunderstanding the goal of the study? Didn't he want to compare the assessment of the survey, with actual patient dispositions, and leave the rest of all treatment untouched, undisturbed and otherwise normal? Ask the questions, do the rest, same as ever, but analyze the opinion formed by the score of the answers to the questions, with the opinion of the professional caregiver beyond the scope of the simple Q&A exam?
SilasX 9y ago
I can't tell how much of this is exaggerated and how much really happened. Sometimes I have a hard time parsing the difference. They demand that you reveal what the study is on and sign in pen? Because the Nazis didn't disclose that stuff and get signatures?
- tedunangst 9y ago
  That's kinda the point. We need rules because Nazis (and Tuskegee, etc.) but it's not clear we need these rules about pen vs pencil.
  - vacri 9y ago
    The pen vs pencil thing is a total canard by the author, who is looking to 'dress-up' his indignation. Hand the form to the subject. Hand a pen to the subject. Subject reads and signs the form. Take the form from the subject. Take the pen from the subject. Job done. It's not like the patient will go bonkers stabbing-crazy the millisecond their flesh touches a pen.
- bshlgrs 9y ago
  The first part is real, and the second part is the joke.
ak_yo 9y ago
The author doesn't seem to take the history of medical research ethics (e.g. the Tuskegee syphilis experiments [1]) seriously. Having a sophisticated understanding of the ethical risks of research is a huge part of doing good science, and this post certainly doesn't communicate a high level of sophistication. If the author thinks that ethics review is "Blindly trusting authority to make our ethical decisions for us", then he clearly doesn't understand what ethics review is or how it works. IRBs have well-documented flaws (for a more comprehensive look, read Laura Stark's book on IRBs [2]) but "IRBs are bad because they say I can't do whatever I want even though it's obviously the right way to do it" is not one of them. [1]: https://en.wikipedia.org/wiki/Tuskegee_syphilis_experiment https://en.wikipedia.org/wiki/Tuskegee_syphilis_experiment [2]: http://press.uchicago.edu/ucp/books/book/chicago/B/bo12182576.html http://press.uchicago.edu/ucp/books/book/chicago/B/bo1218257...
- harryh 9y ago
  The IRB process he described did not have a sophisticated understanding of the ethical risks of research. It imposed arbitrary and useless rules on an experiment with few if any risks.
  - ak_yo 9y ago
    IRB review procedures are flawed, I agree. But ethical review by a third party is enormously important for research with human subjects. I'm alarmed by the notion of researchers deciding unilaterally that a study has only "nonexistent risks" -- that's a recipe for disastrously unethical scientific conduct.
    - harryh 9y ago
      I don't think the author was proposing getting rid of oversight completely, just that said oversight should, you know, make sense.
    - mannykannot 9y ago
      The point is that the decision should not have been unilateral - the author and his colleague had a very good case for their position, based on the fact that they were not asking anything that was not being asked anyway, and not recording anything that was not being recorded anyway. It was mindless dogma on the part of the IRB that made it unilateral. Meanwhile, a practice with a real potential for risk went unstudied. You might argue that there was risk to the patients in the one part of the procedure that was different than what they would experience anyway - the actual asking for consent - but if so, what would you propose doing about it? You have made a straw man out of this one case, and are defending it with all the dogma displayed by this IRB.
tom_b 9y ago
And it's only worse at the staff level. Even getting patient data variables for patients consented to IRB-approved studies is hit-or-miss. Groups build barriers by wanting to "work with" principal investigators (PIs) instead of providing data to PIs and study staff. Data moats are everywhere. My own group (at a research center attached to a university school of medicine) has switched our focus from helping PIs with clinical data management and applications to helping manage research samples and sample processing, mainly by doing housekeeping around genomic processing across different lab teams. It's kind of hard to convey just how much I would warn devs away from healthcare and healthcare research IT gigs.
88e282102ae2e5b 9y ago
The initial meeting with the IRB where they proposed ridiculous and irrational changes to the consent forms was strikingly reminiscent of every conversation I've had in a third world country where someone was asking me for a bribe, but couldn't say it directly.
- gwern 9y ago
  I've noticed how often big projects will allocate something like 5% of their budget for a few things like 'bioethicists' and 'community' grants; you never seem to hear about what they produce or what important findings they make, but the money does get spent...
  - sorokod 9y ago
    Money always gets spent. It is in its nature.
deleted 9y ago
[deleted]
schoen 9y ago
I thought I remembered a proposal to create some new categorical exemptions from IRB review, one of which was somehow related to studies where you just talk to people. (I've been interviewed by several social scientists and very often their "protocols", consisting exclusively of informal interviews, had clearly been through extensive IRB reviews, which felt super-odd because journalists wouldn't need any ethical review at all in order to carry out exactly the same interview in exactly the same way.) On the other hand, you clearly can harm people in various ways by or as a result of interviewing them, for example by breaching confidentiality, by being forced to breach confidentiality, or by making them feel bad about themselves by insulting them during the interview or just by bringing up painful and distressing memories. And also it's hard for researchers who aren't doctors to extend the same level of legal protection to what their research subjects tell them: https://www.socialsciencespace.com/2017/01/social-science-needs-doctor-patient-confidentiality/ https://www.socialsciencespace.com/2017/01/social-science-ne... So maybe one idea would be to create a categorical IRB exemption for studies that combine non-invasive observations, or existing datasets, with interviews not reasonably expected to be distressing to the subjects, and conducted subject to otherwise existing norms on patient confidentiality and medical privacy? (recognizing that limited legal protection for that confidentiality still poses problems for some studies) (I'm certainly happy to apply the Chesterton's fence principle and learn more about the history of unethical experimentation before seriously advocating this.)
- munin 9y ago
  > which felt super-odd because journalists wouldn't need any ethical review at all in order to carry out exactly the same interview in exactly the same way. Journalists don't have a history of injecting you with syphilis just to see what happens, though.
  - schoen 9y ago
    Neither do anthropologists or computer scientists, who are both now subject to IRB review in universities.
    - 9y ago
mjevans 9y ago
I suspect that the proper ethical thing to do depends on the risks to the patients, and the benefits to the subjects and overall population. In my own opinion the author is quite correct in classifying the medical benefits to the subjects (practically speaking, zero; equivalent to talking with any random untrained individual), and to future patients (potentially great benefits). The risks seem to be much more closely associated with accidentally exposing patient information. This is the section that their study protocols (at least what was described in overview) need review and revision. The unblinding documentation should be kept in a secured, designated area. A safe with a very limited ACL would be ideal. Unblinding information could be either the consent forms with the unique non-PII ID information on them or a list for matching unique non-PII IDs to (hard copies of) existing patient record stubs attached to the consent forms. If possible the study should have actually been designed to use as much of the already collected data as possible, and possibly ask the small additional information of all, but not record the answers (or destroy the collected information if they do not consent). Hypothetically they could be asked if they consent anonymously providing some of the questions they were already asked to an informational medical research study. If you ask them for consent AFTER having asked (and gotten answers on) the questions, then you're even actually safe to tell them which data are being collected (what their answers were as well) and what you're attempting to study. I believe that the above is the most ethical way of designing this type of study, and that it sounds similar to a term I've heard professionals in this field use 'retrospective study' (collecting study data based on previously collected medical data).
- harryh 9y ago
  Why should study data subject to higher security standards than information that is collected on the patient during the process of providing care that is much more sensitive?
  - mbreese 9y ago
    All PHI is subject to the same data security standards, regardless of if it is from research or clinical sources.
xr4ti 9y ago
I can't agree. The authors wounds at every stage in the narrative were self inflicted. (1) The study title doesn't work at the top of the form? Change it! Surely there is some other arrangement of English words that would satisfy both the author and IRB. (2) They leave the risk section blank because minimal risk, and then complain about the requirement for pens because the patients might stab themselves if given pens. Looks to me like they didn't spend much effort thinking about risks to the patient. If they had just mentioned the pen stabby thing as a risk factor, and given "don't use pens" as a risk minimization plan, they wouldn't have had a problem. (3) They poopoo the training because Hitler, and then get upset when they unknowingly violate a personnel rule that is most definitely covered in any decent IRB/GCP training. I could go on, but every new thing I enumerate makes me angrier. Life is too short. Look. IRB applications suck, but human experimentation is serious business. The rules aren't there to make the study easy, the are there to (a) look out for the patient (respect), (b) evaluate risk and benefits of the study (beneficence), and (c) ensure that the benefits are distributed to the group bearing the risk (justice).
- harryh 9y ago
  From reading the post it's not at all clear to me that 2 of the 3 goals you list were accomplished by the rules. A) Instead of actually looking out of the patient a list of arbitrary rules were imposed. B) The risk (minimal) was not properly evaluated but instead massively overstated. Given this failure of the process to meet said goals it seems perfectly reasonable to question the process.
  - xr4ti 9y ago
    (A) The rules are not arbitrary. They are the product of much careful deliberation. https://en.wikipedia.org/wiki/Belmont_Report https://en.wikipedia.org/wiki/Belmont_Report (B) Is experimentation on suicidal patients really minimal risk? Maybe, maybe not. But failure of the investigator to show they have thought about it is a huge red flag.
    - harryh 9y ago
      (A) They were arbitrarily imposed in this case. Just because a rule is a good idea doesn't mean it is a good idea all the time. (B) We're not talking about injecting the patients with some experimental drug here. We're talking about asking them questions that they were already being asked anyways. The lack of risk is self evident to anyone who considers the question.
cassowary37 9y ago
So, to recap: you tried to do a human subjects study without a) getting help from someone with prior research experience and b) understanding all the details required to do such a study. The IRB's responsibility (and yours, as the PI) is protection of human subjects. That you're so cavalier about this responsibility and the history behind these requirements suggests you should probably have paid more attention to that course you berate.
- tertius 9y ago
  Or collaborated with someone who cares enough.
trhway 9y ago
>I knew exactly how to handle this: one by one, I sought out the laziest attendings in the hospital and asked “Hey, would you like to have your name on a study as Principal Investigator for free while I do all the actual work?” Yet one by one, all of the doctors refused, as if I was offering them some kind of plague basket full of vermin. when you find yourself in the situation of being the only smart guy surrounded by stupid morons (nevermind the fact that these morons are more highly trained and experienced) ... A related Russian joke. While driving, the husband calls his wife and she tells him that it is on the news right now that some moron is driving in the wrong direction on the highway. The husband exclaims in response "A moron?! There are hundreds of such morons driving in the wrong direction here!"
paulsutter 9y ago
Here's a story about sidestepping IRB processes by going offshore http://khn.org/news/offshore-rush-for-herpes-vaccine-roils-debate-over-u-s-safety-rules/ http://khn.org/news/offshore-rush-for-herpes-vaccine-roils-d... "Neither the Food and Drug Administration nor a safety panel known as an institutional review board, or an IRB, monitored the testing of a vaccine its creators say prevents herpes outbreaks. Most of the 20 participants were Americans with herpes who were flown to the island several times to be vaccinated, according to Rational Vaccines, the company that oversaw the trial."
- tim333 9y ago
  Discussed on HN yesterday https://news.ycombinator.com/item?id=15126557 https://news.ycombinator.com/item?id=15126557 As the people doing that one argue, there is an argument for balancing the risks of the test against the risk to suffers of illness and the like of not doing the tests.
Nition 9y ago
Someone on Reddit says[1]: > Amateur. What you do is you sweet talk the clinicians into using their medical judgement to adopt the form as part of their routine clinical practice and get them to include it as part of the patient's medical records. Later... you approach the IRB for a retrospective chart review study and get blessed with waived consent. Bonus: very likely to also get expedited review. [1] https://www.reddit.com/r/slatestarcodex/comments/6wtylk/my_irb_nightmare/dmas5l5/ https://www.reddit.com/r/slatestarcodex/comments/6wtylk/my_i...
- j88439h84 9y ago
  What form?
  - haltingthoughts 9y ago
    The author was trying to test whether a simple common test for bipolar was accurate. Presumably, the suggestion is to make the test a form filled out by each doctor for each patient. Then months or years later the author could do a chart review and see whether the test results were accurate.
    - nohat 9y ago
      To potentially clarify: the test is just a form. It's a form with some questions about mood, and is currently being used to diagnose bipolar disorder (inappropriately, the author believes).
- kerbalspacepro 9y ago
  That sounds more unethical though?
smithkl42 9y ago
Commenters have mentioned "Chesterton's Fence". I'd add a reference to another of GKC's observations: "When you break the big laws, you do not get liberty; you do not even get anarchy. You get the small laws."
- penpapersw 9y ago
  I can't help but wonder if the Fence comments had anything to do with my blog post last night. Kind of like watching social osmosis in action :D
  - slavik81 9y ago
    Chesterton's fence has long been a staple of HN discussions. Unfortunately, it involves no risk/reward analysis and it invariably leads to needless conservatism. That's probably not a big deal in medical research where risks are often high, but it's a problem when applied as a general principle. (The opposite wisdom is probably grandma's ham [1]. Consider that if grandma had passed away, they might never be allowed to try cooking the whole ham.) I quite enjoyed that blog post, btw. [1]: http://www.angelfire.com/ma/artemis9/humor/joke7.html http://www.angelfire.com/ma/artemis9/humor/joke7.html
    - throwanem 9y ago
      The point of Chesterton's fence is cost/benefit analysis. It's not saying the fence mustn't go; it's saying you need to know why the fence was put there in order to evaluate whether it's best removed or left in place - that is, your analysis needs to be informed by understanding of the status quo in order to be accurate. Grandma's ham is a great example. It might have been some subtle trichinosis-related failure case that cutting the end off a ham reliably prevents, and it's a shame we had to lose your great-uncle Joe to find out about that. Or it might have been an issue of pan length. You can't know until you ask, so you ask if you can. If you can't ask, then sure, you do the best you can with what you have - I should like to hope there are no blind dogmatists here. But if you can ask, you'd be a fool not to.
ryandrake 9y ago
> Faced with someone even more obsessive and bureaucratic than they were, the IRB backed down and gave us preliminary permission to start our study. I love it! You can’t fight mindless rule followers with logic or reason. You need to use even more detailed rule-following. This is true in every bureaucracy where checklists have replaced good judgment.
chi17 9y ago
Man. This sounds just like my workplace.
epmaybe 9y ago
I'm in the process of getting two IRBs approved. I am somewhere in the first quarter of that author's story. I agree that it's a big hurdle to get past, and writing a good consent form is hard. All of this being said, though, it was/is a great experience to have under my belt. And, next time, I'll be able to tackle the regulatory hurdles with far less confusion.
alexandercrohde 9y ago
For those who think the author is in the wrong 1 - Do you think this story indicates a success or a failure of the IRB? 2 - Would you rather a better expedited review process AND more studies (remember the personal effort cost killed this study), neither, or do you see some 3rd option?
- icelancer 9y ago
  >>Do you think this story indicates a success or a failure of the IRB? Failure of the IRB but mostly failure of the author to explore his options. >>Would you rather a better expedited review process AND more studies (remember the personal effort cost killed this study), neither, or do you see some 3rd option? As I've posted in this thread, this problem is solved. Private IRBs exist that are much more responsive.
  - tedunangst 9y ago
    What options do you think were inadequately explored to sign the consent forms? Finger paint?
    - icelancer 9y ago
      >>What options do you think were inadequately explored to sign the consent forms? None. Now, if you asked a question without assuming what you think I was saying, I'd tell you that Alexander's other options were to not use the IRB that he went to, and to explore the private marketplace of IRB organizations that provide much better service than the nightmare he dealt with - a very common practice in all fields.
      - mattkrause 9y ago
        > Alexander's other options were to not use the IRB that he went to This is, frankly, ridiculous. The 50-70% overhead the university takes from every grant already pays for one IRB. If that one isn't working, the solution is not to tap in (almost inevitably tight) research funds to pay for a second for-profit one instead. Given how hard it is to charge things like computers to grants, I'm actually amazed this is even allowed.
yk 9y ago
Well my story with the research bureaucracy is, a coworker had to fill out a notice that he got some money for a conference from outside the university and he dutifully filled out everything. Thing is, as a theoretical physicist you are supposed to fill out the first page, check the box on page 27, that there is not conflict of interest, and sign the thing. Any answer on certain questions will trigger a public hearing by some very obscure board. Public hearing means, that poor guy has to go there in case the public has questions. He filled out everything, so he had to go to the anti-corruption board and the nuclear safety commission and so on. After roughly three weeks of him whining that he always sits there like an idiot, I got curious and came along to one of the hearings, definitely for some reason other than watching him sit there like an idiot, or so I claim. That was the animal cruelty (against) commission and at the hearing were the committee, a very nervous working group, one unhappy postdoc and a grad student who just wants to watch the show. The PI of the working group then gave a presentation on the sad reality, that you can't study pain receptors if you anesthetize the mouse. (Apparently that was not the first time they were there, so I got just the part on why they are certain that there is really no way to get a statistical meaningful result with less mice.) Really drove home the point that these procedures are not designed for the convenience of theoretical physicist, but are designed to tackle really hard problems.
- hawkice 9y ago
  If the ethical problems of rat suffering are really hard problems, why is it utterly mundane for people to buy products at home depot to deliberately poison them for days and days until they succumb to internal bleeding, for no other reason then "it was bothering me, existing in my house"? Perhaps we should or could start caring, but we definitely do not even acknowledge there being a meaningful question there in any other element of human life.
  - munin 9y ago
    Fortunately for us, the IRB only governs human experimentation. You need very little approval to experiment on animals, in the US at least.
icelancer 9y ago
Much of this reminds me of what I have been going through. What SSC should have investigated and wrote up is that hospital and medical IRB is fraught with bureaucracy and that a private IRB industry has developed, entities inside that handle wide ranges of studies and turn around informed consent docs and things very fast (within a week). Our first IC forms got banged multiple times for the same reasons his did. And it’s annoying as shit. But my ethical training focused not on something stupid like Nazis, but something real like American scientists unethically hiding treatment from diseases they were studying, like the Tuskegee Syphilis Experiment (the actual reason IRB exists as is). Or the Milgram Experiment. Etc. A lot of IRB review is stupid. But messing with human subjects doesn’t have a good track record in this country. Our predecessors earned every bit of bureaucracy and oversight based on their absurd experiments. Such is life.
- SilasX 9y ago
  That sounds, to me, eerily close to the mentalities that Alexander rightly criticizes: - "It's no big deal because you can just pay off someone's who's in the know and can work the system rather than read the rules at face value." - "No amount of kafkaesque absurdity is too much, because Nazis."
  - icelancer 9y ago
    What specifically sounds like that? I cannot decipher the context you are using those quotes in regarding my comment.
    - SilasX 9y ago
      For the first one: >"What SSC should have investigated and wrote up is that hospital and medical IRB is fraught with bureaucracy and that a private IRB industry has developed, entities inside that handle wide ranges of studies and turn around informed consent docs and things very fast (within a week). That is, even though Alexander had a long wait time just to hear responses, there are expensive services you can use that make all that wait time and pushback from hidden rules disappear, so you don't see it as a problem. For the second one: >A lot of IRB review is stupid. But messing with human subjects doesn’t have a good track record in this country. Our predecessors earned every bit of bureaucracy and oversight based on their absurd experiments. Such is life. You don't show a standard for the upper bound on when the absurdity is too much, dismissing any such concerns with appeal to some historical atrocities.
Glyptodon 9y ago
Maybe he should have called it a care audit and made it look like they were auditing the doctors success at administering screenings instead of doing any kind of patient research.
michaelbuckbee 9y ago
A separate discussion of this happening in /r/slatestarcodex Top comment there right now: "Amateur. What you do is you sweet talk the clinicians into using their medical judgement to adopt the form as part of their routine clinical practice and get them to include it as part of the patient's medical records. Later... you approach the IRB for a retrospective chart review study and get blessed with waived consent. Bonus: very likely to also get expedited review." Which is an extremely clever "hack" (in the way we like to use it on HN) of the IRB setup.
JonasJSchreiber 9y ago
I got up to the part where the author talks about including the test results in the patient's file. Up to there, every point of contention either seems whiny or completely valid on the institution's part.
alexashka 9y ago
The real nightmare is having read through 3/4 of this, only to realize it's the longest bitchy complaint I've ever read through. No substance and a lot of arrogance. The more curious take-away is that these extremely long winded rants of his have younger people fooled as being worthwhile. If you have something of value to say - you distill it, condense it, so as to ease the mental effort required for someone else to understand the point you're trying to make. It reminds me of what philosophers do - they exhaust you into thinking they're making some sophisticated point simply because they're using fancy words that produce the thought of 'am I missing something? Surely no one is spouting complete nonsense for 400 pages, it must be me'. It isn't you. It's them on a massive ego trip - this guy being a great example.
- sctb 9y ago
  A more charitable interpretation might be that a detailed and clinical account of the absurdity of the most tedious bureaucracy may have made for rather... dry reading.
  - alexashka 9y ago
    I can summarize what he's said in the 3/4s that I've read, down to 2 or 3 paragraphs, with a few bullet points. The author complains about tedious procedure, while writing the most tedious blog I've had the displeasure of reading. The obliviousness and callous arrogance of the author, combined with cult-like reverence some people seem to have for his babbling is what makes this at all worth commenting on.
    - frooxie 9y ago
      I don't want it summarized; every paragraph is funny and interesting as is. You, on the other hand, seem oblivious and arrogant.
vacri 9y ago
The author complains about there not being enough of an ethics gauntlet with the bipolar test in question, and then complains of having to run an ethics gauntlet to do the study. The author keeps on bringing up nazis and suggests he knows about study methods, when it's clear he's pretty clueless about doing studies. Several of the "this is stupid" questions really aren't that stupid, given that these are generic forms for any study. Why the fuck do you care if you're being asked if you're removing organs? Just tick 'no' and move on to the next question. Does the author also complain that other genders are selectable on online forms, and not just his own gender - after all, it's equally useless information? The setup to the story is: 'waaah, this application form isn't tailored exclusively to MY study'. I ran into problems with ethics committees myself when doing an (aborted) PhD that caused significant needless delays. They're annoying as hell, I agree, and there are some genuine complaints in the article, but most of what the author is complaining about makes it clear that he doesn't understand the point of the questions posed.
aidenn0 9y ago
IRBs for psychology really do seem needlessly bureaucratic. This is a case where a controlled experiment is not allowed, but the corresponding natural experiment would be (They would likely have gotten quick approval to investigate long-term outcomes of patients who had charted bipolar on this test). This is not the only time that I've heard of professional psychologists not being allowed to do something considered mundane and ethical as a course of their job if it were part of an experiment. Yes I am aware of trauma caused to people in not just Nazi experiments but in many post-war studies (many of which were of dubious value). It seems like at some point we should decide one of the following: A) Psychology has the potential to significantly alleviate suffering, so it's worth a risk of small harms in order to collect good data. B) Psychology doesn't have the potential to significantly alleviate suffering, so we shouldn't do any of these studies.
dunkelheit 9y ago
The problem with specialized committees that review and approve something is that they have absolutely no incentive to allow the new thing to go forward and every incentive to drag it into the ground with requests for more and more changes and precautions. Well, except when they are paid per review in which case they just rubber-stamp everything.
sprague 9y ago
You can't stop science. Someday, somebody, somewhere will answer the questions that this IRB requirement silenced. Whether it's via underdeveloped countries, or through the use of technologies too new to catch the regulator's eye, science will progress.
- euyyn 9y ago
  Or via a procedure that complies with the requirements of that IRB :)
msteffen 9y ago
Why do a study in the first place? Particularly when the outcome of the study is a foregone conclusion as far as the author is concerned. Just go to professional conferences and say "researchers say this works, but those of us with clinical experience know it doesn't. We write the diagnostic manuals, let's write a diagnostic procedure that actually makes sense to us"

304 points by apsec112 9y ago

Mandatory IRB story: http://www.npr.org/templates/story/story.php?storyId=102393652 http://www.npr.org/templates/story/story.php?storyId=1023936...

waisbrot 9y ago

It sounds like the study was poorly designed. Having an actual PI would have helped. The IRB probably would have seemed a lot less painful with a better-designed study and an engaged PI.

civilian 9y ago
You're probably getting downvoted because you're making a casual armchair judgement and this is Scott fucking Alexander we're talking about. //edit: I only just got to this section of the article, but it feels good to include as a counter-example. > During that year, Dr. W and I worked together on two less ambitious studies, carefully designed not to require any contact with the IRB. One was a case report, the other used publicly available data. > They won 1st and 2nd prize at a regional research competition. I got some nice certificates for my wall and a little prize money. I went on to present one of them at the national meeting of the American Psychiatric Association, a friend helped me write it up formally, and it was recently accepted for publication by a medium-tier journal. > I say this not to boast, but to protest that I’m not as much of a loser as my story probably makes me sound. I’m capable of doing research, I think I have something to contribute to Science. I still think the bipolar screening test sucks, and I still think that patients are being harmed by people’s reliance on it. I still think somebody should look into it and publish the results.
- umanwizard 9y ago
  Scott "fucking" Alexander is a blogger, not some sort of god immune to criticism. He's also not an expert authority on, as far as I can tell, anything but psychiatry.
- tallanvor 9y ago
  Sorry, but this is a real question: Why would we know who Scott Alexander is?
  - civilian 9y ago
    He's written a lot of great and influential things, and I feel like he's entering the "nerd/intellectual" common knowledge. The biggest example for me is Meditations on Moloch, which is the only thing that's ever made me deeply question my libertarianism. http://slatestarcodex.com/2014/07/30/meditations-on-moloch/ http://slatestarcodex.com/2014/07/30/meditations-on-moloch/ "I can tolerate anything except the outgroup" is a depressing piece on American politics. http://slatestarcodex.com/2014/09/30/i-can-tolerate-anything-except-the-outgroup/ http://slatestarcodex.com/2014/09/30/i-can-tolerate-anything... My original comment is not the best, it was a kneejerk reaction. I'm not representing the fandom well. :-/
SubiculumCode 9y ago
IRB stuff can be painful, but in my experience submitting these through the process is a lot less problematic than it was described in the article. An experienced PI knows how to navigate these issues. Practically, there are two issues with which the IRB is concerned. 1) Protection of particpants 2) Protection of the Institution from litigation by participants. Realizing the second oft unspoken goal can help you manage the IRB process

deleted 9y ago

[deleted]

karmajunkie 9y ago

I thought this was going to be about Ruby...

civilian 9y ago
Hah, I think your joke got stomped by a mod re-writing the headline. :]
- karmajunkie 9y ago
  yeah, they probably saw that coming a mile away... :)

jlarocco 9y ago

Poor choice of headline for this site, IMO. I can't be the only one who clicked expecting something about the Ruby REPL. For the benefit of anybody else confused by the title, IRB in this article refers to an "Institutional Review Board" https://en.wikipedia.org/wiki/Institutional_review_board https://en.wikipedia.org/wiki/Institutional_review_board

hirsin 9y ago

My freshman year of college I proposed a study to our hospitals IRB to strap small lasers to three week old infants in an effort to measure concentrations of a chemical in their blood. The most frustrating part was not the arcane insistence on ink and bolded study names, but the hardline insistence that it was impossible (illegal) to test the device before getting IRB approval - even on ourselves. Meaning that without any calibration or testing, our initial study would likely come back with poor results or be a dud, but we couldn't find out until we filled out all the paperwork. Incidentally, I recall it being much less painful than this, at least to get to the approval step. I would expect that each IRB varies greatly from group to group.

jpatokal 9y ago
What stopped you from testing on yourselves anyway outside the study?
- hirsin 9y ago
  Our PI was perhaps stricter than was useful. I believe it was intended as a teaching tool to force us to deal with the irb

omegaworks 9y ago

A take from the opposite end of the spectrum (ethically and logistically): https://arstechnica.com/science/2017/08/bucking-fda-peter-thiel-funds-patently-unethical-herpes-vaccine-trial/ https://arstechnica.com/science/2017/08/bucking-fda-peter-th... I wish the author took more seriously the warnings around patient health and safety. Yes, they may not apply to his particular project, but that doesn't make asking about them useless. To take past abuses lightly undermines his overall point. It's done precisely to avoid a situation like the way he delegated responsibility to his PI: "Dr. W had vaguely hoped that I was taking care of it. I had vaguely hoped that Dr. W was taking care of it." The review board shouldn't make vague assumptions about something as critical and necessary as ethical study design.

apsec112 9y ago
There are very serious costs to excessive safety regulation. When it makes drug development more expensive, marginal drugs don't get put through trials, and those drugs then never reach patients. Rational analysis means weighing the cost of each additional safety rule against the benefit. In many cases, there's a lot of evidence that the benefits of the current IRB process are trivial or nonexistent, and the costs are large.
ScottAlexander 9y ago
I take the possibility of serious ethical violations...well, seriously. I'm not sure whether the global cost of IRBs outweighs the benefits. But I'm pretty sure that the sort of regulatory changes OHRP proposed (mentioned in the linked NYT article), where studies like mine with minimal possible risk are exempted from IRB requirements, seem good and important. I also wonder whether it's possible to monitor even the scary studies with lots of potential risks - like the herpes vaccine trial you link to - with a little more common sense and a little bit less insistence on everything being signed in pen.
- zaroth 9y ago
  BTW, welcome [back] to HN. And just wanted to say, I am immensely enjoying very many of your articles and glad to see them making the front page!

paultopia 9y ago

It's kinda amazing any research gets done at all, with all the bureaucracy. My university requires official approval and a full process even to apply for a small non-institutional grant. I was going to apply for that tiny little AI Grant that was posted on here a while back, then I concluded that the rules probably would require me to get approval to apply for that. So I said "fuck it."

terminado 9y ago

“YOU LET PEOPLE SIGN CONSENT FORMS IN PENCIL, HOW CAN YOU JUSTIFY THAT?!” The funny thing is, contracts are legally binding agreements, whether they are verbal, scribbled on a napkin, signed in pencil, pen or blood, and whether the signature is an easily forged 'X' or elaborate calligraphy. Consent forms are a slightly different use-case, but I think the premise is the same: to show a meeting of the minds. Knowing this, the pencil signature actually protects the patient more than the institution, since the patient could theoretically forge a revocation of consent by erasing their mark. Meanwhile, ink isn't as binding, without some sort of objective third-party official (e.g. a notary public, beholden neither to patient nor to institution) serving as formal witness, further validating a non-forgery, in the case of a simple 'X' or erasure thereof. In practical terms, one solution to render the pencil signature binding would be to create a tamper-evident photocopy, with fraud-marks like a UUID moire pattern sprayed by an ordinary desktop printer, for each serialized document, readily detectable in the photocopy but permanently printed into the background of the form (e.g. yes, this is a photocopy of the same piece of paper, and the forger failed to notice the lightly tinted pattern, when erasing their signature), and retain the pencil copy, to ensure a match. In the event of an erasure, the paper might show an abrasion, and the photocopy would act as a snapshot/backup. Meanwhile, it should be possible to access existing, archived, historical questionnaires and actual clinical diagnoses from normal records, in a de-identified fashion, and compare results, no? Am I misunderstanding the goal of the study? Didn't he want to compare the assessment of the survey, with actual patient dispositions, and leave the rest of all treatment untouched, undisturbed and otherwise normal? Ask the questions, do the rest, same as ever, but analyze the opinion formed by the score of the answers to the questions, with the opinion of the professional caregiver beyond the scope of the simple Q&A exam?

SilasX 9y ago

I can't tell how much of this is exaggerated and how much really happened. Sometimes I have a hard time parsing the difference. They demand that you reveal what the study is on and sign in pen? Because the Nazis didn't disclose that stuff and get signatures?

tedunangst 9y ago
That's kinda the point. We need rules because Nazis (and Tuskegee, etc.) but it's not clear we need these rules about pen vs pencil.
- vacri 9y ago
  The pen vs pencil thing is a total canard by the author, who is looking to 'dress-up' his indignation. Hand the form to the subject. Hand a pen to the subject. Subject reads and signs the form. Take the form from the subject. Take the pen from the subject. Job done. It's not like the patient will go bonkers stabbing-crazy the millisecond their flesh touches a pen.
bshlgrs 9y ago
The first part is real, and the second part is the joke.

ak_yo 9y ago

The author doesn't seem to take the history of medical research ethics (e.g. the Tuskegee syphilis experiments [1]) seriously. Having a sophisticated understanding of the ethical risks of research is a huge part of doing good science, and this post certainly doesn't communicate a high level of sophistication. If the author thinks that ethics review is "Blindly trusting authority to make our ethical decisions for us", then he clearly doesn't understand what ethics review is or how it works. IRBs have well-documented flaws (for a more comprehensive look, read Laura Stark's book on IRBs [2]) but "IRBs are bad because they say I can't do whatever I want even though it's obviously the right way to do it" is not one of them. [1]: https://en.wikipedia.org/wiki/Tuskegee_syphilis_experiment https://en.wikipedia.org/wiki/Tuskegee_syphilis_experiment [2]: http://press.uchicago.edu/ucp/books/book/chicago/B/bo12182576.html http://press.uchicago.edu/ucp/books/book/chicago/B/bo1218257...

harryh 9y ago
The IRB process he described did not have a sophisticated understanding of the ethical risks of research. It imposed arbitrary and useless rules on an experiment with few if any risks.
- ak_yo 9y ago
  IRB review procedures are flawed, I agree. But ethical review by a third party is enormously important for research with human subjects. I'm alarmed by the notion of researchers deciding unilaterally that a study has only "nonexistent risks" -- that's a recipe for disastrously unethical scientific conduct.
  - harryh 9y ago
    I don't think the author was proposing getting rid of oversight completely, just that said oversight should, you know, make sense.
  - mannykannot 9y ago
    The point is that the decision should not have been unilateral - the author and his colleague had a very good case for their position, based on the fact that they were not asking anything that was not being asked anyway, and not recording anything that was not being recorded anyway. It was mindless dogma on the part of the IRB that made it unilateral. Meanwhile, a practice with a real potential for risk went unstudied. You might argue that there was risk to the patients in the one part of the procedure that was different than what they would experience anyway - the actual asking for consent - but if so, what would you propose doing about it? You have made a straw man out of this one case, and are defending it with all the dogma displayed by this IRB.

tom_b 9y ago

And it's only worse at the staff level. Even getting patient data variables for patients consented to IRB-approved studies is hit-or-miss. Groups build barriers by wanting to "work with" principal investigators (PIs) instead of providing data to PIs and study staff. Data moats are everywhere. My own group (at a research center attached to a university school of medicine) has switched our focus from helping PIs with clinical data management and applications to helping manage research samples and sample processing, mainly by doing housekeeping around genomic processing across different lab teams. It's kind of hard to convey just how much I would warn devs away from healthcare and healthcare research IT gigs.

88e282102ae2e5b 9y ago

The initial meeting with the IRB where they proposed ridiculous and irrational changes to the consent forms was strikingly reminiscent of every conversation I've had in a third world country where someone was asking me for a bribe, but couldn't say it directly.

gwern 9y ago
I've noticed how often big projects will allocate something like 5% of their budget for a few things like 'bioethicists' and 'community' grants; you never seem to hear about what they produce or what important findings they make, but the money does get spent...
- sorokod 9y ago
  Money always gets spent. It is in its nature.

deleted 9y ago

[deleted]

schoen 9y ago

I thought I remembered a proposal to create some new categorical exemptions from IRB review, one of which was somehow related to studies where you just talk to people. (I've been interviewed by several social scientists and very often their "protocols", consisting exclusively of informal interviews, had clearly been through extensive IRB reviews, which felt super-odd because journalists wouldn't need any ethical review at all in order to carry out exactly the same interview in exactly the same way.) On the other hand, you clearly can harm people in various ways by or as a result of interviewing them, for example by breaching confidentiality, by being forced to breach confidentiality, or by making them feel bad about themselves by insulting them during the interview or just by bringing up painful and distressing memories. And also it's hard for researchers who aren't doctors to extend the same level of legal protection to what their research subjects tell them: https://www.socialsciencespace.com/2017/01/social-science-needs-doctor-patient-confidentiality/ https://www.socialsciencespace.com/2017/01/social-science-ne... So maybe one idea would be to create a categorical IRB exemption for studies that combine non-invasive observations, or existing datasets, with interviews not reasonably expected to be distressing to the subjects, and conducted subject to otherwise existing norms on patient confidentiality and medical privacy? (recognizing that limited legal protection for that confidentiality still poses problems for some studies) (I'm certainly happy to apply the Chesterton's fence principle and learn more about the history of unethical experimentation before seriously advocating this.)

munin 9y ago
> which felt super-odd because journalists wouldn't need any ethical review at all in order to carry out exactly the same interview in exactly the same way. Journalists don't have a history of injecting you with syphilis just to see what happens, though.
- schoen 9y ago
  Neither do anthropologists or computer scientists, who are both now subject to IRB review in universities.
  - 9y ago

mjevans 9y ago

I suspect that the proper ethical thing to do depends on the risks to the patients, and the benefits to the subjects and overall population. In my own opinion the author is quite correct in classifying the medical benefits to the subjects (practically speaking, zero; equivalent to talking with any random untrained individual), and to future patients (potentially great benefits). The risks seem to be much more closely associated with accidentally exposing patient information. This is the section that their study protocols (at least what was described in overview) need review and revision. The unblinding documentation should be kept in a secured, designated area. A safe with a very limited ACL would be ideal. Unblinding information could be either the consent forms with the unique non-PII ID information on them or a list for matching unique non-PII IDs to (hard copies of) existing patient record stubs attached to the consent forms. If possible the study should have actually been designed to use as much of the already collected data as possible, and possibly ask the small additional information of all, but not record the answers (or destroy the collected information if they do not consent). Hypothetically they could be asked if they consent anonymously providing some of the questions they were already asked to an informational medical research study. If you ask them for consent AFTER having asked (and gotten answers on) the questions, then you're even actually safe to tell them which data are being collected (what their answers were as well) and what you're attempting to study. I believe that the above is the most ethical way of designing this type of study, and that it sounds similar to a term I've heard professionals in this field use 'retrospective study' (collecting study data based on previously collected medical data).

harryh 9y ago
Why should study data subject to higher security standards than information that is collected on the patient during the process of providing care that is much more sensitive?
- mbreese 9y ago
  All PHI is subject to the same data security standards, regardless of if it is from research or clinical sources.

xr4ti 9y ago

I can't agree. The authors wounds at every stage in the narrative were self inflicted. (1) The study title doesn't work at the top of the form? Change it! Surely there is some other arrangement of English words that would satisfy both the author and IRB. (2) They leave the risk section blank because minimal risk, and then complain about the requirement for pens because the patients might stab themselves if given pens. Looks to me like they didn't spend much effort thinking about risks to the patient. If they had just mentioned the pen stabby thing as a risk factor, and given "don't use pens" as a risk minimization plan, they wouldn't have had a problem. (3) They poopoo the training because Hitler, and then get upset when they unknowingly violate a personnel rule that is most definitely covered in any decent IRB/GCP training. I could go on, but every new thing I enumerate makes me angrier. Life is too short. Look. IRB applications suck, but human experimentation is serious business. The rules aren't there to make the study easy, the are there to (a) look out for the patient (respect), (b) evaluate risk and benefits of the study (beneficence), and (c) ensure that the benefits are distributed to the group bearing the risk (justice).

harryh 9y ago
From reading the post it's not at all clear to me that 2 of the 3 goals you list were accomplished by the rules. A) Instead of actually looking out of the patient a list of arbitrary rules were imposed. B) The risk (minimal) was not properly evaluated but instead massively overstated. Given this failure of the process to meet said goals it seems perfectly reasonable to question the process.
- xr4ti 9y ago
  (A) The rules are not arbitrary. They are the product of much careful deliberation. https://en.wikipedia.org/wiki/Belmont_Report https://en.wikipedia.org/wiki/Belmont_Report (B) Is experimentation on suicidal patients really minimal risk? Maybe, maybe not. But failure of the investigator to show they have thought about it is a huge red flag.
  - harryh 9y ago
    (A) They were arbitrarily imposed in this case. Just because a rule is a good idea doesn't mean it is a good idea all the time. (B) We're not talking about injecting the patients with some experimental drug here. We're talking about asking them questions that they were already being asked anyways. The lack of risk is self evident to anyone who considers the question.

cassowary37 9y ago

So, to recap: you tried to do a human subjects study without a) getting help from someone with prior research experience and b) understanding all the details required to do such a study. The IRB's responsibility (and yours, as the PI) is protection of human subjects. That you're so cavalier about this responsibility and the history behind these requirements suggests you should probably have paid more attention to that course you berate.

tertius 9y ago
Or collaborated with someone who cares enough.

trhway 9y ago

>I knew exactly how to handle this: one by one, I sought out the laziest attendings in the hospital and asked “Hey, would you like to have your name on a study as Principal Investigator for free while I do all the actual work?” Yet one by one, all of the doctors refused, as if I was offering them some kind of plague basket full of vermin. when you find yourself in the situation of being the only smart guy surrounded by stupid morons (nevermind the fact that these morons are more highly trained and experienced) ... A related Russian joke. While driving, the husband calls his wife and she tells him that it is on the news right now that some moron is driving in the wrong direction on the highway. The husband exclaims in response "A moron?! There are hundreds of such morons driving in the wrong direction here!"

paulsutter 9y ago

Here's a story about sidestepping IRB processes by going offshore http://khn.org/news/offshore-rush-for-herpes-vaccine-roils-debate-over-u-s-safety-rules/ http://khn.org/news/offshore-rush-for-herpes-vaccine-roils-d... "Neither the Food and Drug Administration nor a safety panel known as an institutional review board, or an IRB, monitored the testing of a vaccine its creators say prevents herpes outbreaks. Most of the 20 participants were Americans with herpes who were flown to the island several times to be vaccinated, according to Rational Vaccines, the company that oversaw the trial."

tim333 9y ago
Discussed on HN yesterday https://news.ycombinator.com/item?id=15126557 https://news.ycombinator.com/item?id=15126557 As the people doing that one argue, there is an argument for balancing the risks of the test against the risk to suffers of illness and the like of not doing the tests.

Nition 9y ago

Someone on Reddit says[1]: > Amateur. What you do is you sweet talk the clinicians into using their medical judgement to adopt the form as part of their routine clinical practice and get them to include it as part of the patient's medical records. Later... you approach the IRB for a retrospective chart review study and get blessed with waived consent. Bonus: very likely to also get expedited review. [1] https://www.reddit.com/r/slatestarcodex/comments/6wtylk/my_irb_nightmare/dmas5l5/ https://www.reddit.com/r/slatestarcodex/comments/6wtylk/my_i...

j88439h84 9y ago
What form?
- haltingthoughts 9y ago
  The author was trying to test whether a simple common test for bipolar was accurate. Presumably, the suggestion is to make the test a form filled out by each doctor for each patient. Then months or years later the author could do a chart review and see whether the test results were accurate.
  - nohat 9y ago
    To potentially clarify: the test is just a form. It's a form with some questions about mood, and is currently being used to diagnose bipolar disorder (inappropriately, the author believes).
kerbalspacepro 9y ago
That sounds more unethical though?

smithkl42 9y ago

Commenters have mentioned "Chesterton's Fence". I'd add a reference to another of GKC's observations: "When you break the big laws, you do not get liberty; you do not even get anarchy. You get the small laws."

penpapersw 9y ago
I can't help but wonder if the Fence comments had anything to do with my blog post last night. Kind of like watching social osmosis in action :D
- slavik81 9y ago
  Chesterton's fence has long been a staple of HN discussions. Unfortunately, it involves no risk/reward analysis and it invariably leads to needless conservatism. That's probably not a big deal in medical research where risks are often high, but it's a problem when applied as a general principle. (The opposite wisdom is probably grandma's ham [1]. Consider that if grandma had passed away, they might never be allowed to try cooking the whole ham.) I quite enjoyed that blog post, btw. [1]: http://www.angelfire.com/ma/artemis9/humor/joke7.html http://www.angelfire.com/ma/artemis9/humor/joke7.html
  - throwanem 9y ago
    The point of Chesterton's fence is cost/benefit analysis. It's not saying the fence mustn't go; it's saying you need to know why the fence was put there in order to evaluate whether it's best removed or left in place - that is, your analysis needs to be informed by understanding of the status quo in order to be accurate. Grandma's ham is a great example. It might have been some subtle trichinosis-related failure case that cutting the end off a ham reliably prevents, and it's a shame we had to lose your great-uncle Joe to find out about that. Or it might have been an issue of pan length. You can't know until you ask, so you ask if you can. If you can't ask, then sure, you do the best you can with what you have - I should like to hope there are no blind dogmatists here. But if you can ask, you'd be a fool not to.

ryandrake 9y ago

> Faced with someone even more obsessive and bureaucratic than they were, the IRB backed down and gave us preliminary permission to start our study. I love it! You can’t fight mindless rule followers with logic or reason. You need to use even more detailed rule-following. This is true in every bureaucracy where checklists have replaced good judgment.

chi17 9y ago

Man. This sounds just like my workplace.

epmaybe 9y ago

I'm in the process of getting two IRBs approved. I am somewhere in the first quarter of that author's story. I agree that it's a big hurdle to get past, and writing a good consent form is hard. All of this being said, though, it was/is a great experience to have under my belt. And, next time, I'll be able to tackle the regulatory hurdles with far less confusion.

alexandercrohde 9y ago

For those who think the author is in the wrong 1 - Do you think this story indicates a success or a failure of the IRB? 2 - Would you rather a better expedited review process AND more studies (remember the personal effort cost killed this study), neither, or do you see some 3rd option?

icelancer 9y ago
>>Do you think this story indicates a success or a failure of the IRB? Failure of the IRB but mostly failure of the author to explore his options. >>Would you rather a better expedited review process AND more studies (remember the personal effort cost killed this study), neither, or do you see some 3rd option? As I've posted in this thread, this problem is solved. Private IRBs exist that are much more responsive.
- tedunangst 9y ago
  What options do you think were inadequately explored to sign the consent forms? Finger paint?
  - icelancer 9y ago
    >>What options do you think were inadequately explored to sign the consent forms? None. Now, if you asked a question without assuming what you think I was saying, I'd tell you that Alexander's other options were to not use the IRB that he went to, and to explore the private marketplace of IRB organizations that provide much better service than the nightmare he dealt with - a very common practice in all fields.
    - mattkrause 9y ago
      > Alexander's other options were to not use the IRB that he went to This is, frankly, ridiculous. The 50-70% overhead the university takes from every grant already pays for one IRB. If that one isn't working, the solution is not to tap in (almost inevitably tight) research funds to pay for a second for-profit one instead. Given how hard it is to charge things like computers to grants, I'm actually amazed this is even allowed.

yk 9y ago

Well my story with the research bureaucracy is, a coworker had to fill out a notice that he got some money for a conference from outside the university and he dutifully filled out everything. Thing is, as a theoretical physicist you are supposed to fill out the first page, check the box on page 27, that there is not conflict of interest, and sign the thing. Any answer on certain questions will trigger a public hearing by some very obscure board. Public hearing means, that poor guy has to go there in case the public has questions. He filled out everything, so he had to go to the anti-corruption board and the nuclear safety commission and so on. After roughly three weeks of him whining that he always sits there like an idiot, I got curious and came along to one of the hearings, definitely for some reason other than watching him sit there like an idiot, or so I claim. That was the animal cruelty (against) commission and at the hearing were the committee, a very nervous working group, one unhappy postdoc and a grad student who just wants to watch the show. The PI of the working group then gave a presentation on the sad reality, that you can't study pain receptors if you anesthetize the mouse. (Apparently that was not the first time they were there, so I got just the part on why they are certain that there is really no way to get a statistical meaningful result with less mice.) Really drove home the point that these procedures are not designed for the convenience of theoretical physicist, but are designed to tackle really hard problems.

hawkice 9y ago
If the ethical problems of rat suffering are really hard problems, why is it utterly mundane for people to buy products at home depot to deliberately poison them for days and days until they succumb to internal bleeding, for no other reason then "it was bothering me, existing in my house"? Perhaps we should or could start caring, but we definitely do not even acknowledge there being a meaningful question there in any other element of human life.
- munin 9y ago
  Fortunately for us, the IRB only governs human experimentation. You need very little approval to experiment on animals, in the US at least.

icelancer 9y ago

Much of this reminds me of what I have been going through. What SSC should have investigated and wrote up is that hospital and medical IRB is fraught with bureaucracy and that a private IRB industry has developed, entities inside that handle wide ranges of studies and turn around informed consent docs and things very fast (within a week). Our first IC forms got banged multiple times for the same reasons his did. And it’s annoying as shit. But my ethical training focused not on something stupid like Nazis, but something real like American scientists unethically hiding treatment from diseases they were studying, like the Tuskegee Syphilis Experiment (the actual reason IRB exists as is). Or the Milgram Experiment. Etc. A lot of IRB review is stupid. But messing with human subjects doesn’t have a good track record in this country. Our predecessors earned every bit of bureaucracy and oversight based on their absurd experiments. Such is life.

SilasX 9y ago
That sounds, to me, eerily close to the mentalities that Alexander rightly criticizes: - "It's no big deal because you can just pay off someone's who's in the know and can work the system rather than read the rules at face value." - "No amount of kafkaesque absurdity is too much, because Nazis."
- icelancer 9y ago
  What specifically sounds like that? I cannot decipher the context you are using those quotes in regarding my comment.
  - SilasX 9y ago
    For the first one: >"What SSC should have investigated and wrote up is that hospital and medical IRB is fraught with bureaucracy and that a private IRB industry has developed, entities inside that handle wide ranges of studies and turn around informed consent docs and things very fast (within a week). That is, even though Alexander had a long wait time just to hear responses, there are expensive services you can use that make all that wait time and pushback from hidden rules disappear, so you don't see it as a problem. For the second one: >A lot of IRB review is stupid. But messing with human subjects doesn’t have a good track record in this country. Our predecessors earned every bit of bureaucracy and oversight based on their absurd experiments. Such is life. You don't show a standard for the upper bound on when the absurdity is too much, dismissing any such concerns with appeal to some historical atrocities.

Glyptodon 9y ago

Maybe he should have called it a care audit and made it look like they were auditing the doctors success at administering screenings instead of doing any kind of patient research.

michaelbuckbee 9y ago

A separate discussion of this happening in /r/slatestarcodex Top comment there right now: "Amateur. What you do is you sweet talk the clinicians into using their medical judgement to adopt the form as part of their routine clinical practice and get them to include it as part of the patient's medical records. Later... you approach the IRB for a retrospective chart review study and get blessed with waived consent. Bonus: very likely to also get expedited review." Which is an extremely clever "hack" (in the way we like to use it on HN) of the IRB setup.

JonasJSchreiber 9y ago

I got up to the part where the author talks about including the test results in the patient's file. Up to there, every point of contention either seems whiny or completely valid on the institution's part.

alexashka 9y ago

The real nightmare is having read through 3/4 of this, only to realize it's the longest bitchy complaint I've ever read through. No substance and a lot of arrogance. The more curious take-away is that these extremely long winded rants of his have younger people fooled as being worthwhile. If you have something of value to say - you distill it, condense it, so as to ease the mental effort required for someone else to understand the point you're trying to make. It reminds me of what philosophers do - they exhaust you into thinking they're making some sophisticated point simply because they're using fancy words that produce the thought of 'am I missing something? Surely no one is spouting complete nonsense for 400 pages, it must be me'. It isn't you. It's them on a massive ego trip - this guy being a great example.

sctb 9y ago
A more charitable interpretation might be that a detailed and clinical account of the absurdity of the most tedious bureaucracy may have made for rather... dry reading.
- alexashka 9y ago
  I can summarize what he's said in the 3/4s that I've read, down to 2 or 3 paragraphs, with a few bullet points. The author complains about tedious procedure, while writing the most tedious blog I've had the displeasure of reading. The obliviousness and callous arrogance of the author, combined with cult-like reverence some people seem to have for his babbling is what makes this at all worth commenting on.
  - frooxie 9y ago
    I don't want it summarized; every paragraph is funny and interesting as is. You, on the other hand, seem oblivious and arrogant.

vacri 9y ago

The author complains about there not being enough of an ethics gauntlet with the bipolar test in question, and then complains of having to run an ethics gauntlet to do the study. The author keeps on bringing up nazis and suggests he knows about study methods, when it's clear he's pretty clueless about doing studies. Several of the "this is stupid" questions really aren't that stupid, given that these are generic forms for any study. Why the fuck do you care if you're being asked if you're removing organs? Just tick 'no' and move on to the next question. Does the author also complain that other genders are selectable on online forms, and not just his own gender - after all, it's equally useless information? The setup to the story is: 'waaah, this application form isn't tailored exclusively to MY study'. I ran into problems with ethics committees myself when doing an (aborted) PhD that caused significant needless delays. They're annoying as hell, I agree, and there are some genuine complaints in the article, but most of what the author is complaining about makes it clear that he doesn't understand the point of the questions posed.

aidenn0 9y ago

IRBs for psychology really do seem needlessly bureaucratic. This is a case where a controlled experiment is not allowed, but the corresponding natural experiment would be (They would likely have gotten quick approval to investigate long-term outcomes of patients who had charted bipolar on this test). This is not the only time that I've heard of professional psychologists not being allowed to do something considered mundane and ethical as a course of their job if it were part of an experiment. Yes I am aware of trauma caused to people in not just Nazi experiments but in many post-war studies (many of which were of dubious value). It seems like at some point we should decide one of the following: A) Psychology has the potential to significantly alleviate suffering, so it's worth a risk of small harms in order to collect good data. B) Psychology doesn't have the potential to significantly alleviate suffering, so we shouldn't do any of these studies.

dunkelheit 9y ago

The problem with specialized committees that review and approve something is that they have absolutely no incentive to allow the new thing to go forward and every incentive to drag it into the ground with requests for more and more changes and precautions. Well, except when they are paid per review in which case they just rubber-stamp everything.

sprague 9y ago

You can't stop science. Someday, somebody, somewhere will answer the questions that this IRB requirement silenced. Whether it's via underdeveloped countries, or through the use of technologies too new to catch the regulator's eye, science will progress.

euyyn 9y ago
Or via a procedure that complies with the requirements of that IRB :)

msteffen 9y ago

Why do a study in the first place? Particularly when the outcome of the study is a foregone conclusion as far as the author is concerned. Just go to professional conferences and say "researchers say this works, but those of us with clinical experience know it doesn't. We write the diagnostic manuals, let's write a diagnostic procedure that actually makes sense to us"