9 ms·

Against pandemic research exceptionalism

dia80 7y ago
I had wondered if a bandit algorithm would be an effective approach. Can anyone who knows more about it than me comment?
- SkyMarshal 7y ago
  I doubt AI is good enough yet to read through research papers and reliability identify methodological flaws that render the conclusion suspect or invalid. Pretty sure that’s still a job for human experts.
  - dia80 7y ago
    As I understood it you would start choosing treatments at random, as one or more treatments started to outperform the others they are chosen with higher frequency. A sort of on-line trial. No AI involved.
    - lixtra 7y ago
      Yes, it would work if human patients were lab rats.
      - James_Henry 7y ago
        You should realize that in clinical trials human patients are very much like lab rats. They receive a randomized treatment.
        gus_massa 7y ago
        Note that the alternative is use the whole humanity as lab rats. Something like: fake quote> We have the gut feeling that this drug is better. We got good results in animals models with a somewhat related illness. Let's try this new drug with everyone for a few years. There are multiple phases of the trials. The first use a small number of people in case there is something very wrong. The last use more people, to try to detect subtle problems.
        James_Henry 7y ago
        I'm not sure I understand what you are getting at here. Studies with lab rats have control groups, placebo groups, populations, etc. We don't give all the rats in the world some drug we have a hunch about.
        gus_massa 7y ago
        I think we agree. Quoting your comment: > human patients are very much like lab rats Some people consider that offensive because lab rats are discardable. (They even kill all the rats after each study, they are not reused in another study or get free in a rat sanctuary.) This is a common excuse against double blind randomized trials with a control group. Why are some people getting the "worse" standard drug instead of the new (perhaps) "magical" drug? Why make experiments with them instead of giving the new drug? Why use blind groups and use them as lab rats? (In every Covid-19 thread, someone propose to use the database of all the known cases and some statistic to simulate a control group (How hard can it be?), and then give everyone in the trial the new drug.) Expanded version of my comment: The alternative to using a good clinical trial with all the modern constrains to differentiate between new drugs that are good and new drugs that are bad is to use only some experiments in animals or in vitro. Not every illness affect every animal, so there are some restrictions, or the test in animals must use something that is similar enough. Imagine that without a good clinical trial in humans, someone approves the new drug. After some time the problems with the new drug may appear. In this case instead of testing the new drug with a small group of people in the trial, you are testing the new drug in the wild with whatever person that is so unlucky to get ill and get the new drug, so you are using the whole humanity as a trial group.
- James_Henry 7y ago
  Yes, there are many papers on looking at clinical trials as bandit problems (because they are bandit problems) and applying different bandit algorithms. You may also want to look at SMAR Trials and adaptive trials.
- gus_massa 7y ago
  The name for an "A/B test" is "randomized control trial" (with two arms). There are also trials with more arms, like * standard treatment + placebo * standard treatment + some drug * standard treatment + double amount of drug and other combinations, like using two drugs in one arm.
  - marcosdumay 7y ago
    Bandit algorithms aren't different only because they have multiple arms. They also follow the least possible amount of harm by updating the conclusions in a Bayesian style during the entire experiment, not only on the end. They also can stand more arms appearing during the experiment after people know more.
roenxi 7y ago
My normal position is fairly anti-government, but there is a role for an agency here that I wish they'd fill. It'd be fantastic to have a centralised agency organise a panel of experts who have gone though all the known papers and studies, synthesised them then just sat for a week doing a massive public Q&A session. Targeted at people who are really quite clever rather than the usual aiming-low that the mass media goes for. The volume of research has gotten far too much for me to keep track of; the media are in full swing polluting the channel with irrelevant and frequently political noise. I really do not care about what politicians think about given drugs, and honestly their justifications for why they are doing X or Y are not going to matter to me for a while because politics doesn't happen quickly. It'd be nice to have a centralised, low bandwidth well moderated information source for the smartest 5-10% of people to get good information in response to their questions and uncertainties. There are great information channels for PhD medicine types & doctors (I hope). There are good sources for people who are at risk of drinking bleach because they heard it might help (try a local news channel). There is a hole for, say, the top 5% smartest people where they really don't have an in to figure out how they might help, because they need information hidden in very technical articles or behind rather black-box modelling as described in the media. But they are otherwise very educated, intelligent, capable, wealthy or what have you. These are coordination problems a government could actually help with.
- deleted 7y ago
  [deleted]
- Jedd 7y ago
  I agree in principle with your wish for some neutral <unquote>, informed, apolitical, insightful, measured, and cautious commentary that distils the current consensus. But it sounds like a lot of the concerns you express are specifically around the problems that are currently being experienced by one poorly administered country. I think elsewhere in the world, a lot of people feel they're already getting this level of information, without the political overload & misdirection.
  - roenxi 7y ago
    I reject the notion that Australia is poorly administered. Our COVID-19 cases are extremely low [0] and the government has been doing a fine job compared to most others. :) [0] https://www.health.nsw.gov.au/news/Pages/20200503_00.aspx https://www.health.nsw.gov.au/news/Pages/20200503_00.aspx
dahart 7y ago
> Numerous trials investigating similar hypotheses risk duplication of effort, and droves of research papers have been rushed to preprint servers, essentially outsourcing peer review to practicing physicians and journalists. I’m curious about this point. Often we have the opposite problem, that we can’t get anyone to replicate studies with strong looking but potentially specious results. Isn’t perhaps a small amount of duplication of effort a good thing, especially if studies are rushing and letting standards slip a little? Aside from coordinating and not using too much of the funding to investigate a particular hypothesis, what are the other reasons to avoid duplication of effort?
- SiempreViernes 7y ago
  Their point is that there is actually a very large amount of duplicate effort is happening now, as everyone separately goes for the first ideas: this makes it more probable that a lot of effort will be wasted on super-duper-triple confirming that the first thing everyone thought of didn't work, slowing the exploration of more nuanced ideas. But besides waste of time and resources, the only other problem they mention is an increase of false positives. These naturally become more common as you try the same thing more often unless you correct for it by keeping track of all the other trials.
  - gus_massa 7y ago
    My problem is that there are too many inconclusive trial. Someone makes a small trial without a control group, so the result is promising but difficult to interpret. Someone else makes a small trial without a control group, so the result is promising but difficult to interpret. Another person makes a small test without a control group, so the result is promising but difficult to interpret. ... So after a few months, we only have promising inconclusive results. Not that all the trials are different, so it is difficult to compare them. IIRC There was a recent trial with remdesivir without good results, but they only tried the drug in patients with strong symptoms. So people asked what happens if they use the drug in the beginning of the infection.
ISL 7y ago
As someone who was arguing for speedy reporting a few weeks ago: the longer the pandemic goes on, the more important rigor becomes. Waiting a short time to get things right today is a a very different matter than a 4-8 weeks ago, where on-the-ground information was completely absent. Speed still really matters, but the longer this goes the more taking our time and doing things right will pay off.
- SiempreViernes 7y ago
  Are you arguing that in the initial stages incorrect information is less harmful than no information?
  - IAmEveryone 7y ago
    No, they are not arguing that. I can tell because their post says something else, and because that would be stupid.
  - occamrazor 7y ago
    In March there wasn’t enough information to decide how strict the social distancing needs to be in order to reduce the spread of the infection. At the same time it was clearly necessary to do something, without knowing in advance how much it would affect the epidemic. Now we can take a little more time to decide what to do next, and take more focused and effective measures.
  - bo1024 7y ago
    There’s a difference between incorrect and incomplete. Facts, like how many people in a trial of 23 recovered, can be shared and useful early on even though they’re incomplete. That’s different from attempting to draw a strong conclusion about what works from n=23, that might be an incorrect conclusion.
    - mattkrause 7y ago
      The form that the incomplete data took was part of the problem. Adding numbers to a spreadsheet is one thing. When the same data takes the form of an NJEM or Lancet paper, it definitely implies--and some cases, outright says--more than "FYI: 37/53 people got better when we did [this]." Part of this is how papers are ordinarily written, but I think the authors (and editors) should have stopped all over the word "improvement" in those manuscripts.
kqr2 7y ago
Not only research, but a lot of diagnostic companies are trying to take advantage of the gold rush: Theranos Would Be Thriving in the Covid-19 Pandemic https://elemental.medium.com/theranos-would-be-thriving-in-the-covid-19-pandemic-63ec6d45c96a https://elemental.medium.com/theranos-would-be-thriving-in-t...
bo1024 7y ago
Really interesting question here. I wish it went a bit more into specifics of how to navigate the tradeoffs. Maybe one way to put this is that it’s fine to do preliminary studies that motivate and direct the actual rigorous research, but we’re stuck in a place where people are only doing the preliminary studies, a lot, and not getting to the rigorous stage.
pmiller2 7y ago
What happened to the title? The actual title is "Against Pandemic Research Exceptionalism". If this was intentional, it was a bad choice, because information was lost.
hartator 7y ago
> perpetuate the perception that, when it comes to the rigors of science, crisis situations demand exceptions to high standards for quality. Where are the high quality studies warranting strict lockdowns?
- pnako 7y ago
  In fact I would be curious if a single medical textbook anywhere in the world mentions this as a possible solution to any epidemic. It seems to be a innovation, coming from politicians more than scientists. The practice so far, still uncommon, was to quarantine sick people, not force the entire population to stay at home.

I had wondered if a bandit algorithm would be an effective approach. Can anyone who knows more about it than me comment?

SkyMarshal 7y ago
I doubt AI is good enough yet to read through research papers and reliability identify methodological flaws that render the conclusion suspect or invalid. Pretty sure that’s still a job for human experts.
- dia80 7y ago
  As I understood it you would start choosing treatments at random, as one or more treatments started to outperform the others they are chosen with higher frequency. A sort of on-line trial. No AI involved.
  - lixtra 7y ago
    Yes, it would work if human patients were lab rats.
    - James_Henry 7y ago
      You should realize that in clinical trials human patients are very much like lab rats. They receive a randomized treatment.
      - gus_massa 7y ago
        Note that the alternative is use the whole humanity as lab rats. Something like: fake quote> We have the gut feeling that this drug is better. We got good results in animals models with a somewhat related illness. Let's try this new drug with everyone for a few years. There are multiple phases of the trials. The first use a small number of people in case there is something very wrong. The last use more people, to try to detect subtle problems.
        James_Henry 7y ago
        I'm not sure I understand what you are getting at here. Studies with lab rats have control groups, placebo groups, populations, etc. We don't give all the rats in the world some drug we have a hunch about.
        gus_massa 7y ago
        I think we agree. Quoting your comment: > human patients are very much like lab rats Some people consider that offensive because lab rats are discardable. (They even kill all the rats after each study, they are not reused in another study or get free in a rat sanctuary.) This is a common excuse against double blind randomized trials with a control group. Why are some people getting the "worse" standard drug instead of the new (perhaps) "magical" drug? Why make experiments with them instead of giving the new drug? Why use blind groups and use them as lab rats? (In every Covid-19 thread, someone propose to use the database of all the known cases and some statistic to simulate a control group (How hard can it be?), and then give everyone in the trial the new drug.) Expanded version of my comment: The alternative to using a good clinical trial with all the modern constrains to differentiate between new drugs that are good and new drugs that are bad is to use only some experiments in animals or in vitro. Not every illness affect every animal, so there are some restrictions, or the test in animals must use something that is similar enough. Imagine that without a good clinical trial in humans, someone approves the new drug. After some time the problems with the new drug may appear. In this case instead of testing the new drug with a small group of people in the trial, you are testing the new drug in the wild with whatever person that is so unlucky to get ill and get the new drug, so you are using the whole humanity as a trial group.
James_Henry 7y ago
Yes, there are many papers on looking at clinical trials as bandit problems (because they are bandit problems) and applying different bandit algorithms. You may also want to look at SMAR Trials and adaptive trials.
gus_massa 7y ago
The name for an "A/B test" is "randomized control trial" (with two arms). There are also trials with more arms, like * standard treatment + placebo * standard treatment + some drug * standard treatment + double amount of drug and other combinations, like using two drugs in one arm.
- marcosdumay 7y ago
  Bandit algorithms aren't different only because they have multiple arms. They also follow the least possible amount of harm by updating the conclusions in a Bayesian style during the entire experiment, not only on the end. They also can stand more arms appearing during the experiment after people know more.

roenxi 7y ago

My normal position is fairly anti-government, but there is a role for an agency here that I wish they'd fill. It'd be fantastic to have a centralised agency organise a panel of experts who have gone though all the known papers and studies, synthesised them then just sat for a week doing a massive public Q&A session. Targeted at people who are really quite clever rather than the usual aiming-low that the mass media goes for. The volume of research has gotten far too much for me to keep track of; the media are in full swing polluting the channel with irrelevant and frequently political noise. I really do not care about what politicians think about given drugs, and honestly their justifications for why they are doing X or Y are not going to matter to me for a while because politics doesn't happen quickly. It'd be nice to have a centralised, low bandwidth well moderated information source for the smartest 5-10% of people to get good information in response to their questions and uncertainties. There are great information channels for PhD medicine types & doctors (I hope). There are good sources for people who are at risk of drinking bleach because they heard it might help (try a local news channel). There is a hole for, say, the top 5% smartest people where they really don't have an in to figure out how they might help, because they need information hidden in very technical articles or behind rather black-box modelling as described in the media. But they are otherwise very educated, intelligent, capable, wealthy or what have you. These are coordination problems a government could actually help with.

deleted 7y ago
[deleted]
Jedd 7y ago
I agree in principle with your wish for some neutral <unquote>, informed, apolitical, insightful, measured, and cautious commentary that distils the current consensus. But it sounds like a lot of the concerns you express are specifically around the problems that are currently being experienced by one poorly administered country. I think elsewhere in the world, a lot of people feel they're already getting this level of information, without the political overload & misdirection.
- roenxi 7y ago
  I reject the notion that Australia is poorly administered. Our COVID-19 cases are extremely low [0] and the government has been doing a fine job compared to most others. :) [0] https://www.health.nsw.gov.au/news/Pages/20200503_00.aspx https://www.health.nsw.gov.au/news/Pages/20200503_00.aspx

dahart 7y ago

> Numerous trials investigating similar hypotheses risk duplication of effort, and droves of research papers have been rushed to preprint servers, essentially outsourcing peer review to practicing physicians and journalists. I’m curious about this point. Often we have the opposite problem, that we can’t get anyone to replicate studies with strong looking but potentially specious results. Isn’t perhaps a small amount of duplication of effort a good thing, especially if studies are rushing and letting standards slip a little? Aside from coordinating and not using too much of the funding to investigate a particular hypothesis, what are the other reasons to avoid duplication of effort?

SiempreViernes 7y ago
Their point is that there is actually a very large amount of duplicate effort is happening now, as everyone separately goes for the first ideas: this makes it more probable that a lot of effort will be wasted on super-duper-triple confirming that the first thing everyone thought of didn't work, slowing the exploration of more nuanced ideas. But besides waste of time and resources, the only other problem they mention is an increase of false positives. These naturally become more common as you try the same thing more often unless you correct for it by keeping track of all the other trials.
- gus_massa 7y ago
  My problem is that there are too many inconclusive trial. Someone makes a small trial without a control group, so the result is promising but difficult to interpret. Someone else makes a small trial without a control group, so the result is promising but difficult to interpret. Another person makes a small test without a control group, so the result is promising but difficult to interpret. ... So after a few months, we only have promising inconclusive results. Not that all the trials are different, so it is difficult to compare them. IIRC There was a recent trial with remdesivir without good results, but they only tried the drug in patients with strong symptoms. So people asked what happens if they use the drug in the beginning of the infection.

ISL 7y ago

As someone who was arguing for speedy reporting a few weeks ago: the longer the pandemic goes on, the more important rigor becomes. Waiting a short time to get things right today is a a very different matter than a 4-8 weeks ago, where on-the-ground information was completely absent. Speed still really matters, but the longer this goes the more taking our time and doing things right will pay off.

SiempreViernes 7y ago
Are you arguing that in the initial stages incorrect information is less harmful than no information?
- IAmEveryone 7y ago
  No, they are not arguing that. I can tell because their post says something else, and because that would be stupid.
- occamrazor 7y ago
  In March there wasn’t enough information to decide how strict the social distancing needs to be in order to reduce the spread of the infection. At the same time it was clearly necessary to do something, without knowing in advance how much it would affect the epidemic. Now we can take a little more time to decide what to do next, and take more focused and effective measures.
- bo1024 7y ago
  There’s a difference between incorrect and incomplete. Facts, like how many people in a trial of 23 recovered, can be shared and useful early on even though they’re incomplete. That’s different from attempting to draw a strong conclusion about what works from n=23, that might be an incorrect conclusion.
  - mattkrause 7y ago
    The form that the incomplete data took was part of the problem. Adding numbers to a spreadsheet is one thing. When the same data takes the form of an NJEM or Lancet paper, it definitely implies--and some cases, outright says--more than "FYI: 37/53 people got better when we did [this]." Part of this is how papers are ordinarily written, but I think the authors (and editors) should have stopped all over the word "improvement" in those manuscripts.

kqr2 7y ago

Not only research, but a lot of diagnostic companies are trying to take advantage of the gold rush: Theranos Would Be Thriving in the Covid-19 Pandemic https://elemental.medium.com/theranos-would-be-thriving-in-the-covid-19-pandemic-63ec6d45c96a https://elemental.medium.com/theranos-would-be-thriving-in-t...

bo1024 7y ago

Really interesting question here. I wish it went a bit more into specifics of how to navigate the tradeoffs. Maybe one way to put this is that it’s fine to do preliminary studies that motivate and direct the actual rigorous research, but we’re stuck in a place where people are only doing the preliminary studies, a lot, and not getting to the rigorous stage.

pmiller2 7y ago

What happened to the title? The actual title is "Against Pandemic Research Exceptionalism". If this was intentional, it was a bad choice, because information was lost.

hartator 7y ago

> perpetuate the perception that, when it comes to the rigors of science, crisis situations demand exceptions to high standards for quality. Where are the high quality studies warranting strict lockdowns?

pnako 7y ago
In fact I would be curious if a single medical textbook anywhere in the world mentions this as a possible solution to any epidemic. It seems to be a innovation, coming from politicians more than scientists. The practice so far, still uncommon, was to quarantine sick people, not force the entire population to stay at home.