7 ms·

One interesting issue with plausible effect sizes is that they are very context dependent. During a graduate seminar, someone asked for a show of hands for peo

by Fomite 5y ago

One interesting issue with plausible effect sizes is that they are very context dependent. During a graduate seminar, someone asked for a show of hands for people who thought a relative risk of 1.10 (i.e. the exposed are 10% more likely to develop a disease than the unexposed) was a big deal. Very few of the infectious disease epidemiologists raised their hands. We are often used to double-digit effect sizes for our risk factors (HPV and cervical cancer, a particular exposure in a foodborne outbreak, etc.) The cancer epidemiology folks were like "Yeah, that's a pretty big deal, and worth digging into." The environmental epidemiology folks were like "Are you kidding? There are going to be lawsuits. Congressional hearings. Maybe arrests."

HWR_14 5y ago
It's true that effect sizes are very context dependent. Heck, you see it in technology as well. Sometimes, it's worth paying someone to spend a week to optimize an instruction or two out of code. Othertimes, the idea that you would spend even an extra 30 seconds looking for that optimization is heresy.
- marginalia_nu 5y ago
  That follows from Amdahls law, doesn't it? Speeding up the most time consuming part by a little is worth a lot more than speeding up a tiny part of the execution time a lot.
  - deleted 5y ago
    [deleted]
  - ohyoutravel 5y ago
    It can, but optimization can mean more than just speed.
    - marginalia_nu 5y ago
      This holds true for any optimization process. You have more money to save cutting costs by a little from your biggest expense than cutting costs by a lot from a small expense.
      - chrisweekly 5y ago
        Your 2nd sentence is very difficult to parse; not nit-picking, just hoping to inspire you to reword for clarity.
  - HWR_14 5y ago
    It doesn't follow from Amdahl's law directly. Amdahl's law sets an upper limit on how much you can gain in total efficiency from effort. But I actually was speaking about how valuable the effort is. Is it the control loop on a self-landing rocket, a rendering engine for a video game or HFT algorithms? Then the efficiency is probably very valuable in its effects. Is it an app for a dating startup? No one will ever care if you could have shaved a millisecond off the match time.
    - bluGill 5y ago
      I suspect such sites have a timer to ensure they take longer so people think it is a lot of work to find a match.
      - sgerenser 5y ago
        Like TurboTax’s fake progress bars making it look like it’s “trying hard” to get you your maximum refund: https://medium.com/the-billfold/the-progress-bars-in-turbotax-are-fake-de451979ef7d https://medium.com/the-billfold/the-progress-bars-in-turbota...
        bluGill 5y ago
        Yeah, before switching to taxact I did my taxes by hand, even itemized deductions, the computer doesn't save time and there isn't actually much money to save for most people anymore. I only do it by computer now because I once forgot to copy line 13b form 2345 to line 47d of some other form. The resulting mess was annoying to take care of, not hard, but annoying.
chmod600 5y ago
Aren't the unexposed at a risk of zero? Or was that your point?
- coldtea 5y ago
  There's no premise that you need to be exposed to develop some issue X. The comparison is between developing disease X while being exposed to something S, with developing it without being exposed. (The example the parent gives is "HPV and cervical cancer" - it's not necessary to have HPV to get the latter, so those not exposed to HPV don't get a 0% chance of that disease). E.g. say a study finds that if you're exposed to VirusX (or ShouldNotHaveBeenApprovedByFDADrugX, as this applies beyond viruses) you have 10% more chances to develop a heart arythmia than without being exposed to it (which, if I'm not mistaken, the latter should be the same as saying: "as compared to the control group").
- Fomite 5y ago
  The unexposed aren't at zero risk, but are often quite close. For example, cervical cancer without HPV infection is extremely rare, but non-zero. Or someone at the party who didn't eat the potato salad could still come down sick by another pathway. On the other hand, for things like environmental exposures, you're often looking at very small increases in risk over baseline. For example, your increased risk of asthma with a higher increase in particulate matter in the air.
- lostlogin 5y ago
  A good example of this is one’s cancer risk, versus one’s cancer risk after a radiation dose. One is non zero, and the other is non zero + X.
bottled_poe 5y ago
I suppose the above scenario is probably contrived, but it seems all of those replies fail to grasp the nature of the problem - I.e. none of them attempt to quantify the risk of doing nothing against the cost of doing something. Instead, just opinions/speculation, which is disappointing coming from a room full of doctors.
- ineptech 5y ago
  You can't listen to someone's anecdote intended to convey one thing and assume they included sufficient detail for you to determine whether it also conveys some other thing.
native_samples 5y ago
The issue here is that infectious disease epidemiologists don't think anything of a large effect size because they're all about models that constantly mis-predict huge effect sizes. They should be surprised by a large effect, but because they never update their beliefs in response to model failure, they think (predicted but unreal) effect sizes are no big deal.
- Fomite 5y ago
  That's not really true. The example I used was a very large effect size, because it's causal in a way few things are (there's a reason you can use Koch's postulates in infectious diseases, but rarely in anything else). And these are all things estimated from data, using standard, boring statistical methods like logistic regression. Dynamical systems models, which is what you're talking about, is a whole different field, and also don't inherently predict huge effect sizes - the last non-COVID one I was working on, for example, was estimating fairly small effect sizes.
derbOac 5y ago
I enjoyed the target paper in question that Gelman was posting about (as is common with Gelman, you have to wade through his comments to find the work of others buried in it). It was interesting and makes a strong case for use of more interesting controls in behavioral research. After I thought about it for a bit though, I had some concerns. I read another paper on a totally different topic where they generated a sort of negative control, and although that paper was interesting too, a problem with it was that it was really easy to argue that the negative control wasn't actually a control, that it was set up in such a way as to inadvertently induce the thing they were trying to eliminate. I kinda wondered about this paper too. They seem to assume you'll go along with the idea that how you imagine a video game character to act will induce the most extreme effect size you could imagine. But is how you report you'll act even remotely comparable to how you imagine a video game character will react? I'm not really sure. I could imagine some people being like "WTH? I have no idea" and blowing the task off, or seeing the video game character as incoherent and having any kind of behavior consistent with it. People's own behavior can be difficult to predict from what they say, so it seems like a step even further removed to put any weight into what they say a videogame character would do. I guess it's an interesting idea that deserves more attention, but in my experience, psychological controls are notoriously difficult to implement, which is why you don't see it used more often.

by Fomite 5y ago

HWR_14 5y ago

It's true that effect sizes are very context dependent. Heck, you see it in technology as well. Sometimes, it's worth paying someone to spend a week to optimize an instruction or two out of code. Othertimes, the idea that you would spend even an extra 30 seconds looking for that optimization is heresy.

marginalia_nu 5y ago
That follows from Amdahls law, doesn't it? Speeding up the most time consuming part by a little is worth a lot more than speeding up a tiny part of the execution time a lot.
- deleted 5y ago
  [deleted]
- ohyoutravel 5y ago
  It can, but optimization can mean more than just speed.
  - marginalia_nu 5y ago
    This holds true for any optimization process. You have more money to save cutting costs by a little from your biggest expense than cutting costs by a lot from a small expense.
    - chrisweekly 5y ago
      Your 2nd sentence is very difficult to parse; not nit-picking, just hoping to inspire you to reword for clarity.
- HWR_14 5y ago
  It doesn't follow from Amdahl's law directly. Amdahl's law sets an upper limit on how much you can gain in total efficiency from effort. But I actually was speaking about how valuable the effort is. Is it the control loop on a self-landing rocket, a rendering engine for a video game or HFT algorithms? Then the efficiency is probably very valuable in its effects. Is it an app for a dating startup? No one will ever care if you could have shaved a millisecond off the match time.
  - bluGill 5y ago
    I suspect such sites have a timer to ensure they take longer so people think it is a lot of work to find a match.
    - sgerenser 5y ago
      Like TurboTax’s fake progress bars making it look like it’s “trying hard” to get you your maximum refund: https://medium.com/the-billfold/the-progress-bars-in-turbotax-are-fake-de451979ef7d https://medium.com/the-billfold/the-progress-bars-in-turbota...
      - bluGill 5y ago
        Yeah, before switching to taxact I did my taxes by hand, even itemized deductions, the computer doesn't save time and there isn't actually much money to save for most people anymore. I only do it by computer now because I once forgot to copy line 13b form 2345 to line 47d of some other form. The resulting mess was annoying to take care of, not hard, but annoying.

chmod600 5y ago

Aren't the unexposed at a risk of zero? Or was that your point?

coldtea 5y ago
There's no premise that you need to be exposed to develop some issue X. The comparison is between developing disease X while being exposed to something S, with developing it without being exposed. (The example the parent gives is "HPV and cervical cancer" - it's not necessary to have HPV to get the latter, so those not exposed to HPV don't get a 0% chance of that disease). E.g. say a study finds that if you're exposed to VirusX (or ShouldNotHaveBeenApprovedByFDADrugX, as this applies beyond viruses) you have 10% more chances to develop a heart arythmia than without being exposed to it (which, if I'm not mistaken, the latter should be the same as saying: "as compared to the control group").
Fomite 5y ago
The unexposed aren't at zero risk, but are often quite close. For example, cervical cancer without HPV infection is extremely rare, but non-zero. Or someone at the party who didn't eat the potato salad could still come down sick by another pathway. On the other hand, for things like environmental exposures, you're often looking at very small increases in risk over baseline. For example, your increased risk of asthma with a higher increase in particulate matter in the air.
lostlogin 5y ago
A good example of this is one’s cancer risk, versus one’s cancer risk after a radiation dose. One is non zero, and the other is non zero + X.

bottled_poe 5y ago

I suppose the above scenario is probably contrived, but it seems all of those replies fail to grasp the nature of the problem - I.e. none of them attempt to quantify the risk of doing nothing against the cost of doing something. Instead, just opinions/speculation, which is disappointing coming from a room full of doctors.

ineptech 5y ago
You can't listen to someone's anecdote intended to convey one thing and assume they included sufficient detail for you to determine whether it also conveys some other thing.

native_samples 5y ago

The issue here is that infectious disease epidemiologists don't think anything of a large effect size because they're all about models that constantly mis-predict huge effect sizes. They should be surprised by a large effect, but because they never update their beliefs in response to model failure, they think (predicted but unreal) effect sizes are no big deal.

Fomite 5y ago
That's not really true. The example I used was a very large effect size, because it's causal in a way few things are (there's a reason you can use Koch's postulates in infectious diseases, but rarely in anything else). And these are all things estimated from data, using standard, boring statistical methods like logistic regression. Dynamical systems models, which is what you're talking about, is a whole different field, and also don't inherently predict huge effect sizes - the last non-COVID one I was working on, for example, was estimating fairly small effect sizes.

derbOac 5y ago

I enjoyed the target paper in question that Gelman was posting about (as is common with Gelman, you have to wade through his comments to find the work of others buried in it). It was interesting and makes a strong case for use of more interesting controls in behavioral research. After I thought about it for a bit though, I had some concerns. I read another paper on a totally different topic where they generated a sort of negative control, and although that paper was interesting too, a problem with it was that it was really easy to argue that the negative control wasn't actually a control, that it was set up in such a way as to inadvertently induce the thing they were trying to eliminate. I kinda wondered about this paper too. They seem to assume you'll go along with the idea that how you imagine a video game character to act will induce the most extreme effect size you could imagine. But is how you report you'll act even remotely comparable to how you imagine a video game character will react? I'm not really sure. I could imagine some people being like "WTH? I have no idea" and blowing the task off, or seeing the video game character as incoherent and having any kind of behavior consistent with it. People's own behavior can be difficult to predict from what they say, so it seems like a step even further removed to put any weight into what they say a videogame character would do. I guess it's an interesting idea that deserves more attention, but in my experience, psychological controls are notoriously difficult to implement, which is why you don't see it used more often.