7 ms·

Thank you for extracting the actual numbers. Constructing underpowered surveys with loaded questions is a common tactic for pushing narratives like this. The he

Thank you for extracting the actual numbers. Constructing underpowered surveys with loaded questions is a common tactic for pushing narratives like this. The headline confirms a lot of people’s beliefs that economic conditions are forcing people out of retirement and that remote work is the way of the future, but it really just a survey of a couple dozen people who are “considering” and “wanting” something, which ultimately has little bearing on what people actually choose to do. These underpowered survey tactics are most frequently used in “__% of Americans are living paycheck to paycheck” headlines. That’s why all of those headlines have drastically different numbers, some as high as 70-80% (which is obviously untrue). It really just means that someone was able to get a few random people on the phone to answer some questions in the middle of the day, which isn’t representative of the population at all.

tqi 4y ago
I think the standard error for this survey is only like +/- 3%, which imo doesn't change the narrative THAT much. That said, without more details on the survey design, it's hard not to think issues like sampling and response bias make the results difficult to generalize...
hammock 4y ago
I work in quantitative research. A cell with 13 or 25 respondents in it ought to be used with caution, but is not necessarily underpowered. My "rule of thumb" is that, given a well-represented and properly weighted sample, having 30+ in a given cell when crosstabbing like this is healthy, and 10+ per cell is the bare minimum I will accept - with caution.
- Retric 4y ago
  Well-represented and properly weighted samples are unusual. As often as it’s used, phone surveys don’t qualify.
kdfjgbdfkjgb 4y ago
why is it obvious that that is untrue?
- SantalBlush 4y ago
  On this site, the null hypothesis is whatever we already believe. When we see evidence of an alternative hypothesis, we find ways to throw that evidence in the trash. Small sample sizes, replication crisis, misleading headlines, research on mice, etc. are all ways to ignore whatever evidence we want. In short, we are always right, research be damned.
  - deleted 4y ago
    [deleted]
  - tqi 4y ago
    There is evidence that people are pretty good at predicting whether or not a study is going to replicate, so they're not necessarily off Ba(ye)se: https://www.theatlantic.com/science/archive/2018/08/scientists-can-collectively-sense-which-psychology-studies-are-weak/568630/ https://www.theatlantic.com/science/archive/2018/08/scientis... https://www.newscientist.com/article/2218843-common-sense-can-predict-if-a-psychology-study-will-ever-be-replicated/ https://www.newscientist.com/article/2218843-common-sense-ca... (in this particular case, I think the fact that this is content marketing is what is causing people to react so skeptically)
    - SantalBlush 4y ago
      This pretty much highlights what I'm saying. Regardless of how strong or weak this research is, it won't be scrutinized by viewers here, because it confirms what they want to believe.
- michaelt 4y ago
  In the parlance of my country, if someone is "living paycheck to paycheck" they're a member of the working poor, people with no savings, desperately fighting to keep their head above water. Here is a survey saying 36% of Americans making >$200,000 are living paycheck to paycheck: https://www.lendingclub.com/asset/ls/cms/media-center/12th-paycheck-to-paycheck-report.pdf?v=7499f609-777c-4fc4-89a8-b50ed1ccb8d2 https://www.lendingclub.com/asset/ls/cms/media-center/12th-p... There might be a handful of people earning that much who are in a difficult financial position - divorced airline pilots with expensive alimony payments, perhaps? - but it is extremely hard to believe that 36% of people earning 4x the US median income are the working poor. On the other hand, the 36% number would make much more sense if the survey had asked people "Do you depend on your salary to pay for your living expenses?" or "Do you have enough savings to pay for 3-6 months of basic living costs?" i.e. much more expansive definitions of what it means to live paycheck-to-paycheck.
spywaregorilla 4y ago
No, you just don't understand how statistics works. If you get 80% on sample of 100 people, your 95% confidence interval is +/- 8%. This assumes a random sampling method.
- deleted 4y ago
  [deleted]
- kristjansson 4y ago
  > This assumes a random sampling method. I would encourage you to explore how difficult it is to actually get 100 responses from 100 randomly selected Americans, and how non-response/self-selection affect your confidence interval.
  - spywaregorilla 4y ago
    It is difficult. That is irrelevant to the bad logic I was responding to.
raisedbyninjas 4y ago
I think this discounts the finding that 575 people, the majority, were retirees that already returned to the workforce. This 2011 study said the rate for this was 15%. https://www.bls.gov/opub/mlr/2011/article/reentering-the-labor-force-after-retirement.htm https://www.bls.gov/opub/mlr/2011/article/reentering-the-lab...
- kgwgk 4y ago
  That's not a "finding". It was their choice to survey mostly retired people who had returned to work.
kristjansson 4y ago
IMO it's less about power than the self-selection of the responders. Naive MoE estimates that assume random sampling completely miss that respondents are just a difference species, esp. w.r.t. financial attributes. It would be shocking if some large fraction of high-earners really were living paycheck-to-paycheck. That a large fraction people who chose to respond to a survey and who stated they earn $xxxK/yr also said they're living paycheck-to-paycheck is ... somewhat less surprising. There should be half a dozen "among survey respondents" comments to headlines like this, the same way there are a dozen "in mice" comments to every breathless report of medical progress in this-or-that.

tqi 4y ago

I think the standard error for this survey is only like +/- 3%, which imo doesn't change the narrative THAT much. That said, without more details on the survey design, it's hard not to think issues like sampling and response bias make the results difficult to generalize...

hammock 4y ago

I work in quantitative research. A cell with 13 or 25 respondents in it ought to be used with caution, but is not necessarily underpowered. My "rule of thumb" is that, given a well-represented and properly weighted sample, having 30+ in a given cell when crosstabbing like this is healthy, and 10+ per cell is the bare minimum I will accept - with caution.

Retric 4y ago
Well-represented and properly weighted samples are unusual. As often as it’s used, phone surveys don’t qualify.

kdfjgbdfkjgb 4y ago

why is it obvious that that is untrue?

SantalBlush 4y ago
On this site, the null hypothesis is whatever we already believe. When we see evidence of an alternative hypothesis, we find ways to throw that evidence in the trash. Small sample sizes, replication crisis, misleading headlines, research on mice, etc. are all ways to ignore whatever evidence we want. In short, we are always right, research be damned.
- deleted 4y ago
  [deleted]
- tqi 4y ago
  There is evidence that people are pretty good at predicting whether or not a study is going to replicate, so they're not necessarily off Ba(ye)se: https://www.theatlantic.com/science/archive/2018/08/scientists-can-collectively-sense-which-psychology-studies-are-weak/568630/ https://www.theatlantic.com/science/archive/2018/08/scientis... https://www.newscientist.com/article/2218843-common-sense-can-predict-if-a-psychology-study-will-ever-be-replicated/ https://www.newscientist.com/article/2218843-common-sense-ca... (in this particular case, I think the fact that this is content marketing is what is causing people to react so skeptically)
  - SantalBlush 4y ago
    This pretty much highlights what I'm saying. Regardless of how strong or weak this research is, it won't be scrutinized by viewers here, because it confirms what they want to believe.
michaelt 4y ago
In the parlance of my country, if someone is "living paycheck to paycheck" they're a member of the working poor, people with no savings, desperately fighting to keep their head above water. Here is a survey saying 36% of Americans making >$200,000 are living paycheck to paycheck: https://www.lendingclub.com/asset/ls/cms/media-center/12th-paycheck-to-paycheck-report.pdf?v=7499f609-777c-4fc4-89a8-b50ed1ccb8d2 https://www.lendingclub.com/asset/ls/cms/media-center/12th-p... There might be a handful of people earning that much who are in a difficult financial position - divorced airline pilots with expensive alimony payments, perhaps? - but it is extremely hard to believe that 36% of people earning 4x the US median income are the working poor. On the other hand, the 36% number would make much more sense if the survey had asked people "Do you depend on your salary to pay for your living expenses?" or "Do you have enough savings to pay for 3-6 months of basic living costs?" i.e. much more expansive definitions of what it means to live paycheck-to-paycheck.

spywaregorilla 4y ago

No, you just don't understand how statistics works. If you get 80% on sample of 100 people, your 95% confidence interval is +/- 8%. This assumes a random sampling method.

deleted 4y ago
[deleted]
kristjansson 4y ago
> This assumes a random sampling method. I would encourage you to explore how difficult it is to actually get 100 responses from 100 randomly selected Americans, and how non-response/self-selection affect your confidence interval.
- spywaregorilla 4y ago
  It is difficult. That is irrelevant to the bad logic I was responding to.

raisedbyninjas 4y ago

I think this discounts the finding that 575 people, the majority, were retirees that already returned to the workforce. This 2011 study said the rate for this was 15%. https://www.bls.gov/opub/mlr/2011/article/reentering-the-labor-force-after-retirement.htm https://www.bls.gov/opub/mlr/2011/article/reentering-the-lab...

kgwgk 4y ago
That's not a "finding". It was their choice to survey mostly retired people who had returned to work.

kristjansson 4y ago

IMO it's less about power than the self-selection of the responders. Naive MoE estimates that assume random sampling completely miss that respondents are just a difference species, esp. w.r.t. financial attributes. It would be shocking if some large fraction of high-earners really were living paycheck-to-paycheck. That a large fraction people who chose to respond to a survey and who stated they earn $xxxK/yr also said they're living paycheck-to-paycheck is ... somewhat less surprising. There should be half a dozen "among survey respondents" comments to headlines like this, the same way there are a dozen "in mice" comments to every breathless report of medical progress in this-or-that.