5 ms·

The conclusion is entirely not based on results from this study, if it can be called a proper study. What can be told from the abstract, is that sample size is

The conclusion is entirely not based on results from this study, if it can be called a proper study. What can be told from the abstract, is that sample size is extremely small (19 individuals), there don't seem to be any control parameters and the distinction of "lonely" and "not lonely" is made based on self reporting, which is yet another source of potential errors. Other than that, the subject studied was reaction speed, which can be interpreted as basic alertness, and even if we consider the results reliable, which we can't, the conclusion could be made almost entirely different - lonely people are more alert to their surroundings, while non-lonely are more calm and relaxed. I'm not sure who are reviewing these studies, but it would never pass as a conclusive study by any standards in non-social sciences.

Osmium 11y ago
> is that sample size is extremely small (19 individuals) This is not meaningful. I'm not sure why this keeps getting repeated, but it's entirely possible to get significant results from small sample sizes. No comment on the rest of the study, but I just don't think sample sizes are worth mentioning, and I wouldn't even call that "extremely small". This is what significance testing is for, after all. With regards to self-reporting, I read a very nice defence on this in a book on the scientific study of consciousness [1], which basically says that we need to let go of this idea that self-reporting is somehow "unscientific", because some phenomena only exist as experiences (and I'd think loneliness is one of them), and therefore the scientific study of these experiences has to incorporate self-reporting on some level. You can't get a blood test for loneliness. [1] https://en.wikipedia.org/wiki/Consciousness_and_the_Brain https://en.wikipedia.org/wiki/Consciousness_and_the_Brain
- wpietri 11y ago
  I don't think self-reporting is unscientific, but I do think it's not particularly reliable. Reported loneliness may correlate with actual loneliness, but they're different phenomena. This was really driven home to me in doing product development and user testing. I've worked with some great user researchers, and they were extremely skeptical about self-reported characteristics. The classic example is, "Would you buy X?" People are terrible at answering that correctly and honestly. But there were a bunch more where we carefully rigged situations so we could measure behavior rather than self-reported internal state, because behavior ended up being a better measure of internal state for us. I could well believe the same is true for loneliness. I'd also disagree that some phenomena only exist as experiences. As far as we know, all experiences are also physical states in the brain. We currently may not have the tools to usefully read that physical state. We may not have a blood test for loneliness, but we may one day have a brain scan for it. Or, perhaps sooner, a pattern in the sort of data that will be gathered by the iPhone 15m (the "m" being for medical, of course).
  - JoeAltmaier 11y ago
    This! We find focus groups nearly useless; folks are so eager to please. You get nothing but "Yes this is great. Maybe change the color to yellow". But you learn nothing about what would really appeal or sell. I'm thinking focus groups should change to some sort of psychological test situation. "You can take only one of these home for free. Which one do you want?" I.e. perceived scarcity (cost) would elicit honest responses.
    - wpietri 11y ago
      Yeah, I have never found focus groups of any value. I don't understand how one separates social effects from actual effects. And many user researchers agree. E.g.: https://medium.com/research-things/focus-groups-are-worthless-7d30891e58f1 https://medium.com/research-things/focus-groups-are-worthles...
    - bcbrown 11y ago
      User focus groups may not be useful, but user testing can be. I'm thinking of studies where you give users some tasks to do, and watch how they interact with the UI to accomplish that.
  - Osmium 11y ago
    > I'd also disagree that some phenomena only exist as experiences. As far as we know, all experiences are also physical states in the brain. Absolutely. But in order to map those experiences to those 'physical states' you have to trust self-reporting. I agree with you that questions such as "Would you buy X?" are incredibly unreliable when it comes to self-reporting, so I understand the skepticism, but I don't think loneliness falls into that category, because it's not something that can be externally tested (we can't observe the person to see if they're "truly" lonely or not).
    - wpietri 11y ago
      Hmm. I'm not following your logic. It sounds like you're saying that reported loneliness isn't unreliable because we can't externally test it. I'd think it's just the opposite; if we have no external tests, that's a great reason to assume it's unreliable. That said, I'm not as confident as you that it's not externally testable. These researchers, for example, believe they have found EEG test results that correlate with loneliness. It would seem to me that loneliness could change both self-generated behavior and response to stimuli, which could yield reliable measures. Sociality is a biologically mediated phenomenon, so there could be metabolic measures, like how cortisol is an indicator of stress. We expect loneliness to happen in the brain, so aside from EEG measures, perhaps we could find an FMRI test that exposes it.
- roymurdock 11y ago
  Can you elaborate on why small sample sizes are not a cause for concern? It's my understanding that when designing a study, you look to strengthen statistical power, which is determined by a combination of a) significance criterion (p values, confidence intervals) b) magnitude of the effect and c) sample size. Increasing the sample size decreases sampling error. Just from a common-sense perspective, if the sample of the study was n=2, then any results derived from significance testing would be nonsensical - how can we extrapolate our findings to a broader population? Is n=19 good enough?
  - Osmium 11y ago
    > It's my understanding that when designing a study, you look to strengthen statistical power, which is determined by a combination of a) significance criterion (p values, confidence intervals) b) magnitude of the effect and c) sample size. Increasing the sample size decreases sampling error. I don't disagree with this at all. You always want to increase your sample size if possible. But as you also say, the magnitude of the effect you're observing is also important; if it's very strong, then it can be revealed by a small sample size. An absurd example could be "cyanide capsules kill (n=2)". Obviously, the smaller the sample size, the harder it is to draw reliable conclusions, but I wouldn't think n=19 is necessarily too small. I think many of phase I clinical trials aren't much more than that. More importantly, mentioning a "small" sample size makes a nice soundbite, and is often used to discredit studies. These studies might be good or they might not be, but talking about sample size can be used as a "thought-terminating cliche". Perhaps a person is looking for evidence that reinforces their own beliefs (we all do it), and they read a criticism of a study, it mentions "small sample sizes", and they can walk away happy in a belief the study was flawed. But all that's really happened is that it's prevented a proper discussion of the study taking place. I'm not sure how much of a problem this is, but I've seen it happen frequently.
    - roymurdock 11y ago
      Got it, thanks for the explanation. I generally trust that the authors of such studies know what they're doing when they design an experiment with a certain n value in mind. Agreed that the insufficient n argument does crop up a lot around HN.
- cLeEOGPw 11y ago
  > ... but it's entirely possible to get significant results from small sample sizes. It is also possible to simply guess some effect. You don't need study for stating a result. A purpose of study is to prove something. A study that can not be told to be proven and is not replicable is not worth anything, as further scientific knowledge cannot be built on top of it. If a study do not prove or at least strengthen confidence in a theory, what is it worth? Not even meta-analysis will consider studies like these to prove some entirely different theory because it is just that unreliable. Now I'm not saying this particular study is entirely like that, but the flaws everyone mentioned are concerning.
- deleted 11y ago
  [deleted]

Osmium 11y ago

> is that sample size is extremely small (19 individuals) This is not meaningful. I'm not sure why this keeps getting repeated, but it's entirely possible to get significant results from small sample sizes. No comment on the rest of the study, but I just don't think sample sizes are worth mentioning, and I wouldn't even call that "extremely small". This is what significance testing is for, after all. With regards to self-reporting, I read a very nice defence on this in a book on the scientific study of consciousness [1], which basically says that we need to let go of this idea that self-reporting is somehow "unscientific", because some phenomena only exist as experiences (and I'd think loneliness is one of them), and therefore the scientific study of these experiences has to incorporate self-reporting on some level. You can't get a blood test for loneliness. [1] https://en.wikipedia.org/wiki/Consciousness_and_the_Brain https://en.wikipedia.org/wiki/Consciousness_and_the_Brain

wpietri 11y ago
I don't think self-reporting is unscientific, but I do think it's not particularly reliable. Reported loneliness may correlate with actual loneliness, but they're different phenomena. This was really driven home to me in doing product development and user testing. I've worked with some great user researchers, and they were extremely skeptical about self-reported characteristics. The classic example is, "Would you buy X?" People are terrible at answering that correctly and honestly. But there were a bunch more where we carefully rigged situations so we could measure behavior rather than self-reported internal state, because behavior ended up being a better measure of internal state for us. I could well believe the same is true for loneliness. I'd also disagree that some phenomena only exist as experiences. As far as we know, all experiences are also physical states in the brain. We currently may not have the tools to usefully read that physical state. We may not have a blood test for loneliness, but we may one day have a brain scan for it. Or, perhaps sooner, a pattern in the sort of data that will be gathered by the iPhone 15m (the "m" being for medical, of course).
- JoeAltmaier 11y ago
  This! We find focus groups nearly useless; folks are so eager to please. You get nothing but "Yes this is great. Maybe change the color to yellow". But you learn nothing about what would really appeal or sell. I'm thinking focus groups should change to some sort of psychological test situation. "You can take only one of these home for free. Which one do you want?" I.e. perceived scarcity (cost) would elicit honest responses.
  - wpietri 11y ago
    Yeah, I have never found focus groups of any value. I don't understand how one separates social effects from actual effects. And many user researchers agree. E.g.: https://medium.com/research-things/focus-groups-are-worthless-7d30891e58f1 https://medium.com/research-things/focus-groups-are-worthles...
  - bcbrown 11y ago
    User focus groups may not be useful, but user testing can be. I'm thinking of studies where you give users some tasks to do, and watch how they interact with the UI to accomplish that.
- Osmium 11y ago
  > I'd also disagree that some phenomena only exist as experiences. As far as we know, all experiences are also physical states in the brain. Absolutely. But in order to map those experiences to those 'physical states' you have to trust self-reporting. I agree with you that questions such as "Would you buy X?" are incredibly unreliable when it comes to self-reporting, so I understand the skepticism, but I don't think loneliness falls into that category, because it's not something that can be externally tested (we can't observe the person to see if they're "truly" lonely or not).
  - wpietri 11y ago
    Hmm. I'm not following your logic. It sounds like you're saying that reported loneliness isn't unreliable because we can't externally test it. I'd think it's just the opposite; if we have no external tests, that's a great reason to assume it's unreliable. That said, I'm not as confident as you that it's not externally testable. These researchers, for example, believe they have found EEG test results that correlate with loneliness. It would seem to me that loneliness could change both self-generated behavior and response to stimuli, which could yield reliable measures. Sociality is a biologically mediated phenomenon, so there could be metabolic measures, like how cortisol is an indicator of stress. We expect loneliness to happen in the brain, so aside from EEG measures, perhaps we could find an FMRI test that exposes it.
roymurdock 11y ago
Can you elaborate on why small sample sizes are not a cause for concern? It's my understanding that when designing a study, you look to strengthen statistical power, which is determined by a combination of a) significance criterion (p values, confidence intervals) b) magnitude of the effect and c) sample size. Increasing the sample size decreases sampling error. Just from a common-sense perspective, if the sample of the study was n=2, then any results derived from significance testing would be nonsensical - how can we extrapolate our findings to a broader population? Is n=19 good enough?
- Osmium 11y ago
  > It's my understanding that when designing a study, you look to strengthen statistical power, which is determined by a combination of a) significance criterion (p values, confidence intervals) b) magnitude of the effect and c) sample size. Increasing the sample size decreases sampling error. I don't disagree with this at all. You always want to increase your sample size if possible. But as you also say, the magnitude of the effect you're observing is also important; if it's very strong, then it can be revealed by a small sample size. An absurd example could be "cyanide capsules kill (n=2)". Obviously, the smaller the sample size, the harder it is to draw reliable conclusions, but I wouldn't think n=19 is necessarily too small. I think many of phase I clinical trials aren't much more than that. More importantly, mentioning a "small" sample size makes a nice soundbite, and is often used to discredit studies. These studies might be good or they might not be, but talking about sample size can be used as a "thought-terminating cliche". Perhaps a person is looking for evidence that reinforces their own beliefs (we all do it), and they read a criticism of a study, it mentions "small sample sizes", and they can walk away happy in a belief the study was flawed. But all that's really happened is that it's prevented a proper discussion of the study taking place. I'm not sure how much of a problem this is, but I've seen it happen frequently.
  - roymurdock 11y ago
    Got it, thanks for the explanation. I generally trust that the authors of such studies know what they're doing when they design an experiment with a certain n value in mind. Agreed that the insufficient n argument does crop up a lot around HN.
cLeEOGPw 11y ago
> ... but it's entirely possible to get significant results from small sample sizes. It is also possible to simply guess some effect. You don't need study for stating a result. A purpose of study is to prove something. A study that can not be told to be proven and is not replicable is not worth anything, as further scientific knowledge cannot be built on top of it. If a study do not prove or at least strengthen confidence in a theory, what is it worth? Not even meta-analysis will consider studies like these to prove some entirely different theory because it is just that unreliable. Now I'm not saying this particular study is entirely like that, but the flaws everyone mentioned are concerning.
deleted 11y ago
[deleted]