3 ms·
I don't think self-reporting is unscientific, but I do think it's not particularly reliable. Reported loneliness may correlate with actual loneliness, but they'
by wpietri 11y ago
I don't think self-reporting is unscientific, but I do think it's not particularly reliable. Reported loneliness may correlate with actual loneliness, but they're different phenomena.
This was really driven home to me in doing product development and user testing. I've worked with some great user researchers, and they were extremely skeptical about self-reported characteristics. The classic example is, "Would you buy X?" People are terrible at answering that correctly and honestly. But there were a bunch more where we carefully rigged situations so we could measure behavior rather than self-reported internal state, because behavior ended up being a better measure of internal state for us. I could well believe the same is true for loneliness.
I'd also disagree that some phenomena only exist as experiences. As far as we know, all experiences are also physical states in the brain. We currently may not have the tools to usefully read that physical state. We may not have a blood test for loneliness, but we may one day have a brain scan for it. Or, perhaps sooner, a pattern in the sort of data that will be gathered by the iPhone 15m (the "m" being for medical, of course).
- JoeAltmaier 11y agoThis! We find focus groups nearly useless; folks are so eager to please. You get nothing but "Yes this is great. Maybe change the color to yellow". But you learn nothing about what would really appeal or sell. I'm thinking focus groups should change to some sort of psychological test situation. "You can take only one of these home for free. Which one do you want?" I.e. perceived scarcity (cost) would elicit honest responses.
- wpietri 11y agoYeah, I have never found focus groups of any value. I don't understand how one separates social effects from actual effects. And many user researchers agree. E.g.: https://medium.com/research-things/focus-groups-are-worthless-7d30891e58f1 https://medium.com/research-things/focus-groups-are-worthles...
- bcbrown 11y agoUser focus groups may not be useful, but user testing can be. I'm thinking of studies where you give users some tasks to do, and watch how they interact with the UI to accomplish that.
- Osmium 11y ago> I'd also disagree that some phenomena only exist as experiences. As far as we know, all experiences are also physical states in the brain. Absolutely. But in order to map those experiences to those 'physical states' you have to trust self-reporting. I agree with you that questions such as "Would you buy X?" are incredibly unreliable when it comes to self-reporting, so I understand the skepticism, but I don't think loneliness falls into that category, because it's not something that can be externally tested (we can't observe the person to see if they're "truly" lonely or not).
- wpietri 11y agoHmm. I'm not following your logic. It sounds like you're saying that reported loneliness isn't unreliable because we can't externally test it. I'd think it's just the opposite; if we have no external tests, that's a great reason to assume it's unreliable. That said, I'm not as confident as you that it's not externally testable. These researchers, for example, believe they have found EEG test results that correlate with loneliness. It would seem to me that loneliness could change both self-generated behavior and response to stimuli, which could yield reliable measures. Sociality is a biologically mediated phenomenon, so there could be metabolic measures, like how cortisol is an indicator of stress. We expect loneliness to happen in the brain, so aside from EEG measures, perhaps we could find an FMRI test that exposes it.
- Osmium 11y ago> It sounds like you're saying that reported loneliness isn't unreliable because we can't externally test it. I'd think it's just the opposite; if we have no external tests, that's a great reason to assume it's unreliable. I'm not disagreeing with you on that--there's always the possibility someone's lying to you, and that has to be designed into your study as best as possible. It's difficult. My point is more that we don't have any other choice if we want to do a scientific study of things like loneliness. Or anything else that only exists within the context of one's own head for that matter. > That said, I'm not as confident as you that it's not externally testable. Perhaps it is. I'm only wary of these things because it's very easy to find correlations. Studies of depression are full of this. You can find a certain brain region or gene or hormone level or whatever correlated with depression, but then you find someone who also feels depressed who doesn't exhibit those symptoms. So do you tell that person they're not "really" feeling depressed? Of course not. And fMRI studies are open to abuse too; I'm sure you've heard of the "dead salmon" study, but if not: http://www.wired.com/2009/09/fmrisalmon/ http://www.wired.com/2009/09/fmrisalmon/ I'm actually optimistic, as you are, that there will be better diagnostic tests for a whole range of mental conditions/experiences/illnesses/etc., but the core criticism still holds: if you want to trust a diagnostic test of a mental state, ultimately, at some point, you will also have needed to trust the self-reporting of that mental state.
- JackFr 11y agoAs part of an ongoing study I have been conducting my whole life, I have found self-reported cleverness, self-reported attractiveness, and self-reported indications that "the guy over there wants fight me" to all be positively correlated with alcohol intake.