4 ms·

Regarding survey's I don't necessarily disagree with surveys (although they can be flawed), but find that mechanical turk seems to give the worst way of doing a

Regarding survey's I don't necessarily disagree with surveys (although they can be flawed), but find that mechanical turk seems to give the worst way of doing a survey. It rewards going through the survey as quick as possible and I would also argue attracts people who optimise toward time (although I don't have hard evidence), not necessarily honesty (an interesting ethics question in itself). Regarding your thoughts on the car study I broadly agree it is not a very good proxy for wealth (also areas with many company cars likely skew results as well). However I do think that it's trying to independently verify peoples self-evaluation using similar experiments what gives us much better insights and there were a number of other studies with similar results. Confirmation bias is always a danger and the paper that was described in the article which could not reproduce the correlation between car cost and behavior tried to follow best practices to minimise it, so probably should be given more weight. I actually wondered if there is a difference also according to culture, i.e. Piff's results were obtained in the US, while the other article was in the Netherlands.

bumby 3y ago
Yeah, I agree. I think the MTurk is not a great approach. Researchers rely on it too much because it's easier to implement than other options. On the other hand, if I were a reviewer of the Piff article, I personally wouldn't have let it pass peer-review as-is. The entire article is based on an unvalidated wealth model. I see this quite a bit with research that comes for peer review: 1) We have this model, 2) Look at all these neat findings we extrapolate from this model They are missing the intermediate step that shows the model measures what it's supposed to. In this case, they never validate that car make/model is a good proxy for wealth. The entire premise is based on a book titled "Luxury Fever"[1] and not on peer-reviewed research. They never actually explain this model, either. What cars are mapped to their 1-5 scale? Is a Lincoln a 5 or a Ferrari? No idea. There's contradictory evidence that many wealthy people drive Hondas and Fords and that less-wealthy people buy luxury cars as a pseudo-status symbol. That's why symbols of wealth (clothes, cars, etc.) are a bad model for actual wealth, especially in a society with a large debt burden and when the models go unvalidated. There are other questions I would have, like small-ish sample sizing, to basing the entire study in Berkeley, to only doing the traffic study between 2-5pm. I would question how well that study can be extrapolated. If you do a quick Bayesian statistical significance test on their vehicle data, it doesn't appear statistically significant. But they report correlations instead. All that to say, there's a lot of contradictory evidence on both sides. Social science is hard, and I was trying to make the main point that we need to be very careful about making strong and broad claims from limited evidence. And I say this as someone who drives a bargain-basement hatchback with mechanical windows :-)

bumby 3y ago

Yeah, I agree. I think the MTurk is not a great approach. Researchers rely on it too much because it's easier to implement than other options. On the other hand, if I were a reviewer of the Piff article, I personally wouldn't have let it pass peer-review as-is. The entire article is based on an unvalidated wealth model. I see this quite a bit with research that comes for peer review: 1) We have this model, 2) Look at all these neat findings we extrapolate from this model They are missing the intermediate step that shows the model measures what it's supposed to. In this case, they never validate that car make/model is a good proxy for wealth. The entire premise is based on a book titled "Luxury Fever"[1] and not on peer-reviewed research. They never actually explain this model, either. What cars are mapped to their 1-5 scale? Is a Lincoln a 5 or a Ferrari? No idea. There's contradictory evidence that many wealthy people drive Hondas and Fords and that less-wealthy people buy luxury cars as a pseudo-status symbol. That's why symbols of wealth (clothes, cars, etc.) are a bad model for actual wealth, especially in a society with a large debt burden and when the models go unvalidated. There are other questions I would have, like small-ish sample sizing, to basing the entire study in Berkeley, to only doing the traffic study between 2-5pm. I would question how well that study can be extrapolated. If you do a quick Bayesian statistical significance test on their vehicle data, it doesn't appear statistically significant. But they report correlations instead. All that to say, there's a lot of contradictory evidence on both sides. Social science is hard, and I was trying to make the main point that we need to be very careful about making strong and broad claims from limited evidence. And I say this as someone who drives a bargain-basement hatchback with mechanical windows :-)