3 ms·

I think part of their intent was to show biases in judgement. The small sample size really hindered this though. Apparently I 100% preferred old people to young

I think part of their intent was to show biases in judgement. The small sample size really hindered this though. Apparently I 100% preferred old people to young people, even though I didn't consider age in my decisions at all. They do have a little disclaimer on the results screen about the sample size though.

macawfish 10y ago
The bias is generated by the lack of real choice in responses. They are offering an unrepresentative range of choices, then going and telling you that your choices are biased. They're not showing existing bias, they're coercing it out of participants. This thing irritated me so bad that I couldn't bring myself to answer a single question. I can't stand the idea that my this study is actually happening. If the results are published and receive non-critical media attention, I'm gonna be irate.
- esfandia 10y ago
  But then you'd need to answer 100 or more test questions instead of 13. This is necessarily simplistic given how much involvement they can ask of participants. Also, I don't think they're implying anything nefarious about the resulting biases shown in the final results. For sure a lot of it has to do with the small sample size of the questions. I didn't run the tests twice, but I imagine there is some randomness involved in the way they are generated. If there are any heuristics at play, the results will indeed show them (in my case there were enough tests to recover the fact that I preferred saving passengers, preferred non-intervention, and preferred saving humans over pets). But it will also come up with some gibberish/noise due to the small sample size.
  - JoshTriplett 10y ago
    > But then you'd need to answer 100 or more test questions instead of 13. Or they could design questions designed to more definitively separate different hypotheses in fewer questions.
  - macawfish 10y ago
    It doesn't matter how big the sample size is if the questions themselves are biased.
fela 10y ago
On a single participant yes, but their real goal, I presume, is to aggregate the data and then they will be able to reach more concrete conclusions.

macawfish 10y ago

The bias is generated by the lack of real choice in responses. They are offering an unrepresentative range of choices, then going and telling you that your choices are biased. They're not showing existing bias, they're coercing it out of participants. This thing irritated me so bad that I couldn't bring myself to answer a single question. I can't stand the idea that my this study is actually happening. If the results are published and receive non-critical media attention, I'm gonna be irate.

esfandia 10y ago
But then you'd need to answer 100 or more test questions instead of 13. This is necessarily simplistic given how much involvement they can ask of participants. Also, I don't think they're implying anything nefarious about the resulting biases shown in the final results. For sure a lot of it has to do with the small sample size of the questions. I didn't run the tests twice, but I imagine there is some randomness involved in the way they are generated. If there are any heuristics at play, the results will indeed show them (in my case there were enough tests to recover the fact that I preferred saving passengers, preferred non-intervention, and preferred saving humans over pets). But it will also come up with some gibberish/noise due to the small sample size.
- JoshTriplett 10y ago
  > But then you'd need to answer 100 or more test questions instead of 13. Or they could design questions designed to more definitively separate different hypotheses in fewer questions.
- macawfish 10y ago
  It doesn't matter how big the sample size is if the questions themselves are biased.

fela 10y ago

On a single participant yes, but their real goal, I presume, is to aggregate the data and then they will be able to reach more concrete conclusions.