3 ms·
What is your concern exactly? This was a first-pass study in a field addressing some of the criticisms leveraged against an earlier study where the spatial rea
by svnt 3y ago
What is your concern exactly?
This was a first-pass study in a field addressing some of the criticisms leveraged against an earlier study where the spatial reasoning problems were viewed to be too hard. They seemingly made the spatial reasoning questions as easy as they could.
The qualifications they put on MTurk are pretty standard if you want humans who care about what they are doing responding to your study. It costs more to do this.
It is a limitation of science that is both budgetary and procedural.
By calling into question their results you seem to be suggesting that an average human would only 33% of the time be able to tell e.g. how many points are inside a box, or whether more points are inside or outside of a box. This is extremely basic spatial reasoning we are talking about.
The problem they were addressing with the settings is just noise in the results by cheap bots and clicky humans trying to earn $0.50. It is endemic on MTurk.
- rileyphone 3y agoIf there are endemic issues using Mechanical Turk, maybe important studies shouldn’t use it, and those that do should be called into question.
- svnt 3y agoIf you have another way to provide actual humans with arbitrary representative demographics for answering ten minute surveys for a total cost of around $1 each the market would love to hear about it. Alternatively if you wanted to fund researchers and studies so they could afford to do these studies in a better way, I don’t think anyone would complain.
- kbenson 3y ago> By calling into question their results Did they? I took it more as a clarification of a point so people could more easily know what it meant, and wishing they did more so we could know more. Their last sentence in fact notes that what the paper says is correct and they just which we could make a more nuanced statement if it was done with more specific constraints.