4 ms·
The human baseline seems flawed. 1. There is no initial screening that would filter out garbage responses. For example, users who just pick the first answer.
by tantalor 8mo ago
The human baseline seems flawed.
1. There is no initial screening that would filter out garbage responses. For example, users who just pick the first answer.
2. They don't ask for reasoning/rationale.
- andreasgl 8mo agoI agree. I wonder what the human baseline is for ”what is 1 + 1” on Rapidata.
- rapidata 8mo agoWe try a bit harder than that my friend.
- andreasgl 8mo agoI actually didn't mean to criticize Rapidata. I just think that a forced-choice question like this begs for low-effort answers. At least the respondents should have had the opportunity to explain their reasoning, like the LLMs did.
- rapidata 8mo agoAll good ^^, its a fair point, we have come up with some fun ways to track peoples reliability over time. But the validation sets contain plenty of forced-choice questions, those that have an empirical true can be used directly to calculate a reliability, those that are subjective need to be re-asked after sometime to ensure consistency. People that don't pass thresholds would not be part of the 10'000 here. But of course. If every human was told to take 3 minutes to deeply think about it and told that its a trick question, then they most likely will all get it right. But its the same with the LLMs, if you ask them like that they will get it right most of the time. The low effort is kinda the point here.
- slongfield 8mo agoMy favorite example of this was the Pew Research study: https://www.pewresearch.org/short-reads/2024/03/05/online-opt-in-polls-can-produce-misleading-results-especially-for-young-people-and-hispanic-adults/ https://www.pewresearch.org/short-reads/2024/03/05/online-op... They found that ~15% of US adults under 30 claim to have been trained to operate a nuclear submarine.
- felix089 8mo agoRE 1, they actually do have a pre-screening screening of the participants in general, you can check how they do it in detail: https://www.rapidata.ai/ https://www.rapidata.ai/
- tantalor 8mo agoAh, that's good to hear. I didn't see anything like that in the data dump so I assumed they don't do that. Glad to be corrected.
- mwigdahl 8mo agoLizardman's Constant is famously 4%. https://en.wikipedia.org/wiki/Slate_Star_Codex#Lizardman's_Constant https://en.wikipedia.org/wiki/Slate_Star_Codex#Lizardman's_C...