9 ms·
This paper evaluates performance compared to a 'human' which presumably is an average adult human without cognitive impairment. I had to dig in to the reference
by krona 3y ago
This paper evaluates performance compared to a 'human' which presumably is an average adult human without cognitive impairment. I had to dig in to the references:
In the first batch of participants collected via Amazon Mechanical Turk, each received 11 problems
(this batch also only had two “minimal Problems,” as opposed to three such problems for everyone
else). However, preliminary data examination showed that some participants did not fully follow
the study instructions and had to be excluded (see Section 5.2). In response, we made the screening
criteria more strict (requiring a Master Worker qualification, 99% of HITs approved with at least
2000 HIT history, as opposed to 95% approval requirement in the first batch). Participants in all
but the first batch were paid $10 upon completing the experiment. Participants in the first batch
were paid $5. In all batches, the median pay-per-hour exceeded the U.S. minimal wage.
(Arseny Moskvichev et al)
So in conclusion, this isn't a random sample of (adult) humans, and the paper doesn't give standard deviations.
It would've been more interesting of they had sampled an age range of humans which we would place GPT-4 on rather than just 'it's not as good' which is all this paper can say, really.
- cs702 3y agoAlso, it's possible there are LLMs pretending to be human beings on Mechanical Turk!
- jgalt212 3y agoWhat per cent of Mechanical Turk hits are being performed by an LLM these days?
- dmd 3y agoSeveral researchers where I work used to use mturk regularly and now have been forced to stop using it entirely because of this. They started getting "As a large language model" answers ... in even things where the only possible answer was 'true' or 'false' or '1-7'.
- ttul 3y ago“I hope this email finds you well…”
- EGreg 3y agoI love how your CompanyName does XYZ in field ABC. Buy our product. Let’s talk!
- kordlessagain 3y agoClick here for virtual coffee with me!
- jgalt212 3y agoAccording to Eric Schmidt, OpenAI is worth $90B. LLMs being unable to master mturk and $90B are incongruent. https://twitter.com/ericschmidt/status/1725625144519909648 https://twitter.com/ericschmidt/status/1725625144519909648 Who's right? Who's wrong?
- cratermoon 3y ago> According to Eric Schmidt, OpenAI is worth $90B. LLMs being unable to master mturk and $90B are incongruent. Top 5 reasons why OpenAI was probably never really worth $86 billion <https://garymarcus.substack.com/p/top-5-reasons-why-openai-was-probably https://garymarcus.substack.com/p/top-5-reasons-why-openai-w...> The company is overhyped and over-valued, and the capabilities of LLMs aren't All That and a Bag of Chips
- dmd 3y agoGood question. Let's go ask our expert, an internet commenter who calls himself "John Galt", to step in with an opinion based solely on market value!
- jgalt212 3y agotroll.
- svat 3y agoSee the "Artificial Artificial Artificial Intelligence" paper: https://arxiv.org/abs/2306.07899 https://arxiv.org/abs/2306.07899
- colincooke 3y agoMy wife studys people for living (experimental cognitive psychologist), the quality of MTurk is laughable, if that's our standard for higher level cognition then the bar is low. You'll see the most basic "attention check" questions ("answer option C if you read the question") be failed routinely, honestly at this point I think GPT4 would to a better job than most MTurkers at these tasks... She has found that prolific is substantially better (you have to pay more for it as well), however that may only be because it's a higher cost/newer platform.
- varelse 3y ago[dead]
- PaulHoule 3y agoMy take is the tasks on Turk are awful and will drive away anybody decent. I had a time when I was running enough HITs to get a customer rep and felt I was getting OK results. I wanted to get better at running HITs so I thought I would “go native” as a Turk and try to make $50 or so but I could not find tasks to do that were at all reasonable. Instead they’d want me to “OCR” a receipt that was crumpled up and torn and unreadable in spots and said they’d punish me for any mistakes.
- amelius 3y ago> honestly at this point I think GPT4 would to a better job than most MTurkers at these tasks... From the article: > Our experimental results support the conclusion that neither version of GPT-4 has developed robust abstraction abilities at humanlike levels. This makes the conclusion only worse for GPT-4 ...
- cosmojg 3y ago> In the first batch of participants collected via Amazon Mechanical Turk, each received 11 problems (this batch also only had two “minimal Problems,” as opposed to three such problems for everyone else). However, preliminary data examination showed that some participants did not fully follow the study instructions and had to be excluded (see Section 5.2). If they stuck to the average Mechanical Turk worker instead of filtering for "Master Workers," the parent's conclusions likely would've aligned with those of the study. Unfortunately, it seems the authors threw out the only data that didn't support their hypothesis as GPT-4 did, in fact, outperform the median Mechanical Turk worker, particularly in terms of instruction following.
- petermcneeley 3y agoThis critique in no way invalidates the conclusions of the paper.
- krona 3y agoAs I point out, the critique is simply that the paper isn't very good. In fact I'd go as far as to say its egregiously bad, even if the conclusion is 'correct' in as far as something was measured and they reported it.
- arp242 3y agoThe conclusion of "GPT-4 can't even compete against low-quality human effort like that found on MTurk" seems alright. The article would be better if it didn't claim to compare "humans", because it doesn't – or at least, not in the common sense definition of "comparing humans vs GPT-4". But to be honest I think that's a bit pedantic. It would be different if it had found that GPT-4 outperformed MTurk, but if GPT-4 can't compete against MTurk on these tasks, then I don't think it will fare better against you or me with our full attention.
- avs733 3y agoIf it’s making a claim about humans vs llms but shittified capitalism means that the humans are just using llms to make money churning on mechanical turn then you can’t make the underlying claim. That ignores random sample problems and the simplistic statistical analysis…so yeah it unfortunately does. It doesn’t prove they are wrong, but it’s not credible to say they are right
- a1j9o94 3y agoAnother thing I take issue with is this doesn't seem to be using known ways to improve performance of LLMs such as chain of thought and tree of thought prompting.
- arjvik 3y agoThere’s also the “take a deep breath” and other “unreal engine”-esque prompting tricks
- hackerlight 3y agoI also want to see GPT-3 vs GPT-4 comparison on these tasks.
- karmakaze 3y agoThis is interesting in a 'human interest news' report way but doesn't do anything to judge current systems any more the average people thinking older less capable chatbots were human.
- svnt 3y agoWhat is your concern exactly? This was a first-pass study in a field addressing some of the criticisms leveraged against an earlier study where the spatial reasoning problems were viewed to be too hard. They seemingly made the spatial reasoning questions as easy as they could. The qualifications they put on MTurk are pretty standard if you want humans who care about what they are doing responding to your study. It costs more to do this. It is a limitation of science that is both budgetary and procedural. By calling into question their results you seem to be suggesting that an average human would only 33% of the time be able to tell e.g. how many points are inside a box, or whether more points are inside or outside of a box. This is extremely basic spatial reasoning we are talking about. The problem they were addressing with the settings is just noise in the results by cheap bots and clicky humans trying to earn $0.50. It is endemic on MTurk.
- rileyphone 3y agoIf there are endemic issues using Mechanical Turk, maybe important studies shouldn’t use it, and those that do should be called into question.
- svnt 3y agoIf you have another way to provide actual humans with arbitrary representative demographics for answering ten minute surveys for a total cost of around $1 each the market would love to hear about it. Alternatively if you wanted to fund researchers and studies so they could afford to do these studies in a better way, I don’t think anyone would complain.
- kbenson 3y ago> By calling into question their results Did they? I took it more as a clarification of a point so people could more easily know what it meant, and wishing they did more so we could know more. Their last sentence in fact notes that what the paper says is correct and they just which we could make a more nuanced statement if it was done with more specific constraints.