3 ms·
Also "the per-annotation cost of ChatGPT is less than $0.003 -- about twenty times cheaper than MTurk." It's interesting that the best available MTurk Master c
by ftxbro 4y ago
Also "the per-annotation cost of ChatGPT is less than $0.003 -- about twenty times cheaper than MTurk."
It's interesting that the best available MTurk Master crowd-workers located in the US are paid about six cents per task.
- 8organicbits 4y agoGiven the low quality results of the humans, I suspect they were paying too low and the workers were giving low quality answers. I wonder if they should have paid more per task to get a better comparison. It doesn't feel surprising that GPT is cheaper and more accurate than a human who is randomly clicking buttons. Custom software can already do that.
- arbol 4y agoThe per annotation cost of simple image tasks in MTurk can be as low as $0.0003
- calvinmorrison 4y agoI guess my surprise is that the machine is only 20x cheaper than the cheapest human available.
- sebzim4500 4y agoGiven how badly the humans performed, I suspect a lot of them are either not bothering to read the prompt or using a worse bot than ChatGPT. Not surprsing they are so cheap. If you tried to pay people to do it properly then presumably it would cost much more.
- ummonk 4y agoIt’s worse because a US-based MTurker is far from the cheapest human available.
- taberiand 4y agoHey, with any luck, running one of these bots with the capability to replace me will cost just as much as hiring me. (Ha! As if. Costs are only going to come down)
- whitemary 4y agoAgainst the cost of living in the US, workers in the US are about as cheap as they come, second to maybe sweatshop workers in free trade zones or slaves in Dubai.
- exitb 4y agoChatGPT costs the same, regardless of the difficulty of the task, which is not the case for humans.
- andruby 4y agoWhen will we see a form of "arbitrage" where someone uses ChatGPT to do the work of an MTurk and pockets the difference? Will that lead to MTurk prices converging to ChatGPT prices?
- ftxbro 4y agoEven before LLMs, MTurk has been in a war with the botters, and my understanding is that the MTurkers have to periodically do captchas but many still use bots or various automated tools or utilities. The LLMs and especially the multimodal ones will only make the bots even better and break the captchas even harder. MTurk's business model specifically wants human workers categorized into fine grained categories so they will try to keep fighting the bots however they can.
- bogwog 4y agoCaptchas are futile. I remember many years ago when I was in highschool, I came across a shady job posting on an eBay listing. The work involved solving captchas. There was a web page they set up that just displayed one captcha after another, and it was your job to sit there solving them. I didn’t do it because I can’t imagine a worse hell than having to solve so many captchas, but it opened my eyes to how creative botters can be. I can imagine someone doing a similar setup for mturk, where all you need to do is remotely solve a captcha on your phone every few hours/minutes while ChatGPT does its thing. I also wouldn’t be surprised if that already exists.
- elif 4y agoIs there such a thing as a captcha in the age of GPT4? Maybe openAI put safety barriers on their model... But surely the botters have a model without that by now?
- arbol 4y agoIf you have to select the matching images from 9 images and you are feeding each of those images to GPT4 and asking "is this a <TARGET>?", I don't think you're going to get a response back to complete the captcha in time (e.g. <30s). This will change as the models advance.