12 ms·
Mechanical Turk workers are using AI to automate being human
- Lolaccount 3y agoSeeds of their own destruction? At what point do Amazon just switch off Mech Turk and leave it to AI? I guess it's best to make hay in the interim
- deleted 3y ago[deleted]
- jfengel 3y agoThey don't need to switch MTurk off, at least not for the clients. They could turn off the workers, and replace them with AI. I dunno who would continue to use it, rather than use the AI engine directly, but there could be a market for it. But they'd certainly need to make certain clients knew about it, since one major use of MTurk is to do surveys and studies whose whole point is to sample human beings.
- hyperhopper 3y ago> They could turn off the workers, and replace them with AI. That's exactly what they meant.
- mrguyorama 3y agoIt's probably more profitable for amazon to charge a premium for "human" work, so they will probably make using these kind of systems against the TOS while updating requirements and expectations to sort of require use of these systems to keep up.
- bentt 3y agoHonestly… I love it
- sillysaurusx 3y agoThis is in fact a precise definition of AGI, by the way. It’s the only true test. If your AI can make money via mturk, you’re on the right path. Superintelligence is when you can make all the money because all the problems are solvable. Right now only some are.
- sexy_seedbox 3y agoWhy is making money the end goal? When everyone, rich or poor, can use the same tools and make more or the less the same amount of money, then what's the point of a monetary system?
- mensetmanusman 3y agoWe will know AGI is starting to exist when driverless cars are everywhere.
- pixl97 3y agoI'd say that's mid stage AGI. Your latency requirements for driving require the hardware with you. If the early gen AGI is still power hungry then you'll see it plugged into a wall at first.
- Turskarama 3y agoA "general" intelligence is one that can do tasks it wasn't explicitly trained to do. The first self driving cars (that already exist) are powered by an AI that can only drive cars, so it's not general. Also if self driving cars are everywhere then we won't have AGI, because real AGI largely does away with the need for cars.
- bmmayer1 3y agoThis is going to cause problems as many Mechanical Turk tasks are being used to train AI models...
- 1323portloo 3y agoI think this is wonderful. It's a human-bootstrapped AI centipede with predictable, inevitable results.
- logicprog 3y agoTwenty fingered monster hands and nonsense garbled word salad here we come! The AI oroborus begins
- boesboes 3y agoAnd even fine-tuning using AI causes the models to quickly degrade I've seen here previously. Perhaps we are already at peak of these LLMs now and the next gen won't come because all training data will be poisened with AI generate crap. That would be kinda funny to me tbh
- sillysaurusx 3y agoThe invention of nukes caused a similar problem. Scientists had to harvest metal from pre-ww2 sunken battleships, since that’s the only metal not contaminated with some measurable amount of radiation. I forget the details, but it’s very likely pre-chatgpt training data will be prioritized, with exceptions for news and current events (which largely doesn’t matter whether it’s AI generated anyway).
- TZubiri 3y agoAlso happens with wikipedia. Later edits with web references are much more prone to citogenesis, while an early edit with a web reference is probably clear, even if the link is dead. Even printed references can be based on wikipedia now
- TrackerFF 3y agoAre the MT's paid by the hour, or by completed tasks? The latter would obviously incentivize people to use AI. But then again, so would the former, if it means that they can just let some script/program do the work, while they're going on with their day.
- icoder 3y agoI played with it a long time ago (doing actual work, mainly to understand the platform better, especially from a worker perspective) and back then it was all or almost all paid by task.
- wincy 3y agoIt’s paid by task. I used to make $7 an hour doing almost mindless tasks while I was watching TV. But if I went to the bathroom I’d lose out so my effective pay was lower, not to mention I’d get a 1099 on my $300 a year I made or whatever.
- usrbinbash 3y agoWell, there is a very simple solution to that problem: Pay people extra money for the privilege of having them fulfill the tasks the way you want them to.
- famouswaffles 3y agoOr just use the tools directly since they no longer underperform human actors. GPT-4 going toe to toe with the experts who set the benchmarks, way overperforming the crowdworkers https://www.artisana.ai/articles/gpt-4-outperforms-elite-crowdworkers-saving-researchers-usd500-000-and-20 https://www.artisana.ai/articles/gpt-4-outperforms-elite-cro... GPT-3.5 outperforming crowdworkers https://arxiv.org/abs/2303.15056 https://arxiv.org/abs/2303.15056 More money won't stop people from automating whatever they can. Especially when the automation provides such good results.
- hugh-avherald 3y agoHow did the researchers work out which of the human or GPT4 response was the correct one?
- Filligree 3y agoDon't think that'll work for training. Humans go insane if we lack sufficiently varied sensory input. Dreaming doesn't really help with that, and I recall a recent paper that showed something similar for AI.
- raverbashing 3y agoWhich would mean working in a controlled environment or installing invasive software in their computers (something that would block chatgpt for example)
- dgb23 3y agoAlternatively you can establish a trust relationship.
- usrbinbash 3y ago
- geraldwhen 3y agoThis is not novel. Mturk has always been automated since its very first release. I remember worker forums sharing scripts to query amazon for the expected results that would get you paid. Amazon put up a bunch of very lucrative test jobs, but you only got paid if your answer matched the majority. The majority were automating, so unless you used the sometimes incorrect answers, you got nothing.
- JustBreath 3y agoIt's surreal that even when it comes to AI we still fundamentally rely on underpaid labor.
- ironmagma 3y agoI used to think that. But when you think more about what value is, how it’s created and how we extract it to pay our insane salaries, it becomes less surprising and more sad.
- elvis10ten 3y agoCan you say more on this (value and how it’s extracted)?
- ironmagma 3y agoIf you are doing something that has inherent value — extracting gold from a mine, performing music, working a factory line — you are the top of funnel for value creation. Unfortunately for you, the only way for society to run is to have your would-be profits extracted to the maximum extent by other parties. Therefore the rich are the ones who actually don’t tend to produce as much and those who do are taken advantage of. The father away you are from value production, generally the better off you are.
- chii 3y ago> The father away you are from value production, generally the better off you are. you are only counting the human labour involved in the creation of value. what about the capital? Would you, as a hired gold miner, be able to accept zero pay for your work, until the gold is sold (then you got the full price of the sold gold)? What if you starved between the time you mined it, and the time it takes to sell? And how would you have gotten the equipment to do the mining in the first place, let alone the ownership of the mine.
- dkersten 3y agoSo the AI that asked MTurk to solve a captcha will now va e MTurk asking the AI to solve the captcha which will ask Mturk to…
- KingOfCoders 3y agoNice how they help Amazon automate them away. I believe Amazon is looking very closely on what is happening.
- deleted 3y ago[deleted]
- samstave 3y agoAs much as people want to ignore bezos and Amazon, they are far more evil thAn both fb and goog. Amzn is literally bad for everything on the planet aside from commerce - they are the seed of the Ferenghi
- KingOfCoders 3y agoNot sure if they are the seed of the Ferenghi, but as someone who sells self published books on Amazon, they are a very nasty company to deal with.
- Eumenes 3y agoThis just seems like one layer added on top of using captchas or human assisted reinforced learning
- jwie 3y ago“AI” eating itself is the best outcome for everyone. It’s a mass energy balance. The inputs equal the outputs; garbage in, garbage out.
- lacrimacida 3y agoIt’s not “AI”, it’s just LLMs. When/if AI becomes a reality there will be no need for mechanical turks
- kortex 3y agoIt is AI; artificial intelligence has been around since the 60s. It's not AGI, and no one (credible) is claiming LLMs are AGI.
- Aperocky 3y agoI like Apple in which they absolutely avoid potentially misleading terms and instead label all of this as Machine Learning (ML). Much more accurate imo. You'll never hear AI from Apple announcements, this contrast sharply with the few other large tech companies..
- ilyt 3y agoI mean yeah, while wasting a lot of energy and brainpower... Kinda like crypto but on smaller scale.
- m3kw9 3y agoWhy would you not root for AI? Its potential to improve humanity is huge even if there are potential bad sides. Some People bunching crypto with AI on the usefulness scale is quite ridiculous
- pdntspa 3y agoI will root for AI a lot more readily when we have a good alternative (UBI?) for people whos jobs are being displaced faster than they can retrain Y'all cheerleaders are going to create an underclass the likes of which you've never seen before. I hope they take up torches and pitchforks
- mensetmanusman 3y agoWith the potential end of Q and A on stackoverflow to add signal to AI training, and future websites being output of AIs themselves, we will probably reach a local maximum in usefulness. What we have now might be ~~ as good as it gets.
- nomilk 3y agoIt could be detected like so: 1. Give a sample of mechanical turk workers the same data to classify without telling them 2. Once submitted, look carefully at the ones they get wrong 3. If the turk worker gets similar ones wrong to other turk workers, they're probably not cheating 4. If the turk worker is frequently wrong on questions other turk workers all got right (e.g. if 99.9% of other workers got it right) then they've likely used AI to generate their answers and then fiddled around with a few afterwards to make it look human That presumes they're fiddling with a few responses for the human touch. If they're submitting raw AI output that should be easily discoverable by running similar models against the same data (like detection of chess cheating compares games against moves form several engines, not just one). I think that would work, unless everybody uses AI. If that latter were suspected, get a group to label data in person (where you can be 100% sure they can't cheat), then use those answers as the basis for comparison.
- jvm___ 3y agoWhat's the word for maps? City map makers didn't want people to copy their hard work, so they'd add a short street or some other feature that doesn't actually exist in reality.
- ceejayoz 3y agohttps://en.wikipedia.org/wiki/Trap_street https://en.wikipedia.org/wiki/Trap_street
- nomilk 3y ago> Mapmakers deliberately put slight mistakes in their maps so they can tell when someone copies them. If another map has the same mistake, that's very convincing evidence. http://www.paulgraham.com/say.html http://www.paulgraham.com/say.html
- deleted 3y ago[deleted]
- mahathu 3y ago
- htrp 3y agohttps://news.ycombinator.com/item?id=36432279 https://news.ycombinator.com/item?id=36432279 Comments from 2 weeks ago. This research study is using a very simple task, summarization, that is well solved by AI (to the point that everyone points this out as the first use case). Of course, the MTurkers are gonna paste it into ChatGPT. (Also n=44)
- andyjohnson0 3y ago"I speak of none but the computer that is to come after me,” intoned Deep Thought, his voice regaining its accustomed declamatory tones. “A computer whose merest operational parameters I am not worthy to calculate—and yet I will design it for you. A computer that can calculate the Question to the Ultimate Answer, a computer of such infinite and subtle complexity that organic life itself shall form part of its operational matrix. And you yourselves shall take on new forms and go down into the computer to navigate its ten-million-year program. -- h2g2
- mrweasel 3y agoA great opportunity is opening up in the "Do things that don't scale" department: Verifiable human work and content.
- gumballindie 3y agoHere is a radical idea. Inspired by south park’s human centipede: AI customers. It works by automating production AND the people buying it. These customers can pay using a crypto currency made just for that. To see the vision behind this all one needs is a little bit of microdosing and a reality distortion field.
- gregw134 3y agoSet up a whole virtual economy, with AI agents paying rent, having jobs, inventing goods to sell and deciding whether to buy. Fast forward 100 years and see what happens.
- gumballindie 3y agoGuaranteed some of them will start grifting, taking bribes, dodging taxes and selling snake oil (perhaps even a little ai of their own making), while others will protest endlessly to deaf ears, struggle paying rent or starting families and living a life of misery, and those in high paying jobs, while still worked to death and lacking freedom, will turn a blind eye to everyone's suffering. Free pizza and bean bags.
- flaminHotSpeedo 3y agoThis study is useless, the experiment design is so fundamentally flawed that it provides virtually no real information. As others have pointed out, mturk always had workers using automation, and the researchers set up a task that is an obvious application for LLM's, who's output makes it difficult to different humans generated output, and isn't even a good use case for mturk (since the output is free form text that would have to be validated by humans rather than an automated quorum check). This is simply an insult to science
- gumby 3y agoThis reminds me of the problem Google Translate encountered: after starting eith. manually curated set of texts (e.g. UN and EU documents), it later read translated pages on the web...until people started automatically translating their sites with google translate leading to a bad feedback loop.
- iancmceachern 3y agoIt's funny because I've also seen articles pointing out that some AI companies are using mechanical turks to simulate AI. Turtles all the way down.
- 13years 3y agoIf you can’t be 100% sure something was done by a human, you’re probably better off assuming it wasn’t. That’s a depressing principle to have to adhere to, but here we are. It seems we will be soon drowning in an endless sea of autogenerated noise. For all the talk of X-risk, this is the imminent tangible risk we can perceive and reason about. If we can't find solutions to the problems that primitive AI will create, then solving the harder problems of AGI/ASI might become moot as we self destruct before we even get that far. FYI, my summary of many other potential road blocks and stumbles we are likely moving towards in the near term. https://www.mindprison.cc/p/ai-and-the-end-to-all-things https://www.mindprison.cc/p/ai-and-the-end-to-all-things