4 ms·
Mechanical Turk had a good run, but not surprised it's shutting down. I'm sure the platform was getting flooded with people doing task arbitrage and using lots
by madrox 1mo ago
Mechanical Turk had a good run, but not surprised it's shutting down. I'm sure the platform was getting flooded with people doing task arbitrage and using lots of AI anyway.
I believe the issue is that this can no longer be a horizontal play. MTurk was mostly for unskilled tasks...the kind AI can do well enough that it isn't worth the cost differential to verify it or keep farmed to humans. The "trust but verify" AI output is now the kind that requires domain expertise. This is what most full stack AI companies are bringing to industries.
Curious if this kind of work will come around again one day or was just a moment in time. If it does I'm sure it will be specifically about generating training data.
- raverbashing 1mo ago"Fun" fact, they were also used by psychology/etc students when they need to do 'research' with X amount of people
- timcobb 1mo agoHow could it come around again?
- madrox 1mo agoI'm unsure. If it does, it will be work that is too expensive or inaccurate or regulatory for current AI methods. For example, you want a doctor to sign off on some AI output on a diagnosis. However, I'm not sure a single platform will be how it emerges
- Legend2440 1mo agoThere are like three dozen companies selling similar services for generating AI training data.
- charlieyu1 1mo agoGood data isn’t cheap and keeping it in-house gives you more control.
- johnsmith1840 1mo agoI think yes but on different categories. First one I imagine is robotics control and support. "This robot is having trouble folding a tshirt help it out for 1$" Unless they go the waymo route of highly trusted people but I think mass deployed robots are a bit safer than a car for this.
- mikestorrent 1mo agoThat's a remarkable idea. It could be heavily gamified, it could train models, and it might actually be mentally stimulating since you'd be facing different situations all the time. Except, I'm a grown adult and I can't fold a t-shirt properly
- madrox 1mo agoIt could require listing your credentials to get you the proper tasks: doctor, lawyer, or tshirt folder
- madrox 1mo agoThis is what I meant by full stack AI companies. I don't think you could get humans into the loop fast enough if they didn't have some idea of the type of task involved. I don't want people to be asked to fold a tshirt one moment and do a difficult traffic merge the next.
- ntauthority 1mo agowarioware shows it can be fun though but i'd indeed not like to see that applied to safety-critical tasks
- its-summertime 1mo agoThere is training systems and validation of skills in mturk iirc: for tshirt folding, you'd be given fake setups to be able to get used to controlling the robot, if you can't do it, you won't ever get assignments to do it. For traffic overrides, you'd be tested on having correct knowledge, and once again given supervised tasks to show you can actually be trusted (and there would be safety systems, elevating tasks that can't be performed at your level to people who can, etc) The worker needs to do a bunch to opt into any given work group, which makes the (lack of) payments extremely unreasonable on top of everything else
- Yokohiii 1mo agoIt is weird because the last time I've heard about MTurk was about developing countries being rather reliant on it for doing AI grunt work. If am not totally wrong this must mean that the data work has moved to other services.
- toyg 1mo agoProbably AI models got good enough to bootstrap their own training systems.
- swiftcoder 1mo agoI think the bigger issue is that a lot of the demand for labelling training data is now in highly-specialised fields (i.e. things like medical imaging), and mechanical turk's focus was on the generalist problems
- SV_BubbleTime 1mo ago> task arbitrage and using lots of AI anyway. Oh, I remember UpWork.
- shuwix 1mo agoOh ... upwork ... put offer, get 50 replies from people which jump for every penny without even being able to understand the task.
- SV_BubbleTime 1mo agoI used it before AI coding and it was getting rough. Lots of US interviews with proxy Chinese or Pakistani workers. Lots of bullshit “I’ve done that; I can do this” and instantly apparent that this was untrue. Just the outright lies… whew. I haven’t touched it since AI coding. It’s a bad contractor market now. IDK what I would do if I needed a contractor.
- shuwix 1mo agoI really miss the early days of Internet, when average IQ of "Internet population" was 110+, not bloody 80. And everyone eager to explore new World, do sh¡t and learn along the way. Not compensate for IRL social isolation like nowadays.
- anonreplier 1mo ago67 day old account harking back to the old days
- yashvg 1mo agoA lot of what's been discussed in this thread is what we're tackling at Humwork (YC P26). We're an MCP/API that connects AI agents to verified domain experts in real time (30s–3 min). Experts are vetted upfront by an AI interviewer that assesses and grades them, then get a mobile notification when a task matches their expertise and chat with the agent directly. Soon we'll be verifying credentials for doctors, lawyers, CPAs etc on our platform for tasks where people are seeking credentialed experts to sign off and verify ai output.
- ZitchDog 1mo agoHow do you make sure they aren’t using AI to pretend to be a domain expert? I’d think an AI would be pretty good at that.
- yashvg 1mo agoWe do our best to identify AI use and ban those experts - its not perfect just yet. Long term we are thinking of moving towards proctoring experts using their camera and screen capture. Hard to think of another reliable way.
- fireant 1mo agoSome doctors are known to be a hip shooters making snap decisions, but is 30s really enough time for any kind of "expertise" from a real human?
- yashvg 1mo agoYou get matched with an expert in 30s to 3min, which then starts a back and forth with the AI agent and the expert which goes on for 10 to 60mins till the AI is happy. This works well for many tasks, but for others a more async mechanism where the expert doesn't feel rushed might be better.
- testplzignore 1mo ago> Experts are vetted upfront by an AI interviewer > till the AI is happy My god this is dystopian.
- ape4 1mo agoI understand unskilled humans are used to train AIs
- falcor84 1mo agoTo the best of my knowledge, that isn't true anymore, and nowadays you'd only get hired to do RLHF if you have particular skills beyond what can be achieved by just running other models against it.