5 ms·

I know it's popular comparing coding agents to slot machines right now, but the comparison doesn't entirely hold for me. It's more like being hooked on a slot

by simonw 9mo ago

I know it's popular comparing coding agents to slot machines right now, but the comparison doesn't entirely hold for me. It's more like being hooked on a slot machine which pays out 95% of the time because you know how to trick it. (I saw "no actual evidence pointing to these improvements" with a footnote and didn't even need to click that footnote to know it was the METR thing. I wish AI holdouts would find a few more studies.) Steve Yegge of all people published something the other day that has similar conclusions to this piece - that the productivity boost for coding agents can lead to burnout, especially if companies use it to drive their employees to work in unsustainable ways: https://steve-yegge.medium.com/the-ai-vampire-eda6e4f07163 https://steve-yegge.medium.com/the-ai-vampire-eda6e4f07163

fdefitte 9mo ago
That 95% payout only works if you already know what good looks like. The sketchy part is when you can't tell the diff between correct and almost-correct. That's where stuff goes sideways.
mrbungie 9mo ago
If you are really good and fast validating/fixing code output or you are actually not validating it more than just making sure it runs (no judging), I can see it paying out 95% of the time. But for what I've seen both validating my and others coding agents outputs I'd estimate a much lower percentage (Data Engineering/Science work). And, oh boy, some colleages are hooked to generating no matter the quality. Workslop is a very real phenomenon.
- biophysboy 9mo ago
  This matches my experience using LLMs for science. Out of curiosity, I downloaded a randomized study and the CONSORT checklist, and asked Claude code to do a review using the checklist. I was really impressed with how it parsed the structured checklist. I was not at all impressed by how it digested the paper. Lots of disguised errors.
  - baq 9mo ago
    try codex 5.3. it's dry and very obviously AI; if you allow a bit of anthropomorphisation, it's kind of high-functioning autistic. it isn't an oracle, it'll still be wrong, but it's a powerful, completely different from claude tool.
    - biophysboy 9mo ago
      Does it get numbers right? One of the mistakes it made in reading the paper was swapping sets of numbers from the primary/secondary outcomes.
      - baq 9mo ago
        it does get screenshots right for me, but obviously I haven't tried on your specific paper. I can only recommend trying it out, it's also has a much more generous limits in the $20 tier than opus.
        biophysboy 9mo ago
        I see. To clarify, it parsed numbers in the pdf correct, but assigned them the wrong meaning. I was wondering if codex is better at interpreting non text data
    - enraged_camel 9mo ago
      Every time someone suggests Codex I give it a shot. And every time it disappoints. After I read your comment, I gave Codex 5.3 the task of setting up an E2E testing skeleton for one of my repos, using Playwright. It worked for probably 45 minutes and in the end failed miserably: out of the five smoke tests it created, only two of them passed. It gave up on the other three and said they will need “further investigation”. I then stashed all do that code and gave the exact same task to Opus 4.5 (not even 4.6), with the same prompt. After 15 mins it was done. Then I popped Codex’s code from the stash and asked Opus to look at it to see why the three m of the five tests Codex wrote didn’t pass. It looked at them and found four critical issues that Codex had missed. For example, it had failed to detect that my localhost uses https, so the the E2E suite’s API calls from the Vue app kept failing. Opus also found that the two passing tests were actually invalid: they checked for the existence of a div with #app and simply assumed it meant the Vue app booted successfully. This is probably the dozenth comparison I’ve done between Codex and Opus. I think there was only one scenario where Codex performed equally well. Opus is just a much better model in my experience.
Retr0id 9mo ago
It's 95% if you're using it for the stuff it's good at. People inevitably try to push it further than that (which is only natural!), and if you're operating at/beyond the capability frontier then the success rate eventually drops.
jrflowers 9mo ago
> It's more like being hooked on a slot machine which pays out 95% of the time because you know how to trick it. “It’s not like a slot machine, it’s like… a slot machine… that I feel good using” That aside if a slot machine is doing your job correctly 95% of the time it seems like either you aren’t noticing when it’s doing your job poorly or you’ve shifted the way that you work to only allow yourself to do work that the slot machine is good at.
Kiro 9mo ago
Just need to point out that the payout is often above 95% at online casinos. As long as it's below 100 the house still wins.
- mikkupikku 9mo ago
  He means a slot machine that pays you 95% of the time, not a slot machine that pays out 95% of what you put in. Claude Code wasting my time with nonsense output one in twenty times seems roughly correct. The rest of the time it's hitting jackpots.
saulpw 9mo ago
Yeah I'm finding that there's "clock time" (hours) and "calendar time" (days/weeks/months) and pushing people to work 'more' is based on the fallacy that our productivity is based on clock time (like it is in a factory pumping out widgets) rather than calendar time (like it is in art and other creative endeavors). I'm finding that even if the LLM can crank out my requested code in an hour, I'll still need a few days to process how it feels to use. The temptation is to pull the lever 10 times in a row because it was so easy, but now I'll need a few weeks to process the changes as a human. This is just for my own personal projects, and it makes sense that the business incentives would be even more intense. But you can't get around the fact that, no matter how brilliant your software or interface, customers are not going to start paying in a few hours.
- simonw 9mo ago
  > The temptation is to pull the lever 10 times in a row because it was so easy, but now I'll need a few weeks to process the changes as a human Yeah I really feel that! I recently learned the term "cognitive debt" for this from https://margaretstorey.com/blog/2026/02/09/cognitive-debt/ https://margaretstorey.com/blog/2026/02/09/cognitive-debt/ and I think it's a great way to capture this effect. I can churn out features faster, but that means I don't get time to fully absorb each feature and think through its consequences and relationships to other existing or future features.
fy20 9mo ago
> It's more like being hooked on a slot machine which pays out 95% of the time because you know how to trick it Right but the <100% chance is actually why slot machines are addictive. If it pays out continuously the behaviour does not persist as long. It's called the partial reinforcement extinction effect.
r00tanon 9mo ago
I was going to mention Yegge's recent blog posts mirroring this phenomena. There's also this article on hbr.org https://hbr.org/2026/02/ai-doesnt-reduce-work-it-intensifies-it?ab=HP-latest-text-5 https://hbr.org/2026/02/ai-doesnt-reduce-work-it-intensifies... This is a real thing, and it looks like classic addiction.
energy123 9mo ago
Being on a $200 plan is a weird motivator. Seeing the unused weekly limit for codex and the clock ticking down, and knowing I can spam GPT 5.2 Pro "for free" because I already paid for it.
zem 9mo ago
thanks, that steve yegge piece was a very good read.
globular-toast 9mo ago
> It's more like being hooked on a slot machine which pays out 95% of the time because you know how to trick it. I think you are mistaken on what the "payout" is. There's only one reason someone is working all hours and during a party and whatnot: it's to become rich and powerful. The payout is not "more code", it's a big house, fast cars, beautiful women etc. Nobody can trick it into paying out even 1% of the time, let alone 95%.

by simonw 9mo ago

fdefitte 9mo ago

That 95% payout only works if you already know what good looks like. The sketchy part is when you can't tell the diff between correct and almost-correct. That's where stuff goes sideways.

mrbungie 9mo ago

If you are really good and fast validating/fixing code output or you are actually not validating it more than just making sure it runs (no judging), I can see it paying out 95% of the time. But for what I've seen both validating my and others coding agents outputs I'd estimate a much lower percentage (Data Engineering/Science work). And, oh boy, some colleages are hooked to generating no matter the quality. Workslop is a very real phenomenon.

biophysboy 9mo ago
This matches my experience using LLMs for science. Out of curiosity, I downloaded a randomized study and the CONSORT checklist, and asked Claude code to do a review using the checklist. I was really impressed with how it parsed the structured checklist. I was not at all impressed by how it digested the paper. Lots of disguised errors.
- baq 9mo ago
  try codex 5.3. it's dry and very obviously AI; if you allow a bit of anthropomorphisation, it's kind of high-functioning autistic. it isn't an oracle, it'll still be wrong, but it's a powerful, completely different from claude tool.
  - biophysboy 9mo ago
    Does it get numbers right? One of the mistakes it made in reading the paper was swapping sets of numbers from the primary/secondary outcomes.
    - baq 9mo ago
      it does get screenshots right for me, but obviously I haven't tried on your specific paper. I can only recommend trying it out, it's also has a much more generous limits in the $20 tier than opus.
      - biophysboy 9mo ago
        I see. To clarify, it parsed numbers in the pdf correct, but assigned them the wrong meaning. I was wondering if codex is better at interpreting non text data
  - enraged_camel 9mo ago
    Every time someone suggests Codex I give it a shot. And every time it disappoints. After I read your comment, I gave Codex 5.3 the task of setting up an E2E testing skeleton for one of my repos, using Playwright. It worked for probably 45 minutes and in the end failed miserably: out of the five smoke tests it created, only two of them passed. It gave up on the other three and said they will need “further investigation”. I then stashed all do that code and gave the exact same task to Opus 4.5 (not even 4.6), with the same prompt. After 15 mins it was done. Then I popped Codex’s code from the stash and asked Opus to look at it to see why the three m of the five tests Codex wrote didn’t pass. It looked at them and found four critical issues that Codex had missed. For example, it had failed to detect that my localhost uses https, so the the E2E suite’s API calls from the Vue app kept failing. Opus also found that the two passing tests were actually invalid: they checked for the existence of a div with #app and simply assumed it meant the Vue app booted successfully. This is probably the dozenth comparison I’ve done between Codex and Opus. I think there was only one scenario where Codex performed equally well. Opus is just a much better model in my experience.

Retr0id 9mo ago

It's 95% if you're using it for the stuff it's good at. People inevitably try to push it further than that (which is only natural!), and if you're operating at/beyond the capability frontier then the success rate eventually drops.

jrflowers 9mo ago

> It's more like being hooked on a slot machine which pays out 95% of the time because you know how to trick it. “It’s not like a slot machine, it’s like… a slot machine… that I feel good using” That aside if a slot machine is doing your job correctly 95% of the time it seems like either you aren’t noticing when it’s doing your job poorly or you’ve shifted the way that you work to only allow yourself to do work that the slot machine is good at.

Kiro 9mo ago

Just need to point out that the payout is often above 95% at online casinos. As long as it's below 100 the house still wins.

mikkupikku 9mo ago
He means a slot machine that pays you 95% of the time, not a slot machine that pays out 95% of what you put in. Claude Code wasting my time with nonsense output one in twenty times seems roughly correct. The rest of the time it's hitting jackpots.

saulpw 9mo ago

Yeah I'm finding that there's "clock time" (hours) and "calendar time" (days/weeks/months) and pushing people to work 'more' is based on the fallacy that our productivity is based on clock time (like it is in a factory pumping out widgets) rather than calendar time (like it is in art and other creative endeavors). I'm finding that even if the LLM can crank out my requested code in an hour, I'll still need a few days to process how it feels to use. The temptation is to pull the lever 10 times in a row because it was so easy, but now I'll need a few weeks to process the changes as a human. This is just for my own personal projects, and it makes sense that the business incentives would be even more intense. But you can't get around the fact that, no matter how brilliant your software or interface, customers are not going to start paying in a few hours.

simonw 9mo ago
> The temptation is to pull the lever 10 times in a row because it was so easy, but now I'll need a few weeks to process the changes as a human Yeah I really feel that! I recently learned the term "cognitive debt" for this from https://margaretstorey.com/blog/2026/02/09/cognitive-debt/ https://margaretstorey.com/blog/2026/02/09/cognitive-debt/ and I think it's a great way to capture this effect. I can churn out features faster, but that means I don't get time to fully absorb each feature and think through its consequences and relationships to other existing or future features.

fy20 9mo ago

> It's more like being hooked on a slot machine which pays out 95% of the time because you know how to trick it Right but the <100% chance is actually why slot machines are addictive. If it pays out continuously the behaviour does not persist as long. It's called the partial reinforcement extinction effect.

r00tanon 9mo ago

I was going to mention Yegge's recent blog posts mirroring this phenomena. There's also this article on hbr.org https://hbr.org/2026/02/ai-doesnt-reduce-work-it-intensifies-it?ab=HP-latest-text-5 https://hbr.org/2026/02/ai-doesnt-reduce-work-it-intensifies... This is a real thing, and it looks like classic addiction.

energy123 9mo ago

Being on a $200 plan is a weird motivator. Seeing the unused weekly limit for codex and the clock ticking down, and knowing I can spam GPT 5.2 Pro "for free" because I already paid for it.

zem 9mo ago

thanks, that steve yegge piece was a very good read.

globular-toast 9mo ago

> It's more like being hooked on a slot machine which pays out 95% of the time because you know how to trick it. I think you are mistaken on what the "payout" is. There's only one reason someone is working all hours and during a party and whatnot: it's to become rich and powerful. The payout is not "more code", it's a big house, fast cars, beautiful women etc. Nobody can trick it into paying out even 1% of the time, let alone 95%.