12 ms·

The last six months in LLMs in five minutes

804 points by yakkomajuri 5mo ago

hmaddipatla 5mo ago
[dead]
iekekke 5mo ago
It’s good to see dates being hard coded re. Improvements in the models that should deliver material gains. As time progresses one now has a yard stick to measure against progress. No more excuses - show me the money baby.
bb88 5mo ago
I met Simon for the first time this year at pycon. Wow, what a great guy.
zarzavat 5mo ago
Somewhere right now some human artist is being tasked with drawing illustrations of pelicans riding bicycles to be used as training data at a big AI lab.
- minimaxir 5mo ago
  Every modern image-generation model can generate a pelican on a bicycle trivially. The point of the test is to generate SVG text that represents an image, which is more complicated. Yes, there are ways to convert raster images to SVG for use in training data but it's not a good use of anyone's time.
  - jofzar 5mo ago
    I wouldn't wish creating a svg pelican on a bicycle on my worst enemy
  - Mashimo 5mo ago
    > Every modern image-generation model can generate a pelican on a bicycle trivially. Mistral seems to be the exception. Their new model from a few weeks ago is worse then selfhosted gemma.
  - Antibabelic 5mo ago
    I don't understand this response. Human artists can and do make SVGs.
    - captn3m0 5mo ago
      They typically use a visual editor like Inkscape with visual feedback. Nobody is hand-coding a complex SVG.
      - Antibabelic 5mo ago
        The end result is the same: an SVG file. Definitely doesn't matter to an LLM what produced it.
- energy123 5mo ago
  The quality of the Gemini pelican was such a step change in one iteration, while the other benchmarks remained quite flat, that I think you are right. Although whether they targeted Pelicans in particular or just svg, I can't say.
  - deleted 5mo ago
    [deleted]
shepherdjerred 5mo ago
> and there’s zero chance any AI lab would train a model for such a ridiculous task. I'm not sure that's true anymore considering how popular Simon's blog is
- nickvec 5mo ago
  Simon mentions further along in his article that given Jeff Dean’s post referencing the pelican-riding-a-bike task (and how good current models are at doing it), that it’s no longer a great benchmark to use. Enter the opossum riding an e-scooter!
  - aaronbrethorst 5mo ago
    Banana man on the Segway
- _puk 5mo ago
  > So maybe the AI labs have been paying attention after all! > I think this mainly demonstrates that the pelican on the bicycle has firmly exceeded its limits as a useful benchmark. As acknowledged in the article.
  - kzrdude 5mo ago
    Gemini 3.1 basically takes it home on that benchmark, anyway, it's done.
    - sunaookami 5mo ago
      Gemini is heavily benchmaxxed and sucks in agentic coding so no surprise.
- simonw 5mo ago
  That bit probably works better in the talk, it was a setup for a joke later on.
- muzani 5mo ago
throwaway2027 5mo ago
December 2025 was the breakthrough for me. January Claude was euphoric, ChatGPT was up there. February Gemini cooked for a second there. March amazing. April the big bad nerf. May GPT 5.5 is just pure bliss altough 2x limits temporarily, not sure about Claude it's sort of okay still not as good as it felt before, slowly increasing limits with more compute and rebuilding good will.
- dmpk2k 5mo ago
  I find your emotional language truly quite fascinating. I've heard people talk like that about drugs.
  - wilg 5mo ago
    Similarly, I've heard people talk like that about things that are not drugs.
    - sph 5mo ago
      You can get a dopamine rush from anything, from drugs to using LLMs.
  - OvervCW 5mo ago
    I actually thought it was a joke comment, but I'm worried now that it's not the case.
- _puk 5mo ago
  I think Opus 4.6 at its peak was the "how can anyone not get that this is good" for me. Then the nerf, and the massive uplift in tokens for 4.7, a model which I find lazy and prone to hallucinate. It's probably time to try GPT5.5. Like many I'm pretty heavily invested in the anthropic ecosystem at this point, which I suppose gives another strong reason to make the switch.
  - Arn_Thor 5mo ago
    I was a dedicated Claude user but in March/April I started using GPT5.5 on a new project that Claude had tried and failed to execute successfully. GPT knocked it out of the park, and was able to do it within my subscription allocation of tokens. I'd recommend giving it a go at least. Something like OpenClaude can let you use the Claude tools you're used to
aizk 5mo ago
I'm so glad Simon is documenting this. The field is evolving so fast, so rapidly, so hungry for data and money, that few are willing to zoom out and document everything big picture so we can see the changes over time. I mean do you guys remember "Do anything now"? Just a distant memory, a funny party trick.
Insanity 5mo ago
I wonder how much the 'inflection point' is a thing vs marketing. I'm sure the models got somewhat better, but even now when I'm trying to 'vibe code' a game with the latest models (combination of Codex w/ gpt5.5 and gpt5.3-codex), they really do struggle. They definitely get something barebones up and running, but it's far from a fully fledged application.
- righthand 5mo ago
  I mean this blog post and many from this author are pure evangelism and marketing. Can you find anything critical or any dissent from this author about LLMs?
- adgjlsfhk1 5mo ago
  It's very real. Just in the past 2 months or so IMO there's been a pretty big improvement in claude for local dev (although I think a lot of that is less model strength and more harness capability). 1m context is a huge difference (~30 min vs 2.5hr between compact significantly increases the scope of what I get the AI to do before it goes stupid). The other biggest difference I've noticed is a better balance of actually doing the work vs pushing back on bad ideas. I want the AI to tell me if it thinks the thing I am telling it is wrong or a bad idea, but if I confirm, I want it to do that anyway. A couple months ago, the claude was a lot more likely to either say "This is too much work I'm not going to do all of it", tell me the idea was genius (and then pretend to do it) or something equally useless.
  - DeathArrow 5mo ago
    >1m context is a huge difference (~30 min vs 2.5hr between compact significantly increases the scope of what I get the AI to do before it goes stupid) I think the smart zone stays within the first 100k tokens, no mater if the context window is 240k or 1 million. I divide the work to fit within that 100k and use subagent for the tasks.
    - danielbln 5mo ago
      In my experience it's more like 400-500k tokens.
rTX5CMRXIfFG 5mo ago
Am I crazy, or are these differences between the best models so marginal that you’d get roughly the same performance if you use the same high-quality harness (ie preloaded instructions from md files, including custom skills)?
- Sparkyte 5mo ago
  No you're not wrong. Many people will see what you see. Enthusiasts will see it as monumental squeezing out that last drop of performance. In my opinion I think it is okay for enthusiasts to feel that way. I'm just satisfied with getting a tool as an aid. Personal opinion we need to focus more on efficiency instead of how large or complex a model can get as that model creeps into more resource requirements. If the goal is to cost a billion dollars to operate than we've really lost the idea of what models are supposed to be achieving.
- minimaxir 5mo ago
  To an extent. I've had GPT 5.5 solve problems that Opus 4.7 struggled with, using an identical AGENTS.md/CLAUDE.md and no skills.
- bluegatty 5mo ago
  You will immediately notice the difference if you use it at the threshold. It's like most people just watching a 'starting nba player' (not superstar, but just starting player) vs one that sits on the bench. If you were to just watching them play, work out, shoot - you'd never notice the difference. Put them head to head and it's 98-54 and you start to see the patterns. It's pretty interesting actually, someone tell me what the 'science' for this is, I'm sure there is some kind of information theory at work here. Software has innumerable kinds of problems at varying level of complexity and so it provides the perfect testbed for seeing how far models can go in practice. Should add: you're very right to hint that harness, tooling, and models tuned o both the harness and he kinds of things people do on the harness, as well as some other things do make enormous difference. Bu and large, SOTA Codex/Claude Code are substantially better - at least for now. That may change.
  - 5mo ago
bluegatty 5mo ago
'Producing Images' or even 'Some Code that is Valid and Compiles' is in some ways one of the most misleading ways we assess quality of the AI. It is getting very good at producing code that compiles - at the algorithmic level. This is definitely noteworthy - and the AI is crossing a critical 'productivity threshold'. But 'Drawing of a Proper Duck' is almost arbitrary because it may have nothing to do with the 'Specific Duck You Wanted'. Everyone has tried to get AI to 'Draw The Thing They Want' and you notice immediately how it's almost impossible to 'adjust the image' along the vector you want - because ... and this is key: -> the AI doesn't really understand what a Duck is, it's components, or fully how it made the duck <- It just knows how to 'incant' the duck. This becomes very clear when you try to get the AI to write proper documentation - it fails so miserably, even with direct guidance. This is really strong evidence of how poorly the AI is generalizing, and that it is not 'understanding' rather it's 'synthesizing' from patterns. We already kind of knew that - but we have not yet built an intuition for that until now. Only now can we see 'how amazing the pattern synthesis' is - it's almost magic, and yet how it falls off a cliff otherwise This has deep implications for the 'road ahead' and the kinds of things we're going to be able to do with AI. In short: the AI is 'Wizard Level Code Helper, Researcher, and Worker' - but it very clearly lacks capabilities even one level of abstraction above the code itself. LLMs were first trained by 'text' and now ... they are 'trained by our compilers'. Basically g++, javac, tsc are the 'Verifiable Human Rewards' in the post-training and reinforcement learning - and the AI is getting extremely good at producing 'code that compiles', but that's definitely an indirection from 'code that does what we want'. It's astonishing that it took us all this time to internalize and start to discover what I think will be in hindsight a very obvious 'threshold' of it's capabilities. We are constantly 'amazed' at the work that it can do, and therefore over-project it's capabilities. I have no doubt that even with these limitations - the AI will unlock a lot more as it gets better - and - that it will 'creep up' the layers of abstraction of it's understanding. But I strongly believe that the AI is going to get much 'wider' (pattern matching dominance) before it gets 'higher' (intrinsic understanding) - and - that this may be a fundamental limitation. This may be 'the Le Cunn' insight - when he talks about the limitations of LLMs in detail - I believe this is that insight writ large. Even the term AI - or certainly 'AGI' may be a misleading metaphor - were we to have always called it 'Stochastic Algorithms' or something along those lines, it's possible that our intuition would be framed a bit better. The most interesting thing is how it is definitely amazing, world changing, novel and powerful and some ways - and obviously useless in others at the same time. That's the 'threshold' we need to better understand.
DeathArrow 5mo ago
Apart from GLM 5.1 and Qwen 3.6, there are other Chinese models that are noteworthy: Kimi K2.6, Xiaomi MiMo V2.5 Pro, Deepseek v4 and MiniMax M2.7.
- simonw 5mo ago
  100% true - I only had five minutes so I had to edit it down to just a couple, but all of those models are excellent and keep leap-frogging each other.
  - rahimnathwani 5mo ago
    Looking forward to next time, hoping you mention speculative decoding and MTP :) It would support your point about the performance of 20GB local models.
ex-aws-dude 5mo ago
Is the RLVR the key breakthrough for the uplift or is there more to it? Does that suggest the uplift was only for things that are easily verifiable like code?
- 4b11b4 5mo ago
  RL we're gonna find out will get abandoned cuz we don't even know what is getting "aligned", just my naive gut feeling don't take it seriously
- rdedev 5mo ago
  I would say that most improvements are in easily verifiable things like code or math. Atleast that's where all the amazing results seem to be coming from. Other domains I am not sure but I've heard from people like Cal Newport that the rate of increase outside of code and math are not as equally impressive
- vanuatu 5mo ago
  Yes, with good RLVR at scale you can greatly improve performance especially on benchmarks The hope was that good RLVR on relatively contrived datasets (like benchmarks) would be generalized to good software taste, which has somewhat succeeded but also the models fail in horrible ways still And the hope beyond that is that good skills in fundamental problem solving tasks (coding, math) would generalize to tasks beyond math and code, which did happen but less so
tayo42 5mo ago
The claw thing really came and went fast lol
- deleted 5mo ago
  [deleted]
- yieldcrv 5mo ago
  I just started a new job and the person I report to was just excited to tell me about it, here in Mid May "and then you have to get a mac mini, and then, and then" smile and nod, it pays weekly
- viking123 5mo ago
  I mean yeah? It was marketing campaign to boost the model providers and give Steinberger a cozy job at OpenAI. Hook, line and sinker. Wake me up when we have an agent with constant learning and changing weights that I can have personally, not some LLM that can always fall prone to jailbreak and context injection attacks. You think most of this stuff here is organic? Oh boy..
DeathArrow 5mo ago
I think that there's a lot to be improved in harnesses and the way the models are interacting with harnesses. For example, the harness should be able to steer the model when thinking.
tptacek 5mo ago
If you're a vulnerability researcher or a security person generally, there's a big inflection point from Spring of this year.
- thierrydamiba 5mo ago
  Can you be more specific?
  - simonw 5mo ago
    The Claude Mythos / Project Glasswing thing is real: https://www.anthropic.com/glasswing https://www.anthropic.com/glasswing I met a few people at PyCon this week who have been part of Glasswing (they're just starting to be allowed to talk about it) and it really does drive down the cost of finding vulnerabilities. I've been collecting notes on that here: https://simonwillison.net/tags/ai-security-research/ https://simonwillison.net/tags/ai-security-research/
    - krzyk 5mo ago
      People in my company sounded underwhelmed by it. It usually was founding issues by not understanding deployment (or not being fed that info).
      - halflife 5mo ago
        A friend of mine had hands on experience, it’s not the intelligence of it, it’s the speed. You used to have a couple of days to close a breach, now it 2 hours.
      - vessenes 5mo ago
        You might point them at the cloudflare blog about deploying mythos - I found it interesting. Upshot — as your folks discovered, deployment, harness, and utilization method matters for mythos and is a bit different than how you’d deploy a coding agent for writing code, but if you do that, you get something capable of end to end chaining and reasoning about a much broader class of vulnerabilities. No personal experience with it. But the security team writeups I’ve read are significantly more positive about it than you describe, so it might be worth a second look.
grey-area 5mo ago
Haven’t noticed much significant progress in LLMs myself in 6 months (significant as in new or vastly improved capabilities or understanding, not new releases, there are plenty of those). I feel like if anything people started to realise the significant limitations of LLMs when you try to use them as ‘agents’ which was the big direction LLM companies tried to push recently. Best use of LLMs so far IMO is finding vulnerabilities (with human help) and pattern matching in other domains. For generating code and prose they are still mediocre and somewhat unreliable and for use as personal assistant agents I wouldn’t trust them. So what’s happening with openclaw, the biggest experiment in agentic, vibe coded by the agents themselves? The thing that was so hot a few months ago. https://github.com/openclaw/openclaw/pulse?period=daily https://github.com/openclaw/openclaw/pulse?period=daily 279 commits to main from 77 authors in the last 24 hours. Why is there so much churn and how could you trust it with your data? This is changes in ONE day! If these are useful changes, surely it’d be superhuman by now given months of this pace. What are people using this for?
- deleted 5mo ago
  [deleted]
vishal_new 5mo ago
what are your thoughts on Software engineer replacement. My team has already seen big reductions. Q/A team is gone. Software Engineer reduced by a third. Scared for the future
- ShinyLeftPad 5mo ago
  If you're famous, you'll be fine. If you're in retiring age, you don't care. Otherwise, good luck! We put ourselves on the street by not protesting what is happening.
  - munksbeer 5mo ago
    I think the general population earning median wages will have very little sympathy for first world software engineers earning vast amounts of money. What are you going to tell them? Suddenly you're earning what they're earning for sitting at a desk every day?
    - ShinyLeftPad 5mo ago
      General population, you mean non SWEs? Because there are many SWEs around the world who earn median wage and who stand to lose it all as the avalanche of firings is ramping up. Non SWEs (salespeople, clerks, secretaries, assistants, taxi drivers, writers, 3D modelers, artists, designers) are of course going the same way. Unless they are protected (unionized or such), why would they have sympathy for SWEs? People of our ilk are the ones causing this (to them and to ourselves). What I will tell them is to not repeat our mistake, organize and protest.
- simonw 5mo ago
  Ditching the QA team when the single highest challenge is verifying that vibe-coded systems do what they're meant to is extraordinarily short-sighted. Personally, the more time I spend working with coding agents the least worried I am for my career. Getting the best results out of them is really hard. They amplify existing skills and experience, so the more experience you have the better.
dnnddidiej 5mo ago
Also LinkedIn wars of people trying to claim throne as most AI-pilled, throwing down strawmen stories of luddites yelling at data centres who'll lose their job to a single person doing 100x work.
LZ_Khan 5mo ago
I'm curious how the 6 months have looked from a non-programmer's perspective. What kind of co-working tools and similar optimizations have people from other fields experienced?
- deleted 5mo ago
  [deleted]
- conception 5mo ago
  Claude in Office was a tipping point for nontechnical folks around me. Everyone’s slides decks are immaculate now. Finance isn’t needing nearly as much BI help. It’s pretty impressive.
  - RobinL 5mo ago
    Interesting. I don't have to use PowerPoint much, but I hate it when I do. I don't want the llm to write the words but I do want it to make things look nice. So does this work well now?
    - angled 5mo ago
      My pipeline for this is vscode + prompts + markdown templates + GitHub copilot -> markdown docs -> pandoc to produce.docx -> copilot in word for “nice” formatting -> copilot in ppt for nice decks. LLMs all the way down. I find it’s easier to version control and diff the .md artefacts, those remain my authoritative source.
      - asdff 5mo ago
        Wow. Seems like a headache compared to how I make slides the old fashioned way: copy and paste my figures into blank powerpoint.
      - vessenes 5mo ago
        I was doing something like this, and then realized at least with claude that it’s so much better at HTML that it’s better to get an HTML-first deck together, which could then be turned into a PPT template and/or PDF directly, depending on needs. It saved me a fair number of design-tweak steps in the md -> pandoc part of the workflow. Realistically, hand editing claude’s HTML is also easy in most cases, so I didn’t feel like I lost much (for the generative cases). Similarly if it’s mostly what I’ve written directly that’s the source it’ll be in markdown, and I’ve found it’s a faster path to have md -> (LLM-translated HTML deck) -> pdf.
pr337h4m 5mo ago
Something that’s largely been ignored: DeepSeek has made context caching virtually free with V4-Flash.
- hootz 5mo ago
  I swear to god that DeepSeek V4-Flash is the most useful model available right now. It's SO FAST and is good enough for so many tasks that I run it most of the time for almost everything. Even when it messes up, it's so cheap to iterate that I can fix most problems without changing the model to a more "capable" one.
gib444 5mo ago
Starting from zero today, how would someone quickly get upto speed with the latest and greatest AI tooling on an extremely limited budget? Is the only choice to pay for the "max" plans? Or just read so much about it that you bs your way through an interview and then use the company's resources? Simon, I'm curious too how much you invest each month researching all the latest and great AI tech?
- RobinL 5mo ago
  $20 chatgpt pro plan gives pretty generous usage both of codex, general chat
  - gib444 5mo ago
    Ah I'd read so much about the downgrading of that plan I didn't think that was still true?
    - azuanrb 5mo ago
      It depends on what you’re comparing it against. For $20, OpenAI is still probably the best value for SOTA models. In terms of limits, you can use GPT-5.4 instead of 5.5. The intelligence feels similar, but it’s cheaper. You can also experiment with other harnesses like pi. It’s lightweight but capable enough, and its token usage is definitely much more efficient.
- x86cherry 5mo ago
  Opencode has free access to Qwen 3.6 and Deepseek v4 Flash right now. They're on par with Claude and Codex imo - when you still design architecture and know what the output should be. Claude and GPT 5.5 need less guidance with vibe coding, but we're not yet at a point where that's sustainable anyway even with those models.
  - gib444 5mo ago
    Thanks, playing with Opencode now. It just wrote a half-decent Android small app. Pretty good so far!
bunzee 5mo ago
Spot on. Building our tool, we found AI is magic at scraping competitor data, but terrible at market validation. The 'why' is strictly human.
- 2001zhaozhao 5mo ago
  So, the best way to use LLMs is to wait for your competitors to do market validation and then scrape their data. Hmmm......
  - tardedmeme 5mo ago
    It's always been much easier to copy an existing product than to make a new one nobody's thought of before.
- kkarpkkarp 5mo ago
  sorry but how this comment refers to the commented post?
jrowen 5mo ago
There's something fitting about the mystical nature of LLMs and scrolling through a bunch of goofy pelicans on bicycles representing report cards for the bleeding edge of technology. How are these even graded? Qwen3.6-35B-A3B gets high marks for a pelican with a gaping hole in its bill? edit: Just noticed its feet are disconnected from its legs as well (but right on the pedals!). Pardon my French but that's Chinese af.
LarsDu88 5mo ago
My goal post for "AI will definitely replace most SWEs" was to reproduce a particular 90s programming game one shot and then add multiplayer support with minimal prompting. Opus 4.5 hit that point in November.
- grey-area 5mo ago
  I tried this a while ago, haven’t tried again recently. The models were producing code that was clearly lifted from stuff in their training data, and what I ended up with was a fairly decent game in html and js after a bit of tidy up, though it felt like several code paradigms smooshed together rather than a coherent whole, but it mostly worked. Not something I’d want to maintain but it was impressive at the time. They were able to one-shot famous games (like asteroid or pong), I suspect because they had been trained on multiple versions of that game. So like producing Harry Potter, with the right prompt it was able to produce a license stripped version of code it had seen. I tried another arcade game like frogger and it failed really badly and took a lot longer, never got it working. The whole exercise left me feeling they have a long way to go, I don’t see how anyone could think they would replace SWE unless they didn’t look at the code produced, even now.
  - vessenes 5mo ago
    Out of curiosity - what harness did you use, and what model? And how are you prompting? In my mind prompting like: “You’re going to make frogger in javascript. I want a complete clone of functionality for level 1, with amazing 80s era pixel art sprites. I’m super lazy, so you’re going to have to test everything, right from the start. Pick a test harness, write the tests, including tests for having amazing graphics, gameplay, input, UI, sounds, etc, and write a full workplan, then work through that workplan, in parallel where you can. The workplan should emphasize getting a stripped down version up immediately and have workstreams for all the major requirements after that. Add a final test that assesses how fun the game is by reviewing a real video of a test run. Loop on that final test until you can’t improve things any more.” Should produce something playable with no further input. As you say, I’m not sure it would produce a codebase we’d want to look at or work on. But, I’d be surprised if this weren’t successful.
raghavchamadiya 5mo ago
[flagged]
pineapple_opus 5mo ago
All I see is mention of how various models generate image of "pelican riding bicycle(s)"
- emil-lp 5mo ago
  Yes, the "pelican riding a bicycle" is the ultimate test of not understanding how LLMs work. Well, a combination of that and believing that replication of test data is a good measure of progress.
  - vessenes 5mo ago
    Spicy — why does it show ultimate non-understanding?
    - JohnKemeny 5mo ago
      because success comes from reproducing a memorized pattern rather than transferable reasoning? At the same time failure proves little because most humans also could not manually create a correct SVG of a pelican riding a bicycle. What is it exactly that such a test is testing? In which situation would you measure the "competence" of a human being by asking them to write an SVG of a pelican riding a bicycle?
      - okamiueru 5mo ago
        > most humans also could not manually create a correct SVG of a pelican riding a bicycle. Most humans absolutely can write this with a suitable vector graphics tool such as inkscape or illustrator. Surely, you're not suggesting that a fair comparison would be using a text editor? If so, would you suggest an equivalent raster based task would only be fair, if the human would manually assigning RGB values to each pixel?
- ClikeX 5mo ago
  We all know the true test of AI is Will Smith eating spaghetti.
- 5mo ago
nothinkjustai 5mo ago
[flagged]
- rhubarbtree 5mo ago
  Certainly a massive AI booster. What Are the conflicts of interest?
nothinkjustai 5mo ago
[flagged]
wewewedxfgdf 5mo ago
Does this guy have a "publish to front page of HN" button on his blog editor?
- troupo 5mo ago
  He has the most measured (and often quite detailed) posts on LLM and LLM progress, and is the opposite of hype.
- nickvec 5mo ago
  He’s pretty well known in the HN community. https://en.wikipedia.org/wiki/Simon_Willison https://en.wikipedia.org/wiki/Simon_Willison
  - koolala 5mo ago
    thats a cool wiki picture
- koolala 5mo ago
  its better than ex-google CEO spam i see astroturfed everywhere else
- specproc 5mo ago
  He's one of the main developers behind Django.
  - dcminter 5mo ago
    Years ago I used to read his blog on Django and found it quite interesting despite being neither a Django nor even a python user - this must have been at least 10 years ago and perhaps more. When he resurfaced in my feeds as an AI commentator it took me quite a long while to join the dots that he was the same person!
- schnitzelstoat 5mo ago
  I liked the article, so if he has such a button I hope he keeps clicking it.
nothinkjustai 5mo ago
[flagged]
- tomhow 5mo ago
  We've banned this account. We detached this comment from https://news.ycombinator.com/item?id=48189072 https://news.ycombinator.com/item?id=48189072 and marked it off topic.
nothinkjustai 5mo ago
[dead]
ramon156 5mo ago
> Google released the Gemma 4 series of models, which are the most capable open weight models I’ve seen from a US company. Implying another country has a better model? I'm being pokey here because I'm very curious! I know Gemma is efficient, but I also remember Qwen and Kiwi being referred to as optimized. The difference being that Gemma is using less tokens, but maybe Qwen/Kiwi's quality is higher? I dont know.
- pferdone 5mo ago
  The consensus right now is that Qwen3.6 in its 27B and 35B-A3B versions is better for coding whereas Gemma4 is stronger when it comes to OCR, audio transcription and the likes. Margins are slim though and the harness at these model sizes is the most important factor.
- 0xCMP 5mo ago
  In my experience the qwen models are best locally, but gemma ones have always been good. gemma4 is a notable improvement.
jimbobthemighty 5mo ago
I asked Gemini for a video of 'pelican riding a unicycle in hyde park' - I was blown away by the output: https://gemini.google.com/share/55e250c99693 https://gemini.google.com/share/55e250c99693
- grey-area 5mo ago
  That’s really impressive, and slightly worrying for creatives involved in film, animation or modelling.
  - notachatbot123 5mo ago
    Even more worrying are the implications for fakenews, propaganda, fraud, deception and mental health.
    - sevenzero 5mo ago
      This is really my biggest worry when it gets to consumer AI. People already have a hard time informing themselves properly. Now we have technology that just boosts the already existing confirmation bias people have. It's sickening.
    - dzhiurgis 5mo ago
      Maybe short term yes. But longer term people will finally put their guard up against deception that’s been around for decades.
      - layer8 5mo ago
        People will still believe what they want to believe.
      - tonyedgecombe 5mo ago
        If they haven’t in the past I don’t see why they would now.
        5mo ago
bradley13 5mo ago
As someone who uses AI daily (not in agent mode, just user-interactive), I have definitely noticed major quality improvements over the past few months. And that's surprising, because when you use something daily, you tend to overlook the big jumps. I haven't looked into any sort of "agent" mode, just because I don't yet quite trust the AI not to do something dumb. Also, I don't use M365, where Copilot is integrated, so I suppose I would have to set it up myself.
hollowturtle 5mo ago
> The coding agents got really good It's since november 2025, the so called "inflection point", that I'm still wondering for who coding agents become "really good". All I observe they got better at tool call and answering questions about big codebases, especially if the question has a vague pattern to search, and they're superuseful for that! For generating production code even with a lot of steering and baby sitting? Absolutely not, not quite there not even close in my experience. But we should stop talking about 1s and 0s, especially with marketing hype trains, there exist a gradient of capabalities that agents have that really depends on the intricacies of the codebase you're working on, I think everyone has yet to discover how to better apply these tools in their day to day work. But that totally collides with the current narrative, that flattens out our work to be always the same and that can be automated easily in each case, it's not! That's why the debate is so polizered imo, there isn't a shared experience
- Razengan 5mo ago
  > It's since November 2025, the so called "inflection point", that I'm still wondering for who coding agents become "really good". You can dig up my past comments semi-arguing with simonw where I said AI just isn't good enough yet, but lately I've been using Codex mostly just to review existing Godot/GDScript code: https://github.com/InvadingOctopus/comedot https://github.com/InvadingOctopus/comedot and now I'd say that in this day and age one would have to be dumb to not use AI in SOME way :) It's helped me catch a lot of bugs that would have taken me a long time to even notice on my own. I guess it helps that the project is modular enough where most files can be considered standalone, with just 1-2 dependencies and well-commented already, so the AI can look at each file on its own one at a time. You can see the AGENTS.md I use on that repo. Most of my productivity in the last 3 or so months has been thanks to AI, though none of the code there is AI generated. I even bought a MacBook Neo just to use as an "AI thin client" while on travel, even though I already had a beefy MacBook Pro M2 Max that I just keep at home/hotel as a desktop now. Codex's recent remote control features have made it more useful for the moments when I get a cool idea while out at a cafe or on a walk. I don't just copy-paste the AI's output, because it's often inefficient anyway (like creating redundant variables/functions), but I find its findings useful for manually cleaning up my shit. Maybe their training data is not that good with GDScript yet which is a bit of a jank language anyway. So my core code is wholly made by meat, but I do have fun now and then telling Codex to make experimental games using only the library of modular components I have written so far, to test my framework and also the AI's abilities. This kind of work seems like a surprisingly good match for AI: It just has to put existing blocks together, that already have well-defined interfaces/contracts etc. I've been on the $20 ChatGPT plan for about a year now, and only started using Codex since like maybe 4 months ago, almost always on the latest model with "Extended Thinking" or "Extra High", because I want my shared code to be as correct as possible because everything else I do depends on it, and I only hit limits like 2 times in the last 3 months. Claude on the other hand, terrible: https://i.imgur.com/jYawPDY.png https://i.imgur.com/jYawPDY.png Grok is OK for general stuff, never tried it for coding. Gemini's UI/UX and lack of privacy and the AI itself is so terrible I tried it just maybe 2 times ever...and it refused to work on Google's own Flights website and reverse image search! (it told me to do it myself) Deepseek refused to talk about Taiwan or Tiananmen Square so I'm not sure if I can trust it for anything else lol
ionwake 5mo ago
why is there no talk about the world is already run by AI by proxy? ie bureaucrats using chatgpt to make their speeches decisions shopfront designs etc. I just dont seem to read about this, intead its more this nebulous specific date in the future
inglor_cz 5mo ago
"there’s zero chance any AI lab would train a model for such a ridiculous task" Hmm, given how small the nerd community is and how often I met that task either on Hacker News, or on various Substacks, I am not so sure that the AI labs would ignore it completely.
bob1029 5mo ago
It definitely seems like the point of no return has been passed. The size of the codebase doesn't matter anymore. In fact, I am finding that the larger the codebase the better the performance. Starting from scratch with vague ambition is not the same as solving a specific stack trace over a mountain of decade-old code. The later performs better and is also more exciting for the business. It would seem more callers = more constraints to verify against. For the last 3 months I've felt like I've been dropping gps guided bombs from orbit. No one can tell the difference between AI authored and my hand written code, other than via the implication of the radically increased daily work volume. There's definitely AI in there, but it's like a homogeneous cybernetic blend of my work and the computer's. I own all of it, can explain all of it immediately, but I only wrote maybe 10% of it by hand. The development team should be mostly "solved" by now with regard to the AI transformation. If you are still at Home Depot picking out your proverbial hammer, it's time to start heading for the self checkout. The rest of the business is where the real money and headlines will be made at this point. AI writing code is ancient news now. Custom harnesses that business people can use to automate workflows will print a lot more money. Bringing some bacon to the rest of the business may also help to preserve your career path in these uncertain times. Remember what Jobs said about the customer. A lot of times, people don’t know what they want until you show it to them. Most people wouldn't have believed the iPhone was even remotely possible until the moment it was publicly revealed and made available for purchase. I am finding the same effect in the business with AI. What it can actually do when well engineered and applied to the domain will usually outperform the expectations of its users by a wide margin. All these fears about alignment, hallucinations, cost, ethics, the environment, my ego/career, etc., seem to melt away like some kind of luxurious chocolate once the performance becomes clear to the executive staff. I was able to convince the board with an unsolicited, 5 minute demo I didn't even personally deliver. I've never seen these people sign contracts so quickly.
zkmon 5mo ago
What real world problem is closely linked to the skill of drawing a pelican riding a bicycle?
eloisant 5mo ago
About Pelicans on bicycles: > there’s zero chance any AI lab would train a model for such a ridiculous task Well, I think this guy's tests have got enough visibility that I wouldn't be surprised if some AI models are trained on it specifically...
- shantnutiwari 5mo ago
  yeah, simon's blogs have been on the front page multiple times now, I wouldnt be surprised if all of them added s apecial case for it
hansmayer 5mo ago
TL;DR: "Coding agents got really good - here, a bunch of non-releavant slop-pictures of pelicans riding bikes as a key benchmark AND a couple of hardly relevant edge-case demo-projects of mine to prove it right! " Come on man, where is the AI writing all the code in 6 months? We're close to June and Amodei's latest statement from January does not look like going into fulfilling over the next weeks, does it now?
GistNoesis 5mo ago
Last 6 months is humanity losing control of LLMs. - Memory market cornering which mitigated the adoption of local AI despite great open model being released. - Fast penetration of IP exfiltrating tools in companies world-wide. - Developers producing more code that they can read. - Autonomous agents killing Open Source by siphoning the attention economy - Autonomous agents destroyed online communities (including HN) - Autonomous agents being used in warfare (targeting, propaganda...) - Widespread vulnerabilities discovered, Widespread supply chain attacks. - Increasing inequality, fracture in perception, Green indicators, Grim realities.
- originalvichy 5mo ago
  [flagged]
- sigmoid10 5mo ago
  If you only read bad news (i.e. mass news these days since that sells better) this will be the picture. But I have personally seen some insane stuff happen in biotech. Like, I can't believe we're lucky enough to possibly live our life in this kind of future. We already have actual therepeutics developed using Alphafold being tested right now in real clinical trials, but the next generation of stuff that will go into trials in the next 3-5 years will be insane. We will look back at current medicine like we look back at medieval times today.
  - viking123 5mo ago
    Nothing ever happens. See you in 10-30 years when people are still dying of the same shit as today like oesophageal cancer and glioblastoma. Maybe in the next century but by that time you and me both will be under the ground, and no, Amodei's doubling of human lifespan simply won't happen.
    - sigmoid10 5mo ago
      [dead]
  - biophysboy 5mo ago
nate8bit 5mo ago
The tooling has become so good though - the eco-system around the LLM. The models have become really good, yes - but it's definitely slowed in my opinion. The tooling is what really has become great - "harness" is probably the best word. When folk like Elon/Schmidt/Theil/etc. talk about singularities and industrial revolutions - it sounds extremely out of touch - or actually protective of the massive capex they've potentially sunk. EDIT: Schmidt's booed commencement speech was probably one of the most out-of-touch speeches (outside of a tech interview) I've heard.
- sharperguy 5mo ago
  Much of the recent improvement in models is in being trained specifically to make use of the tools the harnesses give them.
limack0 5mo ago
[flagged]
_josh_meyer_ 5mo ago
a 5-minute video version (with local TTS model) https://tldr-api.manatee.work/v/dmYg0U https://tldr-api.manatee.work/v/dmYg0U
alain94040 5mo ago
We all have had the client from hell: they don't know what they want, they change their requirements all the time. Whenever they have a new half-baked idea, I need to scramble and re-design the architecture. They have no clue that a small change request has a big impact on the code. Well... Now I can be that client. And let AI deal with my incomplete, always changing requirements. And get it done anyway.
kramit1288 5mo ago
top model changes every other month between Claude, GPT and gemini. but its dominated by GPT overall. Claude has taken lead in coding task but GPT 5.5 has come stronger. gemini was good in between. but its dominated by GPT 5.5 and claude overall. Coding is the area where disruption is hardest. Opencalw early this year was a major breakthrough in agentic AI and it is still making noise and becoming more mature and going toward enterprise. Agentic coding is still in adoption phase where teams are trying it , trying to make sense out of it, running it and not beleving it and eventually it is discussion point over tea. it is still in adoption phase but needle has moved from being alient to being something real which team started discussing and using it like a champ.
prettyjosn 5mo ago
[flagged]
epolanski 5mo ago
I'm tired of the pelican bench, it made sense in the beginning, but at this point it got too popular and old to consider the assumptions from one year ago (absence in the sample/training/reinforcement) to still hold.
MagicMoonlight 5mo ago
They’re definitely RL training the models on the pelican test. They patch any kind of test that shows them performing poorly by hardcoding some answers into the model.
skapadia 5mo ago
[dead]
chrisss395 5mo ago
How much of what is being generated by LLMs is actually value add? My perception is there are lots of great experiments, but little real value. + Developers are more productive, but are you all leaving work at 3p and enjoying a new found sense of work-life balance? + Companies are investing heavily in AI, yet I'm paying more for the same thing. Jamie Dimon still pays me 0% on my checking despite spending billions on AI. It may be that simply adopting AI isn't enough. Could new startups that are born-in-AI buck this trend? I wonder what Clayton Christensen would say if he were still around.
Razengan 5mo ago
AI is like Sauron's Ring: it only amplifies the user's innate abilities. It can either help you conquer the world if you were already doing that anyway or it can make you spend your life in a cave before throwing you into a fucking volcano.
abstractbill 5mo ago
"... though when it tried to animate it the bicycle bounced off into the top and the bicycle got warped." Should be the pelican bounced off.
deleted 5mo ago
[deleted]
romaniv 5mo ago
> there’s zero chance any AI lab would train a model for such a ridiculous task. A lot of people here stated that this is a ridiculous metric, but no one seems to remember that it was introduced in the initial GPT report ("Sparks of Artificial General Intelligence: Early experiments with GPT-4" [1]) by Microsoft about 3 years ago. Shortly after that it was parroted by a network of booster accounts and became a thing every clueless AI hype peddler does to "test" models. 100% marketing, 0% science. [1] https://arxiv.org/pdf/2303.12712 https://arxiv.org/pdf/2303.12712
- godelski 5mo ago
  For those curious, Simon's first public usage of it is Oct 25th, 2024[0]. While I'm not aware of any specific "pelican riding a bicycle" prompts being tested in a paper[1], the GPT paper did several SVG and tikz tests and the actual image is rather arbitrary. You wouldn't want to optimize for a singular image but also if you're doing halfway decent training a pelican riding a bicycle shouldn't be too hard to draw, and well... you can see several good examples if you look through different pages on [0]. [0] https://simonwillison.net/tags/pelican-riding-a-bicycle/?page=4 https://simonwillison.net/tags/pelican-riding-a-bicycle/?pag... [1] I'm sure there is because of Simon's fame
- joe_the_user 5mo ago
  My own informal test when generative AI came out has been "a picture of an old man riding a bicycle over a river". I just ran it for chatgpt with the standard model I have (5.5). It shows the old man on an old bicycle with the bicycle on a slack line and the slack line extending over the river with a medieval village in the background. The point is that the prompt has a subtle ambiguity - "how is the old man going over the river?". My sense is that most humans would quickly imagine a conventional bridge with a road on it leading over a river and with the river background being in an area developed enough to allow bridge going over it. So the implication I draw is these things can find/generate stuff that roughly satisfies the conditions (and are getting better at this) but they still fail add the assumptions that people would draw. So my conclusion is that LLMs are getting better and better at "what they" but there are going to be places where they fail to satisfy human common assumptions.
Shocka1 5mo ago
> One of my projects was a vibe-coded implementation of JavaScript in Python—a loose port of MicroQuickJS—which I called micro-javascript. You can try it out in your browser in this playground. I'd like to remind everyone here that people on this forum used to actually code truly remarkable and pointless stuff like this, with zero LLMs, using nothing but their brains and motivation from who the heck knows where from.
qiine 5mo ago
>pelicans can’t ride bicycles... and there’s zero chance any AI lab would train a model for such a ridiculous task. humm
subarctic 5mo ago
Is there a video of this talk?
max_unbearable 5mo ago
The honest summary that doesn't show up in the six-month roundup: the unevenness. Boilerplate, tests, scaffolding, glue code: dramatically faster, sometimes 5-10x. Architecture, data modeling, careful security work, judgment calls about what to build: same as before, sometimes slower because tab-completion sneaks in plausible-but-wrong defaults you then have to undo. The thing headline numbers ("AI made me 3x faster") hide is which 30% of the work the AI sped up and which 70% didn't move. For a solo dev the survivable bet got smaller, and that's the real change, not raw productivity. AI made certain projects worth attempting at all that wouldn't have been viable six months earlier.
3l3ktr4 5mo ago
I'm always surprised to see HN people saying models aren't good. What are these guys building? The best engineers I know, from startup to big tech admit these models are incredible. Including people I don't know personally, foundational engineers from every area. The average HN person though, is doing some quantum-alien computation that not even the best developers in the world can grasp.
atlasagentsuite 5mo ago
[dead]
exabrial 5mo ago
There's also an inflection point in Feb-April: Claude got considerably worse, and arguably has not really recovered since then. They claim it's fixed, but my experience it is not as great as it once was. 4.7 is still useless. Waiting for the next event at this point. Hoping that "inference becomes cheap" when Groq hardware gets delivered.
ivandotcodes 5mo ago
Reading through the thread, a lot of the inflection point debate seems to come down to people talking past each other about what got better. My read is that the models themselves didn't really jump in capability around November, but the harnesses around them got considerably more reliable, and the RLVR work earlier in 2025 had been training the models specifically to behave well inside those harnesses, so when the two met you got a compounding effect that felt like a step change even though neither piece was that dramatic on its own. I think that's probably why everyone in this thread has such different experiences - someone whose workflow is mostly asking a model for code and pasting it in would have seen modest improvement and would reasonably wonder what the fuss is about whereas someone who was already running agents on 20-step loops would have felt a much bigger shift, because the thing that used to kill those runs was the failure at step 12 cascading into garbage by step 20, and that got a lot better. The local model story Simon kind of glosses over is interesting for the same reason - a 20GB model drawing a decent pelican on a laptop is a cute data point in isolation. The thing worth noticing is that a competent local model inside a good harness now gets you closer to frontier performance than running the frontier model without a harness does.
bschwindHN 5mo ago
https://imgur.com/a/UlGcBou https://imgur.com/a/UlGcBou
pamcake 5mo ago
> I put together these annotated slides from my five minute lightning talk at PyCon US 2026 Is there a video or audio of this talk?
ErystelaThevale 5mo ago
[flagged]
jonnyasmar 5mo ago
This is most impressive because the last 6 months in LLMs has actually been more like a hyper-compression of decades of tech progress.
jnakano89 5mo ago
[dead]

804 points by yakkomajuri 5mo ago

[dead]

It’s good to see dates being hard coded re. Improvements in the models that should deliver material gains. As time progresses one now has a yard stick to measure against progress. No more excuses - show me the money baby.

bb88 5mo ago

I met Simon for the first time this year at pycon. Wow, what a great guy.

zarzavat 5mo ago

Somewhere right now some human artist is being tasked with drawing illustrations of pelicans riding bicycles to be used as training data at a big AI lab.

minimaxir 5mo ago
Every modern image-generation model can generate a pelican on a bicycle trivially. The point of the test is to generate SVG text that represents an image, which is more complicated. Yes, there are ways to convert raster images to SVG for use in training data but it's not a good use of anyone's time.
- jofzar 5mo ago
  I wouldn't wish creating a svg pelican on a bicycle on my worst enemy
- Mashimo 5mo ago
  > Every modern image-generation model can generate a pelican on a bicycle trivially. Mistral seems to be the exception. Their new model from a few weeks ago is worse then selfhosted gemma.
- Antibabelic 5mo ago
  I don't understand this response. Human artists can and do make SVGs.
  - captn3m0 5mo ago
    They typically use a visual editor like Inkscape with visual feedback. Nobody is hand-coding a complex SVG.
    - Antibabelic 5mo ago
      The end result is the same: an SVG file. Definitely doesn't matter to an LLM what produced it.
energy123 5mo ago
The quality of the Gemini pelican was such a step change in one iteration, while the other benchmarks remained quite flat, that I think you are right. Although whether they targeted Pelicans in particular or just svg, I can't say.
- deleted 5mo ago
  [deleted]

shepherdjerred 5mo ago

> and there’s zero chance any AI lab would train a model for such a ridiculous task. I'm not sure that's true anymore considering how popular Simon's blog is

nickvec 5mo ago
Simon mentions further along in his article that given Jeff Dean’s post referencing the pelican-riding-a-bike task (and how good current models are at doing it), that it’s no longer a great benchmark to use. Enter the opossum riding an e-scooter!
- aaronbrethorst 5mo ago
  Banana man on the Segway
_puk 5mo ago
> So maybe the AI labs have been paying attention after all! > I think this mainly demonstrates that the pelican on the bicycle has firmly exceeded its limits as a useful benchmark. As acknowledged in the article.
- kzrdude 5mo ago
  Gemini 3.1 basically takes it home on that benchmark, anyway, it's done.
  - sunaookami 5mo ago
    Gemini is heavily benchmaxxed and sucks in agentic coding so no surprise.
simonw 5mo ago
That bit probably works better in the talk, it was a setup for a joke later on.
muzani 5mo ago

throwaway2027 5mo ago

December 2025 was the breakthrough for me. January Claude was euphoric, ChatGPT was up there. February Gemini cooked for a second there. March amazing. April the big bad nerf. May GPT 5.5 is just pure bliss altough 2x limits temporarily, not sure about Claude it's sort of okay still not as good as it felt before, slowly increasing limits with more compute and rebuilding good will.

dmpk2k 5mo ago
I find your emotional language truly quite fascinating. I've heard people talk like that about drugs.
- wilg 5mo ago
  Similarly, I've heard people talk like that about things that are not drugs.
  - sph 5mo ago
    You can get a dopamine rush from anything, from drugs to using LLMs.
- OvervCW 5mo ago
  I actually thought it was a joke comment, but I'm worried now that it's not the case.
_puk 5mo ago
I think Opus 4.6 at its peak was the "how can anyone not get that this is good" for me. Then the nerf, and the massive uplift in tokens for 4.7, a model which I find lazy and prone to hallucinate. It's probably time to try GPT5.5. Like many I'm pretty heavily invested in the anthropic ecosystem at this point, which I suppose gives another strong reason to make the switch.
- Arn_Thor 5mo ago
  I was a dedicated Claude user but in March/April I started using GPT5.5 on a new project that Claude had tried and failed to execute successfully. GPT knocked it out of the park, and was able to do it within my subscription allocation of tokens. I'd recommend giving it a go at least. Something like OpenClaude can let you use the Claude tools you're used to

aizk 5mo ago

I'm so glad Simon is documenting this. The field is evolving so fast, so rapidly, so hungry for data and money, that few are willing to zoom out and document everything big picture so we can see the changes over time. I mean do you guys remember "Do anything now"? Just a distant memory, a funny party trick.

Insanity 5mo ago

I wonder how much the 'inflection point' is a thing vs marketing. I'm sure the models got somewhat better, but even now when I'm trying to 'vibe code' a game with the latest models (combination of Codex w/ gpt5.5 and gpt5.3-codex), they really do struggle. They definitely get something barebones up and running, but it's far from a fully fledged application.

righthand 5mo ago
I mean this blog post and many from this author are pure evangelism and marketing. Can you find anything critical or any dissent from this author about LLMs?
adgjlsfhk1 5mo ago
It's very real. Just in the past 2 months or so IMO there's been a pretty big improvement in claude for local dev (although I think a lot of that is less model strength and more harness capability). 1m context is a huge difference (~30 min vs 2.5hr between compact significantly increases the scope of what I get the AI to do before it goes stupid). The other biggest difference I've noticed is a better balance of actually doing the work vs pushing back on bad ideas. I want the AI to tell me if it thinks the thing I am telling it is wrong or a bad idea, but if I confirm, I want it to do that anyway. A couple months ago, the claude was a lot more likely to either say "This is too much work I'm not going to do all of it", tell me the idea was genius (and then pretend to do it) or something equally useless.
- DeathArrow 5mo ago
  >1m context is a huge difference (~30 min vs 2.5hr between compact significantly increases the scope of what I get the AI to do before it goes stupid) I think the smart zone stays within the first 100k tokens, no mater if the context window is 240k or 1 million. I divide the work to fit within that 100k and use subagent for the tasks.
  - danielbln 5mo ago
    In my experience it's more like 400-500k tokens.

rTX5CMRXIfFG 5mo ago

Am I crazy, or are these differences between the best models so marginal that you’d get roughly the same performance if you use the same high-quality harness (ie preloaded instructions from md files, including custom skills)?

Sparkyte 5mo ago
No you're not wrong. Many people will see what you see. Enthusiasts will see it as monumental squeezing out that last drop of performance. In my opinion I think it is okay for enthusiasts to feel that way. I'm just satisfied with getting a tool as an aid. Personal opinion we need to focus more on efficiency instead of how large or complex a model can get as that model creeps into more resource requirements. If the goal is to cost a billion dollars to operate than we've really lost the idea of what models are supposed to be achieving.
minimaxir 5mo ago
To an extent. I've had GPT 5.5 solve problems that Opus 4.7 struggled with, using an identical AGENTS.md/CLAUDE.md and no skills.
bluegatty 5mo ago
You will immediately notice the difference if you use it at the threshold. It's like most people just watching a 'starting nba player' (not superstar, but just starting player) vs one that sits on the bench. If you were to just watching them play, work out, shoot - you'd never notice the difference. Put them head to head and it's 98-54 and you start to see the patterns. It's pretty interesting actually, someone tell me what the 'science' for this is, I'm sure there is some kind of information theory at work here. Software has innumerable kinds of problems at varying level of complexity and so it provides the perfect testbed for seeing how far models can go in practice. Should add: you're very right to hint that harness, tooling, and models tuned o both the harness and he kinds of things people do on the harness, as well as some other things do make enormous difference. Bu and large, SOTA Codex/Claude Code are substantially better - at least for now. That may change.
- 5mo ago

bluegatty 5mo ago

'Producing Images' or even 'Some Code that is Valid and Compiles' is in some ways one of the most misleading ways we assess quality of the AI. It is getting very good at producing code that compiles - at the algorithmic level. This is definitely noteworthy - and the AI is crossing a critical 'productivity threshold'. But 'Drawing of a Proper Duck' is almost arbitrary because it may have nothing to do with the 'Specific Duck You Wanted'. Everyone has tried to get AI to 'Draw The Thing They Want' and you notice immediately how it's almost impossible to 'adjust the image' along the vector you want - because ... and this is key: -> the AI doesn't really understand what a Duck is, it's components, or fully how it made the duck <- It just knows how to 'incant' the duck. This becomes very clear when you try to get the AI to write proper documentation - it fails so miserably, even with direct guidance. This is really strong evidence of how poorly the AI is generalizing, and that it is not 'understanding' rather it's 'synthesizing' from patterns. We already kind of knew that - but we have not yet built an intuition for that until now. Only now can we see 'how amazing the pattern synthesis' is - it's almost magic, and yet how it falls off a cliff otherwise This has deep implications for the 'road ahead' and the kinds of things we're going to be able to do with AI. In short: the AI is 'Wizard Level Code Helper, Researcher, and Worker' - but it very clearly lacks capabilities even one level of abstraction above the code itself. LLMs were first trained by 'text' and now ... they are 'trained by our compilers'. Basically g++, javac, tsc are the 'Verifiable Human Rewards' in the post-training and reinforcement learning - and the AI is getting extremely good at producing 'code that compiles', but that's definitely an indirection from 'code that does what we want'. It's astonishing that it took us all this time to internalize and start to discover what I think will be in hindsight a very obvious 'threshold' of it's capabilities. We are constantly 'amazed' at the work that it can do, and therefore over-project it's capabilities. I have no doubt that even with these limitations - the AI will unlock a lot more as it gets better - and - that it will 'creep up' the layers of abstraction of it's understanding. But I strongly believe that the AI is going to get much 'wider' (pattern matching dominance) before it gets 'higher' (intrinsic understanding) - and - that this may be a fundamental limitation. This may be 'the Le Cunn' insight - when he talks about the limitations of LLMs in detail - I believe this is that insight writ large. Even the term AI - or certainly 'AGI' may be a misleading metaphor - were we to have always called it 'Stochastic Algorithms' or something along those lines, it's possible that our intuition would be framed a bit better. The most interesting thing is how it is definitely amazing, world changing, novel and powerful and some ways - and obviously useless in others at the same time. That's the 'threshold' we need to better understand.

DeathArrow 5mo ago

Apart from GLM 5.1 and Qwen 3.6, there are other Chinese models that are noteworthy: Kimi K2.6, Xiaomi MiMo V2.5 Pro, Deepseek v4 and MiniMax M2.7.

simonw 5mo ago
100% true - I only had five minutes so I had to edit it down to just a couple, but all of those models are excellent and keep leap-frogging each other.
- rahimnathwani 5mo ago
  Looking forward to next time, hoping you mention speculative decoding and MTP :) It would support your point about the performance of 20GB local models.

ex-aws-dude 5mo ago

Is the RLVR the key breakthrough for the uplift or is there more to it? Does that suggest the uplift was only for things that are easily verifiable like code?

4b11b4 5mo ago
RL we're gonna find out will get abandoned cuz we don't even know what is getting "aligned", just my naive gut feeling don't take it seriously
rdedev 5mo ago
I would say that most improvements are in easily verifiable things like code or math. Atleast that's where all the amazing results seem to be coming from. Other domains I am not sure but I've heard from people like Cal Newport that the rate of increase outside of code and math are not as equally impressive
vanuatu 5mo ago
Yes, with good RLVR at scale you can greatly improve performance especially on benchmarks The hope was that good RLVR on relatively contrived datasets (like benchmarks) would be generalized to good software taste, which has somewhat succeeded but also the models fail in horrible ways still And the hope beyond that is that good skills in fundamental problem solving tasks (coding, math) would generalize to tasks beyond math and code, which did happen but less so

tayo42 5mo ago

The claw thing really came and went fast lol

deleted 5mo ago
[deleted]
yieldcrv 5mo ago
I just started a new job and the person I report to was just excited to tell me about it, here in Mid May "and then you have to get a mac mini, and then, and then" smile and nod, it pays weekly
viking123 5mo ago
I mean yeah? It was marketing campaign to boost the model providers and give Steinberger a cozy job at OpenAI. Hook, line and sinker. Wake me up when we have an agent with constant learning and changing weights that I can have personally, not some LLM that can always fall prone to jailbreak and context injection attacks. You think most of this stuff here is organic? Oh boy..

DeathArrow 5mo ago

I think that there's a lot to be improved in harnesses and the way the models are interacting with harnesses. For example, the harness should be able to steer the model when thinking.

tptacek 5mo ago

If you're a vulnerability researcher or a security person generally, there's a big inflection point from Spring of this year.

thierrydamiba 5mo ago
Can you be more specific?
- simonw 5mo ago
  The Claude Mythos / Project Glasswing thing is real: https://www.anthropic.com/glasswing https://www.anthropic.com/glasswing I met a few people at PyCon this week who have been part of Glasswing (they're just starting to be allowed to talk about it) and it really does drive down the cost of finding vulnerabilities. I've been collecting notes on that here: https://simonwillison.net/tags/ai-security-research/ https://simonwillison.net/tags/ai-security-research/
  - krzyk 5mo ago
    People in my company sounded underwhelmed by it. It usually was founding issues by not understanding deployment (or not being fed that info).
    - halflife 5mo ago
      A friend of mine had hands on experience, it’s not the intelligence of it, it’s the speed. You used to have a couple of days to close a breach, now it 2 hours.
    - vessenes 5mo ago
      You might point them at the cloudflare blog about deploying mythos - I found it interesting. Upshot — as your folks discovered, deployment, harness, and utilization method matters for mythos and is a bit different than how you’d deploy a coding agent for writing code, but if you do that, you get something capable of end to end chaining and reasoning about a much broader class of vulnerabilities. No personal experience with it. But the security team writeups I’ve read are significantly more positive about it than you describe, so it might be worth a second look.

grey-area 5mo ago

Haven’t noticed much significant progress in LLMs myself in 6 months (significant as in new or vastly improved capabilities or understanding, not new releases, there are plenty of those). I feel like if anything people started to realise the significant limitations of LLMs when you try to use them as ‘agents’ which was the big direction LLM companies tried to push recently. Best use of LLMs so far IMO is finding vulnerabilities (with human help) and pattern matching in other domains. For generating code and prose they are still mediocre and somewhat unreliable and for use as personal assistant agents I wouldn’t trust them. So what’s happening with openclaw, the biggest experiment in agentic, vibe coded by the agents themselves? The thing that was so hot a few months ago. https://github.com/openclaw/openclaw/pulse?period=daily https://github.com/openclaw/openclaw/pulse?period=daily 279 commits to main from 77 authors in the last 24 hours. Why is there so much churn and how could you trust it with your data? This is changes in ONE day! If these are useful changes, surely it’d be superhuman by now given months of this pace. What are people using this for?

deleted 5mo ago
[deleted]

vishal_new 5mo ago

what are your thoughts on Software engineer replacement. My team has already seen big reductions. Q/A team is gone. Software Engineer reduced by a third. Scared for the future

ShinyLeftPad 5mo ago
If you're famous, you'll be fine. If you're in retiring age, you don't care. Otherwise, good luck! We put ourselves on the street by not protesting what is happening.
- munksbeer 5mo ago
  I think the general population earning median wages will have very little sympathy for first world software engineers earning vast amounts of money. What are you going to tell them? Suddenly you're earning what they're earning for sitting at a desk every day?
  - ShinyLeftPad 5mo ago
    General population, you mean non SWEs? Because there are many SWEs around the world who earn median wage and who stand to lose it all as the avalanche of firings is ramping up. Non SWEs (salespeople, clerks, secretaries, assistants, taxi drivers, writers, 3D modelers, artists, designers) are of course going the same way. Unless they are protected (unionized or such), why would they have sympathy for SWEs? People of our ilk are the ones causing this (to them and to ourselves). What I will tell them is to not repeat our mistake, organize and protest.
simonw 5mo ago
Ditching the QA team when the single highest challenge is verifying that vibe-coded systems do what they're meant to is extraordinarily short-sighted. Personally, the more time I spend working with coding agents the least worried I am for my career. Getting the best results out of them is really hard. They amplify existing skills and experience, so the more experience you have the better.

dnnddidiej 5mo ago

Also LinkedIn wars of people trying to claim throne as most AI-pilled, throwing down strawmen stories of luddites yelling at data centres who'll lose their job to a single person doing 100x work.

LZ_Khan 5mo ago

I'm curious how the 6 months have looked from a non-programmer's perspective. What kind of co-working tools and similar optimizations have people from other fields experienced?

deleted 5mo ago
[deleted]
conception 5mo ago
Claude in Office was a tipping point for nontechnical folks around me. Everyone’s slides decks are immaculate now. Finance isn’t needing nearly as much BI help. It’s pretty impressive.
- RobinL 5mo ago
  Interesting. I don't have to use PowerPoint much, but I hate it when I do. I don't want the llm to write the words but I do want it to make things look nice. So does this work well now?
  - angled 5mo ago
    My pipeline for this is vscode + prompts + markdown templates + GitHub copilot -> markdown docs -> pandoc to produce.docx -> copilot in word for “nice” formatting -> copilot in ppt for nice decks. LLMs all the way down. I find it’s easier to version control and diff the .md artefacts, those remain my authoritative source.
    - asdff 5mo ago
      Wow. Seems like a headache compared to how I make slides the old fashioned way: copy and paste my figures into blank powerpoint.
    - vessenes 5mo ago
      I was doing something like this, and then realized at least with claude that it’s so much better at HTML that it’s better to get an HTML-first deck together, which could then be turned into a PPT template and/or PDF directly, depending on needs. It saved me a fair number of design-tweak steps in the md -> pandoc part of the workflow. Realistically, hand editing claude’s HTML is also easy in most cases, so I didn’t feel like I lost much (for the generative cases). Similarly if it’s mostly what I’ve written directly that’s the source it’ll be in markdown, and I’ve found it’s a faster path to have md -> (LLM-translated HTML deck) -> pdf.

pr337h4m 5mo ago

Something that’s largely been ignored: DeepSeek has made context caching virtually free with V4-Flash.

hootz 5mo ago
I swear to god that DeepSeek V4-Flash is the most useful model available right now. It's SO FAST and is good enough for so many tasks that I run it most of the time for almost everything. Even when it messes up, it's so cheap to iterate that I can fix most problems without changing the model to a more "capable" one.

gib444 5mo ago

Starting from zero today, how would someone quickly get upto speed with the latest and greatest AI tooling on an extremely limited budget? Is the only choice to pay for the "max" plans? Or just read so much about it that you bs your way through an interview and then use the company's resources? Simon, I'm curious too how much you invest each month researching all the latest and great AI tech?

RobinL 5mo ago
$20 chatgpt pro plan gives pretty generous usage both of codex, general chat
- gib444 5mo ago
  Ah I'd read so much about the downgrading of that plan I didn't think that was still true?
  - azuanrb 5mo ago
    It depends on what you’re comparing it against. For $20, OpenAI is still probably the best value for SOTA models. In terms of limits, you can use GPT-5.4 instead of 5.5. The intelligence feels similar, but it’s cheaper. You can also experiment with other harnesses like pi. It’s lightweight but capable enough, and its token usage is definitely much more efficient.
x86cherry 5mo ago
Opencode has free access to Qwen 3.6 and Deepseek v4 Flash right now. They're on par with Claude and Codex imo - when you still design architecture and know what the output should be. Claude and GPT 5.5 need less guidance with vibe coding, but we're not yet at a point where that's sustainable anyway even with those models.
- gib444 5mo ago
  Thanks, playing with Opencode now. It just wrote a half-decent Android small app. Pretty good so far!

bunzee 5mo ago

Spot on. Building our tool, we found AI is magic at scraping competitor data, but terrible at market validation. The 'why' is strictly human.

2001zhaozhao 5mo ago
So, the best way to use LLMs is to wait for your competitors to do market validation and then scrape their data. Hmmm......
- tardedmeme 5mo ago
  It's always been much easier to copy an existing product than to make a new one nobody's thought of before.
kkarpkkarp 5mo ago
sorry but how this comment refers to the commented post?

jrowen 5mo ago

There's something fitting about the mystical nature of LLMs and scrolling through a bunch of goofy pelicans on bicycles representing report cards for the bleeding edge of technology. How are these even graded? Qwen3.6-35B-A3B gets high marks for a pelican with a gaping hole in its bill? edit: Just noticed its feet are disconnected from its legs as well (but right on the pedals!). Pardon my French but that's Chinese af.

LarsDu88 5mo ago

My goal post for "AI will definitely replace most SWEs" was to reproduce a particular 90s programming game one shot and then add multiplayer support with minimal prompting. Opus 4.5 hit that point in November.

grey-area 5mo ago
I tried this a while ago, haven’t tried again recently. The models were producing code that was clearly lifted from stuff in their training data, and what I ended up with was a fairly decent game in html and js after a bit of tidy up, though it felt like several code paradigms smooshed together rather than a coherent whole, but it mostly worked. Not something I’d want to maintain but it was impressive at the time. They were able to one-shot famous games (like asteroid or pong), I suspect because they had been trained on multiple versions of that game. So like producing Harry Potter, with the right prompt it was able to produce a license stripped version of code it had seen. I tried another arcade game like frogger and it failed really badly and took a lot longer, never got it working. The whole exercise left me feeling they have a long way to go, I don’t see how anyone could think they would replace SWE unless they didn’t look at the code produced, even now.
- vessenes 5mo ago
  Out of curiosity - what harness did you use, and what model? And how are you prompting? In my mind prompting like: “You’re going to make frogger in javascript. I want a complete clone of functionality for level 1, with amazing 80s era pixel art sprites. I’m super lazy, so you’re going to have to test everything, right from the start. Pick a test harness, write the tests, including tests for having amazing graphics, gameplay, input, UI, sounds, etc, and write a full workplan, then work through that workplan, in parallel where you can. The workplan should emphasize getting a stripped down version up immediately and have workstreams for all the major requirements after that. Add a final test that assesses how fun the game is by reviewing a real video of a test run. Loop on that final test until you can’t improve things any more.” Should produce something playable with no further input. As you say, I’m not sure it would produce a codebase we’d want to look at or work on. But, I’d be surprised if this weren’t successful.

raghavchamadiya 5mo ago

[flagged]

pineapple_opus 5mo ago

All I see is mention of how various models generate image of "pelican riding bicycle(s)"

emil-lp 5mo ago
Yes, the "pelican riding a bicycle" is the ultimate test of not understanding how LLMs work. Well, a combination of that and believing that replication of test data is a good measure of progress.
- vessenes 5mo ago
  Spicy — why does it show ultimate non-understanding?
  - JohnKemeny 5mo ago
    because success comes from reproducing a memorized pattern rather than transferable reasoning? At the same time failure proves little because most humans also could not manually create a correct SVG of a pelican riding a bicycle. What is it exactly that such a test is testing? In which situation would you measure the "competence" of a human being by asking them to write an SVG of a pelican riding a bicycle?
    - okamiueru 5mo ago
      > most humans also could not manually create a correct SVG of a pelican riding a bicycle. Most humans absolutely can write this with a suitable vector graphics tool such as inkscape or illustrator. Surely, you're not suggesting that a fair comparison would be using a text editor? If so, would you suggest an equivalent raster based task would only be fair, if the human would manually assigning RGB values to each pixel?
ClikeX 5mo ago
We all know the true test of AI is Will Smith eating spaghetti.
5mo ago

nothinkjustai 5mo ago

[flagged]

rhubarbtree 5mo ago
Certainly a massive AI booster. What Are the conflicts of interest?

nothinkjustai 5mo ago

[flagged]

wewewedxfgdf 5mo ago

Does this guy have a "publish to front page of HN" button on his blog editor?

troupo 5mo ago
He has the most measured (and often quite detailed) posts on LLM and LLM progress, and is the opposite of hype.
nickvec 5mo ago
He’s pretty well known in the HN community. https://en.wikipedia.org/wiki/Simon_Willison https://en.wikipedia.org/wiki/Simon_Willison
- koolala 5mo ago
  thats a cool wiki picture
koolala 5mo ago
its better than ex-google CEO spam i see astroturfed everywhere else
specproc 5mo ago
He's one of the main developers behind Django.
- dcminter 5mo ago
  Years ago I used to read his blog on Django and found it quite interesting despite being neither a Django nor even a python user - this must have been at least 10 years ago and perhaps more. When he resurfaced in my feeds as an AI commentator it took me quite a long while to join the dots that he was the same person!
schnitzelstoat 5mo ago
I liked the article, so if he has such a button I hope he keeps clicking it.

nothinkjustai 5mo ago

[flagged]

tomhow 5mo ago
We've banned this account. We detached this comment from https://news.ycombinator.com/item?id=48189072 https://news.ycombinator.com/item?id=48189072 and marked it off topic.

nothinkjustai 5mo ago

[dead]

ramon156 5mo ago

> Google released the Gemma 4 series of models, which are the most capable open weight models I’ve seen from a US company. Implying another country has a better model? I'm being pokey here because I'm very curious! I know Gemma is efficient, but I also remember Qwen and Kiwi being referred to as optimized. The difference being that Gemma is using less tokens, but maybe Qwen/Kiwi's quality is higher? I dont know.

pferdone 5mo ago
The consensus right now is that Qwen3.6 in its 27B and 35B-A3B versions is better for coding whereas Gemma4 is stronger when it comes to OCR, audio transcription and the likes. Margins are slim though and the harness at these model sizes is the most important factor.
0xCMP 5mo ago
In my experience the qwen models are best locally, but gemma ones have always been good. gemma4 is a notable improvement.

jimbobthemighty 5mo ago

I asked Gemini for a video of 'pelican riding a unicycle in hyde park' - I was blown away by the output: https://gemini.google.com/share/55e250c99693 https://gemini.google.com/share/55e250c99693

grey-area 5mo ago
That’s really impressive, and slightly worrying for creatives involved in film, animation or modelling.
- notachatbot123 5mo ago
  Even more worrying are the implications for fakenews, propaganda, fraud, deception and mental health.
  - sevenzero 5mo ago
    This is really my biggest worry when it gets to consumer AI. People already have a hard time informing themselves properly. Now we have technology that just boosts the already existing confirmation bias people have. It's sickening.
  - dzhiurgis 5mo ago
    Maybe short term yes. But longer term people will finally put their guard up against deception that’s been around for decades.
    - layer8 5mo ago
      People will still believe what they want to believe.
    - tonyedgecombe 5mo ago
      If they haven’t in the past I don’t see why they would now.
      - 5mo ago

bradley13 5mo ago

As someone who uses AI daily (not in agent mode, just user-interactive), I have definitely noticed major quality improvements over the past few months. And that's surprising, because when you use something daily, you tend to overlook the big jumps. I haven't looked into any sort of "agent" mode, just because I don't yet quite trust the AI not to do something dumb. Also, I don't use M365, where Copilot is integrated, so I suppose I would have to set it up myself.

hollowturtle 5mo ago

> The coding agents got really good It's since november 2025, the so called "inflection point", that I'm still wondering for who coding agents become "really good". All I observe they got better at tool call and answering questions about big codebases, especially if the question has a vague pattern to search, and they're superuseful for that! For generating production code even with a lot of steering and baby sitting? Absolutely not, not quite there not even close in my experience. But we should stop talking about 1s and 0s, especially with marketing hype trains, there exist a gradient of capabalities that agents have that really depends on the intricacies of the codebase you're working on, I think everyone has yet to discover how to better apply these tools in their day to day work. But that totally collides with the current narrative, that flattens out our work to be always the same and that can be automated easily in each case, it's not! That's why the debate is so polizered imo, there isn't a shared experience

Razengan 5mo ago
> It's since November 2025, the so called "inflection point", that I'm still wondering for who coding agents become "really good". You can dig up my past comments semi-arguing with simonw where I said AI just isn't good enough yet, but lately I've been using Codex mostly just to review existing Godot/GDScript code: https://github.com/InvadingOctopus/comedot https://github.com/InvadingOctopus/comedot and now I'd say that in this day and age one would have to be dumb to not use AI in SOME way :) It's helped me catch a lot of bugs that would have taken me a long time to even notice on my own. I guess it helps that the project is modular enough where most files can be considered standalone, with just 1-2 dependencies and well-commented already, so the AI can look at each file on its own one at a time. You can see the AGENTS.md I use on that repo. Most of my productivity in the last 3 or so months has been thanks to AI, though none of the code there is AI generated. I even bought a MacBook Neo just to use as an "AI thin client" while on travel, even though I already had a beefy MacBook Pro M2 Max that I just keep at home/hotel as a desktop now. Codex's recent remote control features have made it more useful for the moments when I get a cool idea while out at a cafe or on a walk. I don't just copy-paste the AI's output, because it's often inefficient anyway (like creating redundant variables/functions), but I find its findings useful for manually cleaning up my shit. Maybe their training data is not that good with GDScript yet which is a bit of a jank language anyway. So my core code is wholly made by meat, but I do have fun now and then telling Codex to make experimental games using only the library of modular components I have written so far, to test my framework and also the AI's abilities. This kind of work seems like a surprisingly good match for AI: It just has to put existing blocks together, that already have well-defined interfaces/contracts etc. I've been on the $20 ChatGPT plan for about a year now, and only started using Codex since like maybe 4 months ago, almost always on the latest model with "Extended Thinking" or "Extra High", because I want my shared code to be as correct as possible because everything else I do depends on it, and I only hit limits like 2 times in the last 3 months. Claude on the other hand, terrible: https://i.imgur.com/jYawPDY.png https://i.imgur.com/jYawPDY.png Grok is OK for general stuff, never tried it for coding. Gemini's UI/UX and lack of privacy and the AI itself is so terrible I tried it just maybe 2 times ever...and it refused to work on Google's own Flights website and reverse image search! (it told me to do it myself) Deepseek refused to talk about Taiwan or Tiananmen Square so I'm not sure if I can trust it for anything else lol

ionwake 5mo ago

why is there no talk about the world is already run by AI by proxy? ie bureaucrats using chatgpt to make their speeches decisions shopfront designs etc. I just dont seem to read about this, intead its more this nebulous specific date in the future

inglor_cz 5mo ago

"there’s zero chance any AI lab would train a model for such a ridiculous task" Hmm, given how small the nerd community is and how often I met that task either on Hacker News, or on various Substacks, I am not so sure that the AI labs would ignore it completely.

bob1029 5mo ago

It definitely seems like the point of no return has been passed. The size of the codebase doesn't matter anymore. In fact, I am finding that the larger the codebase the better the performance. Starting from scratch with vague ambition is not the same as solving a specific stack trace over a mountain of decade-old code. The later performs better and is also more exciting for the business. It would seem more callers = more constraints to verify against. For the last 3 months I've felt like I've been dropping gps guided bombs from orbit. No one can tell the difference between AI authored and my hand written code, other than via the implication of the radically increased daily work volume. There's definitely AI in there, but it's like a homogeneous cybernetic blend of my work and the computer's. I own all of it, can explain all of it immediately, but I only wrote maybe 10% of it by hand. The development team should be mostly "solved" by now with regard to the AI transformation. If you are still at Home Depot picking out your proverbial hammer, it's time to start heading for the self checkout. The rest of the business is where the real money and headlines will be made at this point. AI writing code is ancient news now. Custom harnesses that business people can use to automate workflows will print a lot more money. Bringing some bacon to the rest of the business may also help to preserve your career path in these uncertain times. Remember what Jobs said about the customer. A lot of times, people don’t know what they want until you show it to them. Most people wouldn't have believed the iPhone was even remotely possible until the moment it was publicly revealed and made available for purchase. I am finding the same effect in the business with AI. What it can actually do when well engineered and applied to the domain will usually outperform the expectations of its users by a wide margin. All these fears about alignment, hallucinations, cost, ethics, the environment, my ego/career, etc., seem to melt away like some kind of luxurious chocolate once the performance becomes clear to the executive staff. I was able to convince the board with an unsolicited, 5 minute demo I didn't even personally deliver. I've never seen these people sign contracts so quickly.

zkmon 5mo ago

What real world problem is closely linked to the skill of drawing a pelican riding a bicycle?

eloisant 5mo ago

About Pelicans on bicycles: > there’s zero chance any AI lab would train a model for such a ridiculous task Well, I think this guy's tests have got enough visibility that I wouldn't be surprised if some AI models are trained on it specifically...

shantnutiwari 5mo ago
yeah, simon's blogs have been on the front page multiple times now, I wouldnt be surprised if all of them added s apecial case for it

hansmayer 5mo ago

TL;DR: "Coding agents got really good - here, a bunch of non-releavant slop-pictures of pelicans riding bikes as a key benchmark AND a couple of hardly relevant edge-case demo-projects of mine to prove it right! " Come on man, where is the AI writing all the code in 6 months? We're close to June and Amodei's latest statement from January does not look like going into fulfilling over the next weeks, does it now?

GistNoesis 5mo ago

Last 6 months is humanity losing control of LLMs. - Memory market cornering which mitigated the adoption of local AI despite great open model being released. - Fast penetration of IP exfiltrating tools in companies world-wide. - Developers producing more code that they can read. - Autonomous agents killing Open Source by siphoning the attention economy - Autonomous agents destroyed online communities (including HN) - Autonomous agents being used in warfare (targeting, propaganda...) - Widespread vulnerabilities discovered, Widespread supply chain attacks. - Increasing inequality, fracture in perception, Green indicators, Grim realities.

originalvichy 5mo ago
[flagged]
sigmoid10 5mo ago
If you only read bad news (i.e. mass news these days since that sells better) this will be the picture. But I have personally seen some insane stuff happen in biotech. Like, I can't believe we're lucky enough to possibly live our life in this kind of future. We already have actual therepeutics developed using Alphafold being tested right now in real clinical trials, but the next generation of stuff that will go into trials in the next 3-5 years will be insane. We will look back at current medicine like we look back at medieval times today.
- viking123 5mo ago
  Nothing ever happens. See you in 10-30 years when people are still dying of the same shit as today like oesophageal cancer and glioblastoma. Maybe in the next century but by that time you and me both will be under the ground, and no, Amodei's doubling of human lifespan simply won't happen.
  - sigmoid10 5mo ago
    [dead]
- biophysboy 5mo ago

nate8bit 5mo ago

The tooling has become so good though - the eco-system around the LLM. The models have become really good, yes - but it's definitely slowed in my opinion. The tooling is what really has become great - "harness" is probably the best word. When folk like Elon/Schmidt/Theil/etc. talk about singularities and industrial revolutions - it sounds extremely out of touch - or actually protective of the massive capex they've potentially sunk. EDIT: Schmidt's booed commencement speech was probably one of the most out-of-touch speeches (outside of a tech interview) I've heard.

sharperguy 5mo ago
Much of the recent improvement in models is in being trained specifically to make use of the tools the harnesses give them.

limack0 5mo ago

[flagged]

_josh_meyer_ 5mo ago

a 5-minute video version (with local TTS model) https://tldr-api.manatee.work/v/dmYg0U https://tldr-api.manatee.work/v/dmYg0U

alain94040 5mo ago

We all have had the client from hell: they don't know what they want, they change their requirements all the time. Whenever they have a new half-baked idea, I need to scramble and re-design the architecture. They have no clue that a small change request has a big impact on the code. Well... Now I can be that client. And let AI deal with my incomplete, always changing requirements. And get it done anyway.

kramit1288 5mo ago

top model changes every other month between Claude, GPT and gemini. but its dominated by GPT overall. Claude has taken lead in coding task but GPT 5.5 has come stronger. gemini was good in between. but its dominated by GPT 5.5 and claude overall. Coding is the area where disruption is hardest. Opencalw early this year was a major breakthrough in agentic AI and it is still making noise and becoming more mature and going toward enterprise. Agentic coding is still in adoption phase where teams are trying it , trying to make sense out of it, running it and not beleving it and eventually it is discussion point over tea. it is still in adoption phase but needle has moved from being alient to being something real which team started discussing and using it like a champ.

prettyjosn 5mo ago

[flagged]

epolanski 5mo ago

I'm tired of the pelican bench, it made sense in the beginning, but at this point it got too popular and old to consider the assumptions from one year ago (absence in the sample/training/reinforcement) to still hold.

MagicMoonlight 5mo ago

They’re definitely RL training the models on the pelican test. They patch any kind of test that shows them performing poorly by hardcoding some answers into the model.

skapadia 5mo ago

[dead]

chrisss395 5mo ago

How much of what is being generated by LLMs is actually value add? My perception is there are lots of great experiments, but little real value. + Developers are more productive, but are you all leaving work at 3p and enjoying a new found sense of work-life balance? + Companies are investing heavily in AI, yet I'm paying more for the same thing. Jamie Dimon still pays me 0% on my checking despite spending billions on AI. It may be that simply adopting AI isn't enough. Could new startups that are born-in-AI buck this trend? I wonder what Clayton Christensen would say if he were still around.

Razengan 5mo ago

AI is like Sauron's Ring: it only amplifies the user's innate abilities. It can either help you conquer the world if you were already doing that anyway or it can make you spend your life in a cave before throwing you into a fucking volcano.

abstractbill 5mo ago

"... though when it tried to animate it the bicycle bounced off into the top and the bicycle got warped." Should be the pelican bounced off.

deleted 5mo ago

[deleted]

romaniv 5mo ago

> there’s zero chance any AI lab would train a model for such a ridiculous task. A lot of people here stated that this is a ridiculous metric, but no one seems to remember that it was introduced in the initial GPT report ("Sparks of Artificial General Intelligence: Early experiments with GPT-4" [1]) by Microsoft about 3 years ago. Shortly after that it was parroted by a network of booster accounts and became a thing every clueless AI hype peddler does to "test" models. 100% marketing, 0% science. [1] https://arxiv.org/pdf/2303.12712 https://arxiv.org/pdf/2303.12712

godelski 5mo ago
For those curious, Simon's first public usage of it is Oct 25th, 2024[0]. While I'm not aware of any specific "pelican riding a bicycle" prompts being tested in a paper[1], the GPT paper did several SVG and tikz tests and the actual image is rather arbitrary. You wouldn't want to optimize for a singular image but also if you're doing halfway decent training a pelican riding a bicycle shouldn't be too hard to draw, and well... you can see several good examples if you look through different pages on [0]. [0] https://simonwillison.net/tags/pelican-riding-a-bicycle/?page=4 https://simonwillison.net/tags/pelican-riding-a-bicycle/?pag... [1] I'm sure there is because of Simon's fame
joe_the_user 5mo ago
My own informal test when generative AI came out has been "a picture of an old man riding a bicycle over a river". I just ran it for chatgpt with the standard model I have (5.5). It shows the old man on an old bicycle with the bicycle on a slack line and the slack line extending over the river with a medieval village in the background. The point is that the prompt has a subtle ambiguity - "how is the old man going over the river?". My sense is that most humans would quickly imagine a conventional bridge with a road on it leading over a river and with the river background being in an area developed enough to allow bridge going over it. So the implication I draw is these things can find/generate stuff that roughly satisfies the conditions (and are getting better at this) but they still fail add the assumptions that people would draw. So my conclusion is that LLMs are getting better and better at "what they" but there are going to be places where they fail to satisfy human common assumptions.

Shocka1 5mo ago

> One of my projects was a vibe-coded implementation of JavaScript in Python—a loose port of MicroQuickJS—which I called micro-javascript. You can try it out in your browser in this playground. I'd like to remind everyone here that people on this forum used to actually code truly remarkable and pointless stuff like this, with zero LLMs, using nothing but their brains and motivation from who the heck knows where from.

qiine 5mo ago

>pelicans can’t ride bicycles... and there’s zero chance any AI lab would train a model for such a ridiculous task. humm

subarctic 5mo ago

Is there a video of this talk?

max_unbearable 5mo ago

The honest summary that doesn't show up in the six-month roundup: the unevenness. Boilerplate, tests, scaffolding, glue code: dramatically faster, sometimes 5-10x. Architecture, data modeling, careful security work, judgment calls about what to build: same as before, sometimes slower because tab-completion sneaks in plausible-but-wrong defaults you then have to undo. The thing headline numbers ("AI made me 3x faster") hide is which 30% of the work the AI sped up and which 70% didn't move. For a solo dev the survivable bet got smaller, and that's the real change, not raw productivity. AI made certain projects worth attempting at all that wouldn't have been viable six months earlier.

3l3ktr4 5mo ago

I'm always surprised to see HN people saying models aren't good. What are these guys building? The best engineers I know, from startup to big tech admit these models are incredible. Including people I don't know personally, foundational engineers from every area. The average HN person though, is doing some quantum-alien computation that not even the best developers in the world can grasp.

atlasagentsuite 5mo ago

[dead]

exabrial 5mo ago

There's also an inflection point in Feb-April: Claude got considerably worse, and arguably has not really recovered since then. They claim it's fixed, but my experience it is not as great as it once was. 4.7 is still useless. Waiting for the next event at this point. Hoping that "inference becomes cheap" when Groq hardware gets delivered.

ivandotcodes 5mo ago

Reading through the thread, a lot of the inflection point debate seems to come down to people talking past each other about what got better. My read is that the models themselves didn't really jump in capability around November, but the harnesses around them got considerably more reliable, and the RLVR work earlier in 2025 had been training the models specifically to behave well inside those harnesses, so when the two met you got a compounding effect that felt like a step change even though neither piece was that dramatic on its own. I think that's probably why everyone in this thread has such different experiences - someone whose workflow is mostly asking a model for code and pasting it in would have seen modest improvement and would reasonably wonder what the fuss is about whereas someone who was already running agents on 20-step loops would have felt a much bigger shift, because the thing that used to kill those runs was the failure at step 12 cascading into garbage by step 20, and that got a lot better. The local model story Simon kind of glosses over is interesting for the same reason - a 20GB model drawing a decent pelican on a laptop is a cute data point in isolation. The thing worth noticing is that a competent local model inside a good harness now gets you closer to frontier performance than running the frontier model without a harness does.

bschwindHN 5mo ago

https://imgur.com/a/UlGcBou https://imgur.com/a/UlGcBou

pamcake 5mo ago

> I put together these annotated slides from my five minute lightning talk at PyCon US 2026 Is there a video or audio of this talk?

ErystelaThevale 5mo ago

[flagged]

jonnyasmar 5mo ago

This is most impressive because the last 6 months in LLMs has actually been more like a hyper-compression of decades of tech progress.

jnakano89 5mo ago

[dead]