5 ms·

That's a pretty trivial example for one of these IDEs to knock out. Assembly is certainly in their training sets, and obviously docker is too. I've watched curs

That's a pretty trivial example for one of these IDEs to knock out. Assembly is certainly in their training sets, and obviously docker is too. I've watched cursor absolutely run amok when I let it play around in some of my codebase. I'm bullish it'll get there sooner rather than later, but we're not there yet.

simonw 1y ago
I think the hardest problem in computer science right now may be coming up with an LLM demo that doesn't get called "pretty trivial".
- fragmede 1y ago
  I think Cloudflare's oauth library qualifies https://news.ycombinator.com/item?id=44159166 https://news.ycombinator.com/item?id=44159166
  - gen6acd60af 1y ago
    This one? >Claude's output was thoroughly reviewed by Cloudflare engineers with careful attention paid to security and compliance with standards. >To emphasize, this is not "vibe coded". Every line was thoroughly reviewed and cross-referenced with relevant RFCs, by security experts with previous experience with those RFCs. Some time later... https://github.com/advisories/GHSA-4pc9-x2fx-p7vj https://github.com/advisories/GHSA-4pc9-x2fx-p7vj / CVE-2025-4143 >The OAuth implementation in workers-oauth-provider that is part of MCP framework https://github.com/cloudflare/workers-mcp https://github.com/cloudflare/workers-mcp, did not correctly validate that redirect_uri was on the allowed list of redirect URIs for the given client registration.
    - kentonv 1y ago
      Sorry, my code has bugs sometimes.
- skydhash 1y ago
  Because they are trivial in a way that you can go on GitHub and copy one of those while not pretending LLM isn't a mashup of the internet. What people agree on being non-trivial is working on a real project. There's a lot of opensource projects that could benefit from a useful code contribution. But they only got slop thrown at them.
  - simonw 1y ago
    [flagged]
    - skydhash 1y ago
      I took the time to investigate the work being done there (all those years learning assembly and computer architecture come in handy), and it confirms (to me) that the key aspect of using LLM is pattern matching. Meaning you know that there's a solution out there (in this case, anything involving multiplying/dividing by a power of 2 can use such trick) and framing your problem (intentionally or not) and you'll get a derived text that will contain a possible solution. But there's nothing truly novel in the result. The key aspect is being similar enough to something that's already in the training data so that the LLM can extrapolate the rest. The hint can be quite useful and sometimes you have something that shorten the implementation time, but you have to at least have some basic understanding of the domain in order to recognize the signs. The issue is that the result is always tainted by your prompt. The signs may be there because of your prompt and not because there's some kind of data that need s to be explored further. And sometimes it's a bad fit, similar but different (what you want and what you get). So for the few domain that's valuable to me, I prefer to construct my own mental database that can lead me to concrete artifacts (books, articles, blog posts,...) that exists outside the influence of my query. ADDENDUM I can use LLMs with great results and I've done so. But it's more rewarding (and more useful to me) to actually think through the problem and learning from references. Instead of getting a perfect (or wobbly or the wrong category) circle that fits my query, I go to find a strange polygon formed (by me) from other strange polygon. Then because I know I need a circle, I only need to find its center and its radius. It's slower, but the next time I need another circle (or a square) from the same polygon, it's going to be faster and faster.
- th0ma5 1y ago
  Maybe you should try something other than demos? Have you tried creating a reliable system?
  - deleted 1y ago
    [deleted]
- jkhdigital 1y ago
  No the hardest problem is teaching CS undergrads. I just started this year (no background in academia, just 75% of a PhD and well-rounded life experience) and I’ve basically torn up the entire curriculum they handed to me and started vibe-teaching.
- 1dom 1y ago
  I'm very pro LLM and AI. But I completely agree with the comment about how many pieces praising LLMs are doing so with trivial examples. Trivial might not be the right word, but I can't think of a better one that doesn't have a negative connotation, but this shouldn't be negative. Your examples are good and useful, and capture a bunch of tasks a software engineer would do. I'd say your mandelbrot debug and the LLVM patch are both "trivial" in the same sense: they're discrete, well defined, clear-success-criteria-tasks that could be assigned to any mid/senior software engineer in a relevant domain and they could chip through it in a few weeks. Don't get me wrong, that's an insane power and capability of LLMs, I agree. But ultimately it's just doing a day job that millions of people can do sleep deprived and hungover. Non-trivial examples are things that would take a team of different specialist skillsets months to create. One obvious potential reason why there's few non-trivial AI examples is because non-trivial AI examples require non-trivial amount of time to be able to generate and verify. A non-trivial example isn't an example you can look at the output and say "yup, AI's done well here". It requires someone spends time going into what's been produced, assessing it, essentially redesigning it as a human to figure out all the complexity of a modern non-trivial system to confirm the AI actually did all that stuff correctly. An in depth audit of a complex software system can take months or even years and is a thorough and tedious task for a human, and the Venn diagrams of humans who are thinking "I want to spend more time doing thorough, tedious code tasks" and "I want to mess around with AI coding" is 2 separate circles.
  - sokoloff 1y ago
    > ultimately it's just doing a day job that millions of people can do sleep deprived and hungover. Doing for < $10 and under an hour what could be done in a few weeks by $10K+ worth of senior staff time is pretty valuable.
    - 1dom 1y ago
      If it's something a single senior staff member can do, then - personally - I'd consider it not complex, it's relatively trivial: it can be done by literally a single person. I'm pro AI, I'm not saying it's not valuable for trivial things. But that's a distinct discussion to the trivial nature of many LLM examples/demos in relation to genuinely complex computer systems.
- cranium 1y ago
  Instead of "pretty trivial", I'd say it's "well-defined and generally understood". The implicit decisions it had to make were also inconsequential, eg. selection of ASCII chars, color or not, bounds of the domain,... However, it shows that agents are powerful translators / extractors of general knowledge!
- sroussey 1y ago
  Convert react-stockcharts to react v19. I’ve tried Claude Code and Cursor but only ended up with hilariously bad results.
  - simonw 1y ago
    I had great success with o4-mini via ChatGPT for they kind of upgrade, since of can use its search tool to look up what's changed. I used this prompt a few weeks ago: > This code needs to be upgraded to the new recommended JavaScript library from Google. Figure out what that is and then look up enough documentation to port this code to it. https://simonwillison.net/2025/Apr/21/ai-assisted-search/#lazily-porting-code-to-a-new-library-version-via-search https://simonwillison.net/2025/Apr/21/ai-assisted-search/#la...
- j45 1y ago
  Many big problems are made up of small problems.
- afro88 1y ago
  It coming from computer science might be the issue. There's a lot of open source repos out there that have tricky bugs, and todo lists of features that are too complex or time consuming for casual contributors to tackle. Adding significant value to an open source project is a pretty nice demo that won't get called "pretty trivial". Can't be too far off!
- raxxorraxor 1y ago
  The complexity of the problem masqerades the common problem of providing sensible context to your AI of choice to have it doing something constructive in your personal codebase. Or giving it tools to check the truth of one of its assertions. Something a developer does countless times.
- pydry 1y ago
  Really? This paper cut through the same kind of bullshit with puzzles: https://ml-site.cdn-apple.com/papers/the-illusion-of-thinking.pdf https://ml-site.cdn-apple.com/papers/the-illusion-of-thinkin... What do you think is so difficult about doing the same thing with coding problems?
  - simonw 1y ago
    I don't understand the connection between that paper and my comment.
    - pydry 1y ago
      They created an environment to expose LLMs to problems and test their performance which were immune from benchmark hacking using puzzles. Your comment was about how this was unreasonably hard (for coding challenges). Anecdotally Ive seen LLMs do all sorts of amazing shit which was obviously drawn from their training set and fall flat on their faces doing simple coding tasks which are novel enough to not appear in the training set.
      - simonw 1y ago
        That Apple paper mainly demonstrated that "reasoning" LLMs - with no access to additional tools - can't solve problems that deliberately exceed their token context length. I don't think it has much relevance at all to a conversational about how good LLMs are at solving programming problems by running tools in a loop. I keep seeing this idea that LLMs can't handle problems that aren't in their training data and it's frustrating because anyone who has spent significant time working with these systems knows that it obviously isn't true.
        pydry 1y ago
        It demonstrated that there was a hard limit on the complexity of a puzzle that LLMs could solve no matter how many tokens they threw at it (using a form of puzzle construction that it ensured that the LLM couldn't just refer to its training data to solve it).
- dust42 1y ago
  I have one for you: implement gemma 3n multimodel support in llama.cpp
- x0x0 1y ago
  I have one: features I've tried this on in my codebase. Because claude and gemini have both failed pretty badly. So it's pretty stupid to just assume that critics haven't tried. Example feature: send analytics events on app start triggered by notifications. Both Gemini and Claude completely failed to understand the component tree; rewrote hundreds of lines of code in broken ways; and even when prompted with the difficulty (this is happening outside of the component tree), failed to come up with a good solution. And even when deliberately prompted not to, like to simultaneously make cosmetic code changes to other pieces of the files they're touching.
- kayge 1y ago
  The "No True Scotsware" problem? :)
Havoc 1y ago
I suspect personal tools are as close as we're going to get to this mythical demo that satisfies all critics. i.e. here is a list of problems i've solved with just AI. Strikes a balance between simplicity and real world usefulness
- simonw 1y ago
  I tried that with https://tools.simonwillison.net/colophon https://tools.simonwillison.net/colophon - over 100 personal tools, some of which I use on a daily basis.

simonw 1y ago

I think the hardest problem in computer science right now may be coming up with an LLM demo that doesn't get called "pretty trivial".

fragmede 1y ago
I think Cloudflare's oauth library qualifies https://news.ycombinator.com/item?id=44159166 https://news.ycombinator.com/item?id=44159166
- gen6acd60af 1y ago
  This one? >Claude's output was thoroughly reviewed by Cloudflare engineers with careful attention paid to security and compliance with standards. >To emphasize, this is not "vibe coded". Every line was thoroughly reviewed and cross-referenced with relevant RFCs, by security experts with previous experience with those RFCs. Some time later... https://github.com/advisories/GHSA-4pc9-x2fx-p7vj https://github.com/advisories/GHSA-4pc9-x2fx-p7vj / CVE-2025-4143 >The OAuth implementation in workers-oauth-provider that is part of MCP framework https://github.com/cloudflare/workers-mcp https://github.com/cloudflare/workers-mcp, did not correctly validate that redirect_uri was on the allowed list of redirect URIs for the given client registration.
  - kentonv 1y ago
    Sorry, my code has bugs sometimes.
skydhash 1y ago
Because they are trivial in a way that you can go on GitHub and copy one of those while not pretending LLM isn't a mashup of the internet. What people agree on being non-trivial is working on a real project. There's a lot of opensource projects that could benefit from a useful code contribution. But they only got slop thrown at them.
- simonw 1y ago
  [flagged]
  - skydhash 1y ago
    I took the time to investigate the work being done there (all those years learning assembly and computer architecture come in handy), and it confirms (to me) that the key aspect of using LLM is pattern matching. Meaning you know that there's a solution out there (in this case, anything involving multiplying/dividing by a power of 2 can use such trick) and framing your problem (intentionally or not) and you'll get a derived text that will contain a possible solution. But there's nothing truly novel in the result. The key aspect is being similar enough to something that's already in the training data so that the LLM can extrapolate the rest. The hint can be quite useful and sometimes you have something that shorten the implementation time, but you have to at least have some basic understanding of the domain in order to recognize the signs. The issue is that the result is always tainted by your prompt. The signs may be there because of your prompt and not because there's some kind of data that need s to be explored further. And sometimes it's a bad fit, similar but different (what you want and what you get). So for the few domain that's valuable to me, I prefer to construct my own mental database that can lead me to concrete artifacts (books, articles, blog posts,...) that exists outside the influence of my query. ADDENDUM I can use LLMs with great results and I've done so. But it's more rewarding (and more useful to me) to actually think through the problem and learning from references. Instead of getting a perfect (or wobbly or the wrong category) circle that fits my query, I go to find a strange polygon formed (by me) from other strange polygon. Then because I know I need a circle, I only need to find its center and its radius. It's slower, but the next time I need another circle (or a square) from the same polygon, it's going to be faster and faster.
th0ma5 1y ago
Maybe you should try something other than demos? Have you tried creating a reliable system?
- deleted 1y ago
  [deleted]
jkhdigital 1y ago
No the hardest problem is teaching CS undergrads. I just started this year (no background in academia, just 75% of a PhD and well-rounded life experience) and I’ve basically torn up the entire curriculum they handed to me and started vibe-teaching.
1dom 1y ago
I'm very pro LLM and AI. But I completely agree with the comment about how many pieces praising LLMs are doing so with trivial examples. Trivial might not be the right word, but I can't think of a better one that doesn't have a negative connotation, but this shouldn't be negative. Your examples are good and useful, and capture a bunch of tasks a software engineer would do. I'd say your mandelbrot debug and the LLVM patch are both "trivial" in the same sense: they're discrete, well defined, clear-success-criteria-tasks that could be assigned to any mid/senior software engineer in a relevant domain and they could chip through it in a few weeks. Don't get me wrong, that's an insane power and capability of LLMs, I agree. But ultimately it's just doing a day job that millions of people can do sleep deprived and hungover. Non-trivial examples are things that would take a team of different specialist skillsets months to create. One obvious potential reason why there's few non-trivial AI examples is because non-trivial AI examples require non-trivial amount of time to be able to generate and verify. A non-trivial example isn't an example you can look at the output and say "yup, AI's done well here". It requires someone spends time going into what's been produced, assessing it, essentially redesigning it as a human to figure out all the complexity of a modern non-trivial system to confirm the AI actually did all that stuff correctly. An in depth audit of a complex software system can take months or even years and is a thorough and tedious task for a human, and the Venn diagrams of humans who are thinking "I want to spend more time doing thorough, tedious code tasks" and "I want to mess around with AI coding" is 2 separate circles.
- sokoloff 1y ago
  > ultimately it's just doing a day job that millions of people can do sleep deprived and hungover. Doing for < $10 and under an hour what could be done in a few weeks by $10K+ worth of senior staff time is pretty valuable.
  - 1dom 1y ago
    If it's something a single senior staff member can do, then - personally - I'd consider it not complex, it's relatively trivial: it can be done by literally a single person. I'm pro AI, I'm not saying it's not valuable for trivial things. But that's a distinct discussion to the trivial nature of many LLM examples/demos in relation to genuinely complex computer systems.
cranium 1y ago
Instead of "pretty trivial", I'd say it's "well-defined and generally understood". The implicit decisions it had to make were also inconsequential, eg. selection of ASCII chars, color or not, bounds of the domain,... However, it shows that agents are powerful translators / extractors of general knowledge!
sroussey 1y ago
Convert react-stockcharts to react v19. I’ve tried Claude Code and Cursor but only ended up with hilariously bad results.
- simonw 1y ago
  I had great success with o4-mini via ChatGPT for they kind of upgrade, since of can use its search tool to look up what's changed. I used this prompt a few weeks ago: > This code needs to be upgraded to the new recommended JavaScript library from Google. Figure out what that is and then look up enough documentation to port this code to it. https://simonwillison.net/2025/Apr/21/ai-assisted-search/#lazily-porting-code-to-a-new-library-version-via-search https://simonwillison.net/2025/Apr/21/ai-assisted-search/#la...
j45 1y ago
Many big problems are made up of small problems.
afro88 1y ago
It coming from computer science might be the issue. There's a lot of open source repos out there that have tricky bugs, and todo lists of features that are too complex or time consuming for casual contributors to tackle. Adding significant value to an open source project is a pretty nice demo that won't get called "pretty trivial". Can't be too far off!
raxxorraxor 1y ago
The complexity of the problem masqerades the common problem of providing sensible context to your AI of choice to have it doing something constructive in your personal codebase. Or giving it tools to check the truth of one of its assertions. Something a developer does countless times.
pydry 1y ago
Really? This paper cut through the same kind of bullshit with puzzles: https://ml-site.cdn-apple.com/papers/the-illusion-of-thinking.pdf https://ml-site.cdn-apple.com/papers/the-illusion-of-thinkin... What do you think is so difficult about doing the same thing with coding problems?
- simonw 1y ago
  I don't understand the connection between that paper and my comment.
  - pydry 1y ago
    They created an environment to expose LLMs to problems and test their performance which were immune from benchmark hacking using puzzles. Your comment was about how this was unreasonably hard (for coding challenges). Anecdotally Ive seen LLMs do all sorts of amazing shit which was obviously drawn from their training set and fall flat on their faces doing simple coding tasks which are novel enough to not appear in the training set.
    - simonw 1y ago
      That Apple paper mainly demonstrated that "reasoning" LLMs - with no access to additional tools - can't solve problems that deliberately exceed their token context length. I don't think it has much relevance at all to a conversational about how good LLMs are at solving programming problems by running tools in a loop. I keep seeing this idea that LLMs can't handle problems that aren't in their training data and it's frustrating because anyone who has spent significant time working with these systems knows that it obviously isn't true.
      - pydry 1y ago
        It demonstrated that there was a hard limit on the complexity of a puzzle that LLMs could solve no matter how many tokens they threw at it (using a form of puzzle construction that it ensured that the LLM couldn't just refer to its training data to solve it).
dust42 1y ago
I have one for you: implement gemma 3n multimodel support in llama.cpp
x0x0 1y ago
I have one: features I've tried this on in my codebase. Because claude and gemini have both failed pretty badly. So it's pretty stupid to just assume that critics haven't tried. Example feature: send analytics events on app start triggered by notifications. Both Gemini and Claude completely failed to understand the component tree; rewrote hundreds of lines of code in broken ways; and even when prompted with the difficulty (this is happening outside of the component tree), failed to come up with a good solution. And even when deliberately prompted not to, like to simultaneously make cosmetic code changes to other pieces of the files they're touching.
kayge 1y ago
The "No True Scotsware" problem? :)

Havoc 1y ago

I suspect personal tools are as close as we're going to get to this mythical demo that satisfies all critics. i.e. here is a list of problems i've solved with just AI. Strikes a balance between simplicity and real world usefulness

simonw 1y ago
I tried that with https://tools.simonwillison.net/colophon https://tools.simonwillison.net/colophon - over 100 personal tools, some of which I use on a daily basis.