9 ms·

Gemini 2.5 Pro reasons about task feasibility

jrvarela56 2y ago
Would be cool if the LLM can break up the request into sub-requests processable by LLMs. Current talk about agents mention some sort of router/orchestrator that delegates to other agents. But these can be another llm, another agent, another router itself or a simple tool call, etc - all function calls that wrap other llm-enabled sub components. My feeling is that we have the pieces to build AGI. Like humans, we don't need a 400IQ person to solve all problems ('AGI'). What we have is coordination problems and in LLM land it's 'the glue' that's missing. Hopeful it's a matter of patterns/best-practices emerging.
- intellectronica 2y ago
  I bet it can do that if hooked up to an agent system. Rate limits are still very restrictive now in the free API, but as soon as they make it available for more frequent use we'll find out.
- intellectronica 2y ago
  And yes, I share the view / feeling that we basically got the AGI building blocks. Models will continue improving, but we can already get most of what we need just by orchestrating the latest generation of SOTA models. Crazy time to be alive!
- NitpickLawyer 2y ago
  > But these can be another llm Yes! I share the feeling that once LLMs get good enough at some abstraction level, you can always put another "level" on top that should abstract what already works into bite sized pieces. Hassabis also mentions this in a recent podcast, different levels of abstraction. We'll probably see some tooling in this space shortly, to coordinate between the different levels. And then RL it and watch it demolish planning tasks benchmarks. We might very well already be at the point where every level is achievable, we just have to glue them together.
- simonw 2y ago
  "Would be cool if the LLM can break up the request into sub-requests processable by LLMs." It almost certainly can. Try asking Gemini 2.5 Pro to do that and see what happens.
  - jrvarela56 2y ago
    What parts of the stack or what patterns do you feel are missing? Where does your gut tell you is the 80/20?
  - viraptor 2y ago
    The llm itself doesn't even need to do that. The actual system / front end that people interact with can wrap that step. Plandex does it for example and has been doing it for longer than the integrated reasoning models existed. I mean, it's nice when the models can integrate the step-by-step internally... but I feel people have been missing out on the complex interactions by expecting it all in one adhoc prompt.
    - jrvarela56 2y ago
      I think the feeling is that for this to really be AGI, it has to take in a single prompt and then delegate behind the scenes to an enormous tree of sub-agents if needed. One app that comes to mind is Google's Conversational agents. The routing is just done by referencing another agent in the instructions, no need to explicitly link beyond the prompt.
- me_me_me 2y ago
  [dead]
- dimitri-vs 2y ago
  Disagree that we have all pieces for AGI. Memory for one, not only do models need to be able to have long term, short term memory but they also need to be able to selectively forget. Hallucinations are still a big problem, you can easily (unintentionally) put the models in situations where they make up facts. Context limits - comprehension limits are still effectively 8-10k even though the token limits have been raised to infinity.
  - GaggiX 2y ago
    Why "selectively forget" should be a piece for AGI?
    - dnadler 2y ago
      I think maybe this refers to unlearning wrong information?
      - skydhash 2y ago
        Also abstracting. No need to remember every milliseconds in its lifetime and consult them in every query.
      - GaggiX 2y ago
        I can remember for example when I was wrong and how and still responding correctly, I don't have to forget my wrong answer to respond with the correct one.
    - dimitri-vs 2y ago
      I guess we should start with the fact that models currently have no ability to remember at all. You either fine-tune which is a very lossy process that degrades generality or you do in-context learning/RAG. Forgetting in its current form would be eliminating obsolete context, not forgetting would be using 1 million input tokens to answer "what is 2+2?". In any case, any external mechanic to selectively manage context would be far too limiting for AGI.
  - mortarion 2y ago
- eightysixfour 2y ago
  Tools like Claude Code already do this.
coolgoose 2y ago
Next step into LLM evolution is teaching them to procrastinate
- johnisgood 2y ago
  It already refuses to answer to some questions. I wanted to know the mechanism of action of a medication and GPT did not want to answer me, I had to work around it by telling it that I am doing research or it is for university. Like come on, I can't even ask the mechanism of action of medications? This is just one example, there are lots of censorship going on around the models. Which ones are less censored? Are free & open source ones even censored at all or would they answer? I imagine they would. GPT and Claude may not answer so the company can save their asses, so local ones should probably work.
- pjmlp 2y ago
  Yesterday on a German radio I heard about a study on how they can get traumatized if trained on bad human behaviour content, and provide similar output like humans in such situations, and how their output improves if interacted in a way that kind of shows compassion behaviours. Maybe being an LLM psychologist is a job with future.
wiz21c 2y ago
FTA: > I have never seen an LLM do this Interestingly, many of the program we use provide a finite set of functionality that we can discover over time. But LLM's are different: you can't explore them because the input space is too big. Therefore, they can surprise us for a long time. That's cool!
- intellectronica 2y ago
  Exactly! We don't know everything these machines are good for. Even with older and more established models new things that can be done are discovered many times every day.
- patapong 2y ago
  Yes I agree! The more I use LLMs, the more use-cases for them I find. Even if the models stop advancing tomorrow (which is unlikely), I think we could spend many years just exploring what the current models are capable of.
retrofuturism 2y ago
That's interesting, but I wonder if it's _just_ the system prompt dictating that a request that would likely consume too many resources and likely fail should be rejected with such an answer.
simonw 2y ago
I've been having some very impressive results from Gemini 2.5 Pro for complex coding tasks in the few hours I've been experimenting with it so far. I added a section about that to my review last night describing two of the larger examples: https://simonwillison.net/2025/Mar/25/gemini/#update-it-s-very-good-at-code https://simonwillison.net/2025/Mar/25/gemini/#update-it-s-ve... (It's always risky saying anything like this on a forum like Hacker News because it's inevitable someone will find a way to argue that the examples are trivial/unrealistic/show I don't know what I'm doing/clearly just regurgitated from StackOverflow/etc, but I'll take the risk anyway.)
- intellectronica 2y ago
  So far everything I tried indicates that it is superb at coding. Maybe the best yet (though I understand that some of the published benchmarks dispute that). Here it one-shotted a fully functional LISP interpreter for me: https://everything.intellectronica.net/p/the-little-lisper https://everything.intellectronica.net/p/the-little-lisper
  - arresin 2y ago
    hmm but how many examples of that exist on the internet practically verbatim?
    - intellectronica 2y ago
      Not sure that's relevant. Obviously an LLM has to learn from something, but it's not a database. I could also program this myself and I don't think that it's an argument against my coding abilities that I have read the source code to many existing interpreters. I can only do it because I not only read but also understood and internalised.
      - jgalt212 2y ago
        But it's almost a database. See the full glass of wine conundrum.
trash_cat 2y ago
Would be interesting to see the input prompts.
- intellectronica 2y ago
  """ Create a complete reproduction of the ReBirth virtual synth. Create it as a single HTML page with javascript. Use canvas and/or react for the UI and tone.js or something similar for the audio. It should be fully working and playable. Output it as a single HTML page. See screenshot for the UI and description below for how the virtual synth system works. """ (this is followed by long spec of the RB-338 which I also generated and is too long to include here, and a screenshot).
  - gverrilla 2y ago
    This is essential to your text and it's significance - should be there imo.
vladmdgolam 2y ago
I’ve encountered a similar when prompging o1-pro to make palindromes with some words and it actually answered that it’s impossible with some of them because they are gibberish when reversed and then made an example
tkgally 2y ago
Very cool. Inspired by the post, I gave the following prompt to Gemini 2.5 Pro with the temperature set to 0.35: “Create a music synthesizer that I can run in my browser. Give it original functionality, with buttons and other controls that create pleasing but unexpected sounds, melodies, rhythmic patterns, etc. as well as various sliders and dials that can be used to modify the sounds and patterns.” Here is the result: https://www.gally.net/temp/202540326geminisynthesizer/index.html https://www.gally.net/temp/202540326geminisynthesizer/index.... Here’s an excerpt from the reasoning trace: > Brainstorm "Original" and "Unexpected" Functionality: > Standard Synths: Oscillators (sine, square, saw, triangle), filters (lowpass, highpass), envelopes (ADSR), LFOs. These are common. How to make them unexpected? > Generative Aspects: Instead of just playing notes the user clicks, what if the synth generates patterns? > Melody Generation: Use algorithms like random walks, cellular automata, or simple probability rules. > Rhythm Generation: Euclidean rhythms, probabilistic triggers, evolving patterns. ... > Interface Concept: How can the controls themselves be part of the "unexpected" experience? Buttons that trigger complex events, sliders with non-linear responses, dials that control abstract concepts.
- olalonde 2y ago
  Is there something like "Claude Code" for Gemini? Or do you have to manually copy/paste the code in files?
  - notimetorelax 2y ago
    You can use Gemini from VScode. (Well at least copilot can call it)
  - euazOn 2y ago
    Check out Aider [0] or Anon Kode [1] (clone of Claude Code). New models are why I try to build all my tools and infra to be model-independent. On that note, I also prefer to be provider-independent, using OpenRouter [2] or T3 Chat [3] and the like. [0] https://aider.chat/ https://aider.chat/ [1] https://github.com/dnakov/anon-kode https://github.com/dnakov/anon-kode [2] https://openrouter.ai/ https://openrouter.ai/ [3] https://t3.chat/ https://t3.chat/
ofirtwo 2y ago
I'm curious on how the model's going to face intellectual tasks he can't resolve by referring back to the user. Today most LLM's will give multiple answers to "what's the meaning of life?" and immediately move the wand back to the user. It could be interesting if they'll hang with the question more, dive deeper into contradictions and tell, eventually, they don't know.
deleted 2y ago
[deleted]
menzoic 2y ago
"During its thinking session it reached the conclusion that this task is not feasible in one shot. It then stopped and explained that to me." I've seen this happen with GPT-4 with zero shot prompts. Similar to the author "negotiating" allowed it to continue with an iterative approach.
cadamsdotcom 2y ago
It’s a new type of refusal. The model is unlikely to know its own limits. Hopefully these refusals are amenable to prompt engineering: “even if the task seems infeasible, try anyway.” And hopefully next-gen models are trained to have more faith in themselves :)

Would be cool if the LLM can break up the request into sub-requests processable by LLMs. Current talk about agents mention some sort of router/orchestrator that delegates to other agents. But these can be another llm, another agent, another router itself or a simple tool call, etc - all function calls that wrap other llm-enabled sub components. My feeling is that we have the pieces to build AGI. Like humans, we don't need a 400IQ person to solve all problems ('AGI'). What we have is coordination problems and in LLM land it's 'the glue' that's missing. Hopeful it's a matter of patterns/best-practices emerging.

intellectronica 2y ago
I bet it can do that if hooked up to an agent system. Rate limits are still very restrictive now in the free API, but as soon as they make it available for more frequent use we'll find out.
intellectronica 2y ago
And yes, I share the view / feeling that we basically got the AGI building blocks. Models will continue improving, but we can already get most of what we need just by orchestrating the latest generation of SOTA models. Crazy time to be alive!
NitpickLawyer 2y ago
> But these can be another llm Yes! I share the feeling that once LLMs get good enough at some abstraction level, you can always put another "level" on top that should abstract what already works into bite sized pieces. Hassabis also mentions this in a recent podcast, different levels of abstraction. We'll probably see some tooling in this space shortly, to coordinate between the different levels. And then RL it and watch it demolish planning tasks benchmarks. We might very well already be at the point where every level is achievable, we just have to glue them together.
simonw 2y ago
"Would be cool if the LLM can break up the request into sub-requests processable by LLMs." It almost certainly can. Try asking Gemini 2.5 Pro to do that and see what happens.
- jrvarela56 2y ago
  What parts of the stack or what patterns do you feel are missing? Where does your gut tell you is the 80/20?
- viraptor 2y ago
  The llm itself doesn't even need to do that. The actual system / front end that people interact with can wrap that step. Plandex does it for example and has been doing it for longer than the integrated reasoning models existed. I mean, it's nice when the models can integrate the step-by-step internally... but I feel people have been missing out on the complex interactions by expecting it all in one adhoc prompt.
  - jrvarela56 2y ago
    I think the feeling is that for this to really be AGI, it has to take in a single prompt and then delegate behind the scenes to an enormous tree of sub-agents if needed. One app that comes to mind is Google's Conversational agents. The routing is just done by referencing another agent in the instructions, no need to explicitly link beyond the prompt.
me_me_me 2y ago
[dead]
dimitri-vs 2y ago
Disagree that we have all pieces for AGI. Memory for one, not only do models need to be able to have long term, short term memory but they also need to be able to selectively forget. Hallucinations are still a big problem, you can easily (unintentionally) put the models in situations where they make up facts. Context limits - comprehension limits are still effectively 8-10k even though the token limits have been raised to infinity.
- GaggiX 2y ago
  Why "selectively forget" should be a piece for AGI?
  - dnadler 2y ago
    I think maybe this refers to unlearning wrong information?
    - skydhash 2y ago
      Also abstracting. No need to remember every milliseconds in its lifetime and consult them in every query.
    - GaggiX 2y ago
      I can remember for example when I was wrong and how and still responding correctly, I don't have to forget my wrong answer to respond with the correct one.
  - dimitri-vs 2y ago
    I guess we should start with the fact that models currently have no ability to remember at all. You either fine-tune which is a very lossy process that degrades generality or you do in-context learning/RAG. Forgetting in its current form would be eliminating obsolete context, not forgetting would be using 1 million input tokens to answer "what is 2+2?". In any case, any external mechanic to selectively manage context would be far too limiting for AGI.
- mortarion 2y ago
eightysixfour 2y ago
Tools like Claude Code already do this.

coolgoose 2y ago

Next step into LLM evolution is teaching them to procrastinate

johnisgood 2y ago
It already refuses to answer to some questions. I wanted to know the mechanism of action of a medication and GPT did not want to answer me, I had to work around it by telling it that I am doing research or it is for university. Like come on, I can't even ask the mechanism of action of medications? This is just one example, there are lots of censorship going on around the models. Which ones are less censored? Are free & open source ones even censored at all or would they answer? I imagine they would. GPT and Claude may not answer so the company can save their asses, so local ones should probably work.
pjmlp 2y ago
Yesterday on a German radio I heard about a study on how they can get traumatized if trained on bad human behaviour content, and provide similar output like humans in such situations, and how their output improves if interacted in a way that kind of shows compassion behaviours. Maybe being an LLM psychologist is a job with future.

wiz21c 2y ago

FTA: > I have never seen an LLM do this Interestingly, many of the program we use provide a finite set of functionality that we can discover over time. But LLM's are different: you can't explore them because the input space is too big. Therefore, they can surprise us for a long time. That's cool!

intellectronica 2y ago
Exactly! We don't know everything these machines are good for. Even with older and more established models new things that can be done are discovered many times every day.
patapong 2y ago
Yes I agree! The more I use LLMs, the more use-cases for them I find. Even if the models stop advancing tomorrow (which is unlikely), I think we could spend many years just exploring what the current models are capable of.

retrofuturism 2y ago

That's interesting, but I wonder if it's _just_ the system prompt dictating that a request that would likely consume too many resources and likely fail should be rejected with such an answer.

simonw 2y ago

I've been having some very impressive results from Gemini 2.5 Pro for complex coding tasks in the few hours I've been experimenting with it so far. I added a section about that to my review last night describing two of the larger examples: https://simonwillison.net/2025/Mar/25/gemini/#update-it-s-very-good-at-code https://simonwillison.net/2025/Mar/25/gemini/#update-it-s-ve... (It's always risky saying anything like this on a forum like Hacker News because it's inevitable someone will find a way to argue that the examples are trivial/unrealistic/show I don't know what I'm doing/clearly just regurgitated from StackOverflow/etc, but I'll take the risk anyway.)

intellectronica 2y ago
So far everything I tried indicates that it is superb at coding. Maybe the best yet (though I understand that some of the published benchmarks dispute that). Here it one-shotted a fully functional LISP interpreter for me: https://everything.intellectronica.net/p/the-little-lisper https://everything.intellectronica.net/p/the-little-lisper
- arresin 2y ago
  hmm but how many examples of that exist on the internet practically verbatim?
  - intellectronica 2y ago
    Not sure that's relevant. Obviously an LLM has to learn from something, but it's not a database. I could also program this myself and I don't think that it's an argument against my coding abilities that I have read the source code to many existing interpreters. I can only do it because I not only read but also understood and internalised.
    - jgalt212 2y ago
      But it's almost a database. See the full glass of wine conundrum.

trash_cat 2y ago

Would be interesting to see the input prompts.

intellectronica 2y ago
""" Create a complete reproduction of the ReBirth virtual synth. Create it as a single HTML page with javascript. Use canvas and/or react for the UI and tone.js or something similar for the audio. It should be fully working and playable. Output it as a single HTML page. See screenshot for the UI and description below for how the virtual synth system works. """ (this is followed by long spec of the RB-338 which I also generated and is too long to include here, and a screenshot).
- gverrilla 2y ago
  This is essential to your text and it's significance - should be there imo.

vladmdgolam 2y ago

I’ve encountered a similar when prompging o1-pro to make palindromes with some words and it actually answered that it’s impossible with some of them because they are gibberish when reversed and then made an example

tkgally 2y ago

Very cool. Inspired by the post, I gave the following prompt to Gemini 2.5 Pro with the temperature set to 0.35: “Create a music synthesizer that I can run in my browser. Give it original functionality, with buttons and other controls that create pleasing but unexpected sounds, melodies, rhythmic patterns, etc. as well as various sliders and dials that can be used to modify the sounds and patterns.” Here is the result: https://www.gally.net/temp/202540326geminisynthesizer/index.html https://www.gally.net/temp/202540326geminisynthesizer/index.... Here’s an excerpt from the reasoning trace: > Brainstorm "Original" and "Unexpected" Functionality: > Standard Synths: Oscillators (sine, square, saw, triangle), filters (lowpass, highpass), envelopes (ADSR), LFOs. These are common. How to make them unexpected? > Generative Aspects: Instead of just playing notes the user clicks, what if the synth generates patterns? > Melody Generation: Use algorithms like random walks, cellular automata, or simple probability rules. > Rhythm Generation: Euclidean rhythms, probabilistic triggers, evolving patterns. ... > Interface Concept: How can the controls themselves be part of the "unexpected" experience? Buttons that trigger complex events, sliders with non-linear responses, dials that control abstract concepts.

olalonde 2y ago
Is there something like "Claude Code" for Gemini? Or do you have to manually copy/paste the code in files?
- notimetorelax 2y ago
  You can use Gemini from VScode. (Well at least copilot can call it)
- euazOn 2y ago
  Check out Aider [0] or Anon Kode [1] (clone of Claude Code). New models are why I try to build all my tools and infra to be model-independent. On that note, I also prefer to be provider-independent, using OpenRouter [2] or T3 Chat [3] and the like. [0] https://aider.chat/ https://aider.chat/ [1] https://github.com/dnakov/anon-kode https://github.com/dnakov/anon-kode [2] https://openrouter.ai/ https://openrouter.ai/ [3] https://t3.chat/ https://t3.chat/

ofirtwo 2y ago

I'm curious on how the model's going to face intellectual tasks he can't resolve by referring back to the user. Today most LLM's will give multiple answers to "what's the meaning of life?" and immediately move the wand back to the user. It could be interesting if they'll hang with the question more, dive deeper into contradictions and tell, eventually, they don't know.

deleted 2y ago

[deleted]

menzoic 2y ago

"During its thinking session it reached the conclusion that this task is not feasible in one shot. It then stopped and explained that to me." I've seen this happen with GPT-4 with zero shot prompts. Similar to the author "negotiating" allowed it to continue with an iterative approach.

cadamsdotcom 2y ago

It’s a new type of refusal. The model is unlikely to know its own limits. Hopefully these refusals are amenable to prompt engineering: “even if the task seems infeasible, try anyway.” And hopefully next-gen models are trained to have more faith in themselves :)