3 ms·

It's the attention mechanism at work, along with a fair bit of Internet one-up-manship. The LLM has ingested all of the text on the Internet, as well as Github

It's the attention mechanism at work, along with a fair bit of Internet one-up-manship. The LLM has ingested all of the text on the Internet, as well as Github code repositories, pull requests, StackOverflow posts, code reviews, mailing lists, etc. In a number of those content sources, there will be people saying "Actually, if you go into the details of..." or "If you look at the intricacies of the problem" or "If you understood the problem deeply" followed by a very deep, expert-level explication of exactly what you should've done differently. You want the model to use the code in the correction, not the one in the original StackOverflow question. Same reason that "Pretend you are an MIT professor" or "You are a leading Python expert" or similar works in prompts. It tells the model to pay attention to the part of the corpus that has those terms, weighting them more highly than all the other programming samples that it's run across.

deleted 8mo ago
[deleted]
r0b05 8mo ago
This is such a good explanation. Thanks
xscott 8mo ago
Of course I can't be certain, but I think the "mixture of experts" design plays into it too. Metaphorically, there's a mid-level manager who looks at your prompt and tries to decide which experts it should be sent to. If he thinks you won't notice, he saves money by sending it to the undergraduate intern. Just a theory.
- victorbjorklund 8mo ago
  Notice that MOE isn’t different experts for different types of problems. It’s per token and not really connect to problem type. So if you send a python code then the first one in function can be one expert, second another expert and so on.
  - dotancohen 8mo ago
    Can you back this up with documentation? I don't believe that this is the case.
    - pixelmelt 8mo ago
      Check out Unsloths REAP models, you can outright delete a few of the lesser used experts without the model going braindead since they all can handle each token but some are better posed to do so.
    - l33tman 8mo ago
      The router that routes the tokens between the "experts" is part of the training itself as well. The name MoE is really not a good acronym as it makes people believe it's on a more coarse level and that each of the experts somehow is trained by different corpus etc. But what do I know, there are new archs every week and someone might have done a MoE differently.
    - smallerize 8mo ago
      It's not only per token, but also each layer has its own router and can choose different experts. https://huggingface.co/blog/moe#what-is-a-mixture-of-experts-moe https://huggingface.co/blog/moe#what-is-a-mixture-of-experts...
manmal 8mo ago
I don’t think this is a result of the base training data („the internet“). It’s a post training behavior, created during reinforcement learning. Codex has a totally different behavior in that regard. Codex reads per default a lot of potentially relevant files before it goes and writes files. Maybe you remember that, without reinforcement learning, the models of 2019 just completed the sentences you gave them. There were no tool calls like reading files. Tool calling behavior is company specific and highly tuned to their harnesses. How often they call a tool, is not part of the base training data.
- spagettnet 8mo ago
  Modern LLM are certainly fine tuned on data that includes examples of tool use, mostly the tools built into their respective harnesses, but also external/mock tools so they dont overfit on only using the toolset they expect to see in their harnesses.
  - manmal 8mo ago
    IDK the current state, but I remember that, last year, the open source coding harnesses needed to provide exactly the tools that the LLM expected, or the error rate went through the roof. Some, like grok and gemini, only recently managed to make tool calls somewhat reliable.
hbarka 8mo ago
>> Same reason that "Pretend you are an MIT professor" or "You are a leading Python expert" or similar works in prompts. This pretend-you-are-a-[persona] is cargo cult prompting at this point. The persona framing is just decoration. A brief purpose statement describing what the skill [skill.md] does is more honest and just as effective.
- rescbr 8mo ago
  I think it does more harm than good on recent models. The LLM has to override its system prompt to role-play, wasting context and computing cycles instead of working on the task.
- LEDThereBeLight 8mo ago
  It’s not cargo culting, it does make a difference and there are papers on arxiv discussing it. The trouble is that it’s hard to tell whether it’ll help or hurt - telling it to act as an expert in one field may improve your result, or may make it lose some of the other perspectives it has which might be more important for solving the problem.
  - SergeAx 8mo ago
    Papers I read on arxiv mostly say that persona prompting changes tone of voice and attitude, but not the quality of the answer, at least statistically. For example: https://arxiv.org/abs/2512.05858 https://arxiv.org/abs/2512.05858
dakolli 8mo ago
You will never convince me that this isn't confirmation bias, or the equivalent of a slot machine player thinking the order in which they push buttons impacts the output, or some other gambler-esque superstition. These tools are literally designed to make people behave like gamblers. And its working, except the house in this case takes the money you give them and lights it on fire.
- nubg 8mo ago
  Your ignorance is my opportunity. May I ask which markets you are developing for?
  - dakolli 8mo ago
    "The equivalent of saying, which slot machine were you sitting at It'll make me money"
    - nubg 8mo ago
      [flagged]
aakresearch 8mo ago
This is a very useful take, thank you. Really helped me to adjust my mental model without "antropomorphising" the machinery. Upvoted. If I may, I would re-phrase/expand the last sentence of yours in a way that makes it even more useful for me, personally. Maybe it could help other people too. I think it is fair to say that in presence of hints like "Pretend you are X" or "Take a deeper look" the inference mechanism (driven by it's training weights, and now influenced by those hints via "attention math") is not "satisfied" until it pulls more relevant tokens into "working context" ("more" and "relevant" being modulated by the particular hint).

deleted 8mo ago

[deleted]

r0b05 8mo ago

This is such a good explanation. Thanks

xscott 8mo ago

Of course I can't be certain, but I think the "mixture of experts" design plays into it too. Metaphorically, there's a mid-level manager who looks at your prompt and tries to decide which experts it should be sent to. If he thinks you won't notice, he saves money by sending it to the undergraduate intern. Just a theory.

victorbjorklund 8mo ago
Notice that MOE isn’t different experts for different types of problems. It’s per token and not really connect to problem type. So if you send a python code then the first one in function can be one expert, second another expert and so on.
- dotancohen 8mo ago
  Can you back this up with documentation? I don't believe that this is the case.
  - pixelmelt 8mo ago
    Check out Unsloths REAP models, you can outright delete a few of the lesser used experts without the model going braindead since they all can handle each token but some are better posed to do so.
  - l33tman 8mo ago
    The router that routes the tokens between the "experts" is part of the training itself as well. The name MoE is really not a good acronym as it makes people believe it's on a more coarse level and that each of the experts somehow is trained by different corpus etc. But what do I know, there are new archs every week and someone might have done a MoE differently.
  - smallerize 8mo ago
    It's not only per token, but also each layer has its own router and can choose different experts. https://huggingface.co/blog/moe#what-is-a-mixture-of-experts-moe https://huggingface.co/blog/moe#what-is-a-mixture-of-experts...

manmal 8mo ago

I don’t think this is a result of the base training data („the internet“). It’s a post training behavior, created during reinforcement learning. Codex has a totally different behavior in that regard. Codex reads per default a lot of potentially relevant files before it goes and writes files. Maybe you remember that, without reinforcement learning, the models of 2019 just completed the sentences you gave them. There were no tool calls like reading files. Tool calling behavior is company specific and highly tuned to their harnesses. How often they call a tool, is not part of the base training data.

spagettnet 8mo ago
Modern LLM are certainly fine tuned on data that includes examples of tool use, mostly the tools built into their respective harnesses, but also external/mock tools so they dont overfit on only using the toolset they expect to see in their harnesses.
- manmal 8mo ago
  IDK the current state, but I remember that, last year, the open source coding harnesses needed to provide exactly the tools that the LLM expected, or the error rate went through the roof. Some, like grok and gemini, only recently managed to make tool calls somewhat reliable.

hbarka 8mo ago

>> Same reason that "Pretend you are an MIT professor" or "You are a leading Python expert" or similar works in prompts. This pretend-you-are-a-[persona] is cargo cult prompting at this point. The persona framing is just decoration. A brief purpose statement describing what the skill [skill.md] does is more honest and just as effective.

rescbr 8mo ago
I think it does more harm than good on recent models. The LLM has to override its system prompt to role-play, wasting context and computing cycles instead of working on the task.
LEDThereBeLight 8mo ago
It’s not cargo culting, it does make a difference and there are papers on arxiv discussing it. The trouble is that it’s hard to tell whether it’ll help or hurt - telling it to act as an expert in one field may improve your result, or may make it lose some of the other perspectives it has which might be more important for solving the problem.
- SergeAx 8mo ago
  Papers I read on arxiv mostly say that persona prompting changes tone of voice and attitude, but not the quality of the answer, at least statistically. For example: https://arxiv.org/abs/2512.05858 https://arxiv.org/abs/2512.05858

dakolli 8mo ago

You will never convince me that this isn't confirmation bias, or the equivalent of a slot machine player thinking the order in which they push buttons impacts the output, or some other gambler-esque superstition. These tools are literally designed to make people behave like gamblers. And its working, except the house in this case takes the money you give them and lights it on fire.

nubg 8mo ago
Your ignorance is my opportunity. May I ask which markets you are developing for?
- dakolli 8mo ago
  "The equivalent of saying, which slot machine were you sitting at It'll make me money"
  - nubg 8mo ago
    [flagged]

aakresearch 8mo ago

This is a very useful take, thank you. Really helped me to adjust my mental model without "antropomorphising" the machinery. Upvoted. If I may, I would re-phrase/expand the last sentence of yours in a way that makes it even more useful for me, personally. Maybe it could help other people too. I think it is fair to say that in presence of hints like "Pretend you are X" or "Take a deeper look" the inference mechanism (driven by it's training weights, and now influenced by those hints via "attention math") is not "satisfied" until it pulls more relevant tokens into "working context" ("more" and "relevant" being modulated by the particular hint).