3 ms·

Using the word “Mentoring” is anthropomorphic and subconsciously makes you think it will learn. It does not, and it is for the human brain a formidable task to

Using the word “Mentoring” is anthropomorphic and subconsciously makes you think it will learn. It does not, and it is for the human brain a formidable task to remember that something as smart as an LLM does not learn. I keep catching myself making the same mistake. It’s also because it is so annoying to have to manage the memory of the LLM with custom prompts/instructions manually. I have not yet played with the long term memory feature, but I fear it will be even less reliable than prompts, simply because in one year or two years so much will have changed again that this “memory” will have to be redone multiple times by then.

timschmidt 5mo ago
They can form new associations between concepts via their input prompts and thinking text. That is a form of learning. Just not very durable. I liken it to https://en.wikipedia.org/wiki/Anterograde_amnesia https://en.wikipedia.org/wiki/Anterograde_amnesia
- illiac786 5mo ago
  yeah, I should have been more specific: I meant the type of learning that mentoring fosters, the long term learning.
  - timschmidt 5mo ago
    I hear you. I think we are already seeing some middle ground with agentic systems using RAG, skills.md files, etc. It's a sort of disassociated card catalog memory. An engineer's notebook. Not the integrated, correlated, pre-processed set of relationships in the model. How to go backward from the notebook -> model cheaply without tanking performance is definitely one of those billion dollar questions.
  - pickleRick243 5mo ago
    a little glib, but there is in fact long term learning. It's just that you are not the one mentoring- the models go to intensive OpenAI/Anthropic/Google school for a quarter or half a year and come back (hopefully) improved. You just hope they're getting a good education. Certainly it's a very prestigious one.
freedomben 5mo ago
I mostly agree, though after a mentoring session you can ask it to write skill or a memory and it can be reasonably durable. For Claude at least, the memories work pretty well (though I am still at a small scale with them. As they grow it might start to break somewhat. Doesn't always work, but has often enough that I thought it worth a mention.
kybernetikos 5mo ago
Current LLM architecture doesn't learn - and you're right this is a huge piece that normal folks fail to understand, since in many ways, it's the opposite of what years of AI research has been trying to create. However, I think it's important to remember that LLMs are embedded in larger systems, and those larger systems do learn.
- lukewarm707 5mo ago
  exactly like you said - the harness might learn. we do also have training on synthetic data. it might compound.
- baq 5mo ago
  If I was a frontier lab and I solved continual learning, as of today I would absolutely not release it - the society isn't ready for this; society isn't even ready for widespread diffusion of current publicly available frontier models. If however I was a frontier lab who solved continual learning and my competitor also solved and released it, I would release mine immediately, obviously. The point is, continual learning might be solved already, we just don't know and those who might know would rather keep their mouths shut. It isn't my base case (financial situation of frontier labs is such that they'd probably release immediately as long as they have inference compute to serve this revolutionary capability), but it isn't impossible.
  - bigyabai 5mo ago
    You're not a frontier lab, the shareholders own those. And if shareholders get a private briefing about an unprecedented breakthrough in continual learning, they would announce it from the rooftops to take credit for the progress ASAP and reap the rewards for their stock value. The only lab that I can exempt from this is DARPA.
    - baq 5mo ago
      Shareholders are not insiders. Public companies do secret projects all the time of which shareholders know absolutely nothing about and may never learn the details of them if they get cancelled.
stingraycharles 5mo ago
> Using the word “Mentoring” is anthropomorphic and subconsciously makes you think it will learn. I think this is a bit pedantic. Obviously the parent you’re replying to is referring to the concept of “in-context learning”, which is the actual industry / academic term for this. So you feed it a paper, and then it can use that info, and it needs steering / “mentoring” to be guided into the right direction. Heck the whole name of “machine learning” suggests these things can actually learn. “reasoning” suggests that these things can reason, instead of being fancy, directed autocomplete. Etc. In other news: data hydration doesn’t actually make your data wet. People use / misuse words all the time, and that causes their meaning to evolve.
- kasey_junk 5mo ago
  I agree it’s pedantic and personally don’t get bent out of shape with people anthropomorphizing the llms. But I do think you get better results if keep the text prediction machine mental model in your head as you work with them. And that can be very hard to do given the ui we most interact with them in is a chat session.
  - stingraycharles 5mo ago
    Absolutely, but there is no evidence that the grandparent was doing that, all they did was use the word “mentoring” and my argument is not that anthropomorphizing isn’t a problem - it is - but that the response to this particular HN is super pedantic. Obviously the real people that are classifying AI as human intelligence aren’t going to be the top comment on reviewing LLM’s PhD-level papers. They are on very different, much more problematic areas of the internet.
- nozzlegear 5mo ago
  Anthropomorphism is a subtle marketing tool used by these big AI companies, who are financially incentivized to push the myth of AGI and want everyone to believe they're right on the cusp of achieving it. It's good to be pedantic in this case, we shouldn't anthropomorphize these tools.
DoctorOetker 5mo ago
> ... that something as smart as an LLM does not learn. what? training is learning, as long as weights are available continual learning is perfectly feasible: just keep training the LLM with the user corpus alternated with a frozen version to prevent catastrophic drift / collapse. it's not because model providers don't want to provide user specific continual learning, that we don't know how to do it. it would be a lot more expensive to host user-specific model weights, and would prevent amortizing the weights over many requests in batches...

timschmidt 5mo ago

They can form new associations between concepts via their input prompts and thinking text. That is a form of learning. Just not very durable. I liken it to https://en.wikipedia.org/wiki/Anterograde_amnesia https://en.wikipedia.org/wiki/Anterograde_amnesia

illiac786 5mo ago
yeah, I should have been more specific: I meant the type of learning that mentoring fosters, the long term learning.
- timschmidt 5mo ago
  I hear you. I think we are already seeing some middle ground with agentic systems using RAG, skills.md files, etc. It's a sort of disassociated card catalog memory. An engineer's notebook. Not the integrated, correlated, pre-processed set of relationships in the model. How to go backward from the notebook -> model cheaply without tanking performance is definitely one of those billion dollar questions.
- pickleRick243 5mo ago
  a little glib, but there is in fact long term learning. It's just that you are not the one mentoring- the models go to intensive OpenAI/Anthropic/Google school for a quarter or half a year and come back (hopefully) improved. You just hope they're getting a good education. Certainly it's a very prestigious one.

freedomben 5mo ago

I mostly agree, though after a mentoring session you can ask it to write skill or a memory and it can be reasonably durable. For Claude at least, the memories work pretty well (though I am still at a small scale with them. As they grow it might start to break somewhat. Doesn't always work, but has often enough that I thought it worth a mention.

kybernetikos 5mo ago

Current LLM architecture doesn't learn - and you're right this is a huge piece that normal folks fail to understand, since in many ways, it's the opposite of what years of AI research has been trying to create. However, I think it's important to remember that LLMs are embedded in larger systems, and those larger systems do learn.

lukewarm707 5mo ago
exactly like you said - the harness might learn. we do also have training on synthetic data. it might compound.
baq 5mo ago
If I was a frontier lab and I solved continual learning, as of today I would absolutely not release it - the society isn't ready for this; society isn't even ready for widespread diffusion of current publicly available frontier models. If however I was a frontier lab who solved continual learning and my competitor also solved and released it, I would release mine immediately, obviously. The point is, continual learning might be solved already, we just don't know and those who might know would rather keep their mouths shut. It isn't my base case (financial situation of frontier labs is such that they'd probably release immediately as long as they have inference compute to serve this revolutionary capability), but it isn't impossible.
- bigyabai 5mo ago
  You're not a frontier lab, the shareholders own those. And if shareholders get a private briefing about an unprecedented breakthrough in continual learning, they would announce it from the rooftops to take credit for the progress ASAP and reap the rewards for their stock value. The only lab that I can exempt from this is DARPA.
  - baq 5mo ago
    Shareholders are not insiders. Public companies do secret projects all the time of which shareholders know absolutely nothing about and may never learn the details of them if they get cancelled.

stingraycharles 5mo ago

> Using the word “Mentoring” is anthropomorphic and subconsciously makes you think it will learn. I think this is a bit pedantic. Obviously the parent you’re replying to is referring to the concept of “in-context learning”, which is the actual industry / academic term for this. So you feed it a paper, and then it can use that info, and it needs steering / “mentoring” to be guided into the right direction. Heck the whole name of “machine learning” suggests these things can actually learn. “reasoning” suggests that these things can reason, instead of being fancy, directed autocomplete. Etc. In other news: data hydration doesn’t actually make your data wet. People use / misuse words all the time, and that causes their meaning to evolve.

kasey_junk 5mo ago
I agree it’s pedantic and personally don’t get bent out of shape with people anthropomorphizing the llms. But I do think you get better results if keep the text prediction machine mental model in your head as you work with them. And that can be very hard to do given the ui we most interact with them in is a chat session.
- stingraycharles 5mo ago
  Absolutely, but there is no evidence that the grandparent was doing that, all they did was use the word “mentoring” and my argument is not that anthropomorphizing isn’t a problem - it is - but that the response to this particular HN is super pedantic. Obviously the real people that are classifying AI as human intelligence aren’t going to be the top comment on reviewing LLM’s PhD-level papers. They are on very different, much more problematic areas of the internet.
nozzlegear 5mo ago
Anthropomorphism is a subtle marketing tool used by these big AI companies, who are financially incentivized to push the myth of AGI and want everyone to believe they're right on the cusp of achieving it. It's good to be pedantic in this case, we shouldn't anthropomorphize these tools.

DoctorOetker 5mo ago

> ... that something as smart as an LLM does not learn. what? training is learning, as long as weights are available continual learning is perfectly feasible: just keep training the LLM with the user corpus alternated with a frozen version to prevent catastrophic drift / collapse. it's not because model providers don't want to provide user specific continual learning, that we don't know how to do it. it would be a lot more expensive to host user-specific model weights, and would prevent amortizing the weights over many requests in batches...