4 ms·

Clearly LLMs cant do leaps of intuition since their "intuition" is locked after training ends. The only way a LLM can come up with new ideas if the "idea" appe

Clearly LLMs cant do leaps of intuition since their "intuition" is locked after training ends. The only way a LLM can come up with new ideas if the "idea" appeared as a generalisation durring training or if it was achieved using reason in chain of thought.

buzzin__ 2mo ago
Or some randomness is aomehowntroduced in the output, which happens after every word, unless you set the temperature to zero.
- reliablereason 2mo ago
  That would not be intuition that is just randomness. Intuition is not randomness. A jump in intuition comes from automatic processes reorganising the relational structure of conceptual models. There is no reorganisation of the model durring inference.
  - tpolm 2mo ago
    > There is no reorganisation of the model durring inference. one could argue that model can reorganize / interact with prior knowledge captured in text form (edit files) hence there can be reorganisation
    - reliablereason 2mo ago
      That would "using reason in chain of thought".
Enginerrrd 2mo ago
I think this is a bad way to look at it. LLMs can probably conceive of most things that are representable within the embedding space. Ordinarily in mathematics there’s a TON of papers to write just combining low level problems with different techniques. Better still, and often considered groundbreaking is borrowing techniques from other fields and adapting them or creating analogous methods to solve problems. A lot of landmark papers have been written this way. This is also what transformers are sort of good at within other contexts. They have super human breadth so I’m hopeful they’ll become real assets in math for a long time. Though the leaps necessary to adapt a technique in a nonobvious way might be too much for a while longer. We’ll see. Truly novel techniques are quite rate indeed and I don’t know if LLMs can represent them faithfully in their embedding space or not. My inclination is that they probably can most of the time, but I don’t know. Mathematicians would describe such thins as “alien”.

buzzin__ 2mo ago

Or some randomness is aomehowntroduced in the output, which happens after every word, unless you set the temperature to zero.

reliablereason 2mo ago
That would not be intuition that is just randomness. Intuition is not randomness. A jump in intuition comes from automatic processes reorganising the relational structure of conceptual models. There is no reorganisation of the model durring inference.
- tpolm 2mo ago
  > There is no reorganisation of the model durring inference. one could argue that model can reorganize / interact with prior knowledge captured in text form (edit files) hence there can be reorganisation
  - reliablereason 2mo ago
    That would "using reason in chain of thought".

Enginerrrd 2mo ago

I think this is a bad way to look at it. LLMs can probably conceive of most things that are representable within the embedding space. Ordinarily in mathematics there’s a TON of papers to write just combining low level problems with different techniques. Better still, and often considered groundbreaking is borrowing techniques from other fields and adapting them or creating analogous methods to solve problems. A lot of landmark papers have been written this way. This is also what transformers are sort of good at within other contexts. They have super human breadth so I’m hopeful they’ll become real assets in math for a long time. Though the leaps necessary to adapt a technique in a nonobvious way might be too much for a while longer. We’ll see. Truly novel techniques are quite rate indeed and I don’t know if LLMs can represent them faithfully in their embedding space or not. My inclination is that they probably can most of the time, but I don’t know. Mathematicians would describe such thins as “alien”.