3 ms·

It's interesting to me that these models confabulate so readily; I'm curious why it happens at all.

• Llamamoe 1y ago
  Before RLHF, they're just a fancy autocomplete engine trained on the entire web and countless books, and text including stupidly wrong information is simply more common than text which goes "Hold up, that's wrong, it's actually X" midway. Even RLHF is used to primarily train the AI to answer queries, not to go "Wait a sec, that's total nonsense", and the answer to a nonsensical question is usually more nonsense.
  • solarwindy 1y ago
    When framed like this, it's quite unsurprising that LLMs struggle to emulate reasoning through programming problems: there's just not that much signal out there. We tend to commit what already works, without showing much (if any) of the working. A test for generality of intelligence, then: being able to apply abstract reasoning processes from a domain rich in signal to a novel domain. Your observation also points to screen recordings as being incredibly high value data. Good luck persuading anyone already concerned for their job security to go along with that.