4 ms·

>> My take is: does a human programmer in this case would have to consciously talk to themselves to not make a mistake? "Remember, remember, 'el eye en' prints,

>> My take is: does a human programmer in this case would have to consciously talk to themselves to not make a mistake? "Remember, remember, 'el eye en' prints, 'pee ar aye en tee' computes length". Would they have to double-check their output? If yes, then don't expect LLMs to do better in general case, unless you make them do those steps. I'm pretty sure most people would not need to do any of that to answer the question in the motivating example correctly. Remember: it's three lines and a question. Pretty much the Country Music of experiments, eh?

TeMPOraL 3y ago
> Remember: it's three lines and a question. Pretty much the Country Music of experiments, eh? IDK, three lines of math is more than enough for me to make fatal mistakes in. For example, in my secondary and high school times, it took me years to stop carelessly forgetting to carry a minus sign during calculations - even as simple as requiring three lines. Another analogy - you may argue this is goal-post shifting, but I'm really not inventing new analogies here, only bringing up the other one of the two I always use - GPT-4 being like a 4 year old kid in terms of focus. Anyway: the few tests I run tended to show GPT-4 first understanding the problem correctly, and then confusing itself mid-way through its output. This is quite similar to what my 4 y.o. often does. She'll say stuff like "no, X isn't like Y, X is more like Z, it's not like Y", and then suddenly continue as if X was like Y, as if the idea of X being like Y overpowered her earlier assertion that it's not.
- YeGoblynQueenne 3y ago
  I think you're really good at convincing yourself of whatever you want to be convinced :P
  - TeMPOraL 3y ago
    Quite possibly :). Not going to dispute that. But FWIW, it seems to be mostly working well for me, so there must be at least some correlation with objective reality here. Or at least that's what I want to believe :).

TeMPOraL 3y ago

> Remember: it's three lines and a question. Pretty much the Country Music of experiments, eh? IDK, three lines of math is more than enough for me to make fatal mistakes in. For example, in my secondary and high school times, it took me years to stop carelessly forgetting to carry a minus sign during calculations - even as simple as requiring three lines. Another analogy - you may argue this is goal-post shifting, but I'm really not inventing new analogies here, only bringing up the other one of the two I always use - GPT-4 being like a 4 year old kid in terms of focus. Anyway: the few tests I run tended to show GPT-4 first understanding the problem correctly, and then confusing itself mid-way through its output. This is quite similar to what my 4 y.o. often does. She'll say stuff like "no, X isn't like Y, X is more like Z, it's not like Y", and then suddenly continue as if X was like Y, as if the idea of X being like Y overpowered her earlier assertion that it's not.

YeGoblynQueenne 3y ago
I think you're really good at convincing yourself of whatever you want to be convinced :P
- TeMPOraL 3y ago
  Quite possibly :). Not going to dispute that. But FWIW, it seems to be mostly working well for me, so there must be at least some correlation with objective reality here. Or at least that's what I want to believe :).