4 ms·

When you ask ChatGPT to write a piece of code that does a thing, it is not reasoning about what the code should do, modeling the problem in its head, figuring o

When you ask ChatGPT to write a piece of code that does a thing, it is not reasoning about what the code should do, modeling the problem in its head, figuring out how to translate that into code, then writing it. The neural weights are spitting up a piece of code it determines would probably come after your prompt, based on similarity to a kajillion StackOverflow answers, GitHub repos, textbooks, etc etc. It is not coming up with a solution, just giving you the average answer to the million most similar questions. ChatGPT can “write code” in the most literal sense that it will give you code when given a prompt, but (unlike how OpenAI markets-by-omission) it does not reason about code like an AGI could.

nyolfen 3y ago
gpt is clearly capable of constructing and interacting with a model of the world -- there was a famous example in the microsoft bing paper where they described a complex maze and had it reason its way out. 'predicting what it thinks is most similar to stackoverflow' etc does not explain any of its behavior in a meaningful way, unless you are willing to apply this description to human intellectual labor. gpt4 is clearly not sentient, but what is shocking about it is that it demonstrates that honest-to-god cognition can be decoupled from sentience. btw, i think i have tended to see these pessimistic takes by people who have only used 3.5, which is impressive but not reliable -- i strongly recommend using 4 in the course of your job to see how useful it is. every day in the course of interacting with it, it produces novel lines of thought which shock me.
simonw 3y ago
I'm not particularly interested in AGIs (I'm kind of hoping they don't happen) - but for my purposes the illusion of LLMs being able to code is now good enough that I'm using them to write good, working code for me on a daily basis. Here's a recent example: https://simonwillison.net/2023/Aug/6/annotated-presentations/#chatgpt-sessions https://simonwillison.net/2023/Aug/6/annotated-presentations...

nyolfen 3y ago

gpt is clearly capable of constructing and interacting with a model of the world -- there was a famous example in the microsoft bing paper where they described a complex maze and had it reason its way out. 'predicting what it thinks is most similar to stackoverflow' etc does not explain any of its behavior in a meaningful way, unless you are willing to apply this description to human intellectual labor. gpt4 is clearly not sentient, but what is shocking about it is that it demonstrates that honest-to-god cognition can be decoupled from sentience. btw, i think i have tended to see these pessimistic takes by people who have only used 3.5, which is impressive but not reliable -- i strongly recommend using 4 in the course of your job to see how useful it is. every day in the course of interacting with it, it produces novel lines of thought which shock me.

simonw 3y ago

I'm not particularly interested in AGIs (I'm kind of hoping they don't happen) - but for my purposes the illusion of LLMs being able to code is now good enough that I'm using them to write good, working code for me on a daily basis. Here's a recent example: https://simonwillison.net/2023/Aug/6/annotated-presentations/#chatgpt-sessions https://simonwillison.net/2023/Aug/6/annotated-presentations...