Elements used in GPT-3 are capable of transformations such as abstraction (e.g. separate the structure of syllogism from concrete nouns) and logic (ReLU can dir

Elements used in GPT-3 are capable of transformations such as abstraction (e.g. separate the structure of syllogism from concrete nouns) and logic (ReLU can directly implement OR, AND, NOT which is sufficient to do arbitrary logic). We can see that it actually uses abstractions and logic in some cases. E.g. "Bob is a frog. Bob's skin color is ___". Even small GPT-2 models can relate "Bob" to concept "frog" and query "frog" "skin color" attribute. Even basic language modeling requires inference, and GPT-x can inference using transformer blocks. With more layers it can go from inferencing meaning of words to doing inference to solve problems. But the inference it is able to do is limited in scope because of the structure of a language model -- each input token must correspond to one output token. So the model can't take a pause and think about something, it can only think while it produces tokens. Here's an absolutely insane example of embedding symbolic computation into a story which lets GPT-3 to break computation into small steps it can handle. Intermediate results become part of the story: https://twitter.com/kleptid/status/1284069270603866113 https://twitter.com/kleptid/status/1284069270603866113 https://twitter.com/kleptid/status/1284098635689611264 https://twitter.com/kleptid/status/1284098635689611264 So I guess one can make a model which is much better at thinking simply by training in a different way or changing the topology. But the building blocks are good enough.