3 ms·

It can't do math because it is operating in a single neural network path. You, also, cannot do math in a single neural network path. Even when you add two small

It can't do math because it is operating in a single neural network path. You, also, cannot do math in a single neural network path. Even when you add two small numbers like 123+456 your brain is mentally iterating over the digits, detecting if a carry is needed, doing that, etc. That is, you have a looping/recursive process running inside your brain. You only output the final answer. GPT does not have such a looping/recursive process inside its neural net. It's a fixed depth non-recursive neural net. You can get it emulate recursive processes by prompting it with tricks like "think step by step". If you describe the addition algorithm you learn in elementary school (e.g. the digit-by-digit, carry if the sum exceeds 9, etc) in sufficient detail, it can execute that algorithm.

zwaps 4y ago
Gpt3 is a graph neural network with added recursive information through positioninal encoding. It outputs sequentially, but I am not sure why a RNN would be required beyond that. I would agree that the manner of reasoning must differ, since the sequence follows bpe tokens rather than logical steps, however who is to say that another form of mathematical reasoning could not lead to valid results? For instance, Gpt might solve the problem at each output step insofar as required to generate that token. It certainly iterates over each output token, and the encoding of the problem is equivalent so iterating over the characters of the math problem, roughly speaking. But yes, the iterated output does not follow a logical graph externally, it is token by token. But internally, the network can absolutely follow a more complex graph. Could you say what you mean by single network path when we speak about attention based architectures? What sort of operation or information is missing in such architecture? I am aware of of some results relating to certain graphs, but I do not think this would apply to a text describing a math problem, say.

zwaps 4y ago

Gpt3 is a graph neural network with added recursive information through positioninal encoding. It outputs sequentially, but I am not sure why a RNN would be required beyond that. I would agree that the manner of reasoning must differ, since the sequence follows bpe tokens rather than logical steps, however who is to say that another form of mathematical reasoning could not lead to valid results? For instance, Gpt might solve the problem at each output step insofar as required to generate that token. It certainly iterates over each output token, and the encoding of the problem is equivalent so iterating over the characters of the math problem, roughly speaking. But yes, the iterated output does not follow a logical graph externally, it is token by token. But internally, the network can absolutely follow a more complex graph. Could you say what you mean by single network path when we speak about attention based architectures? What sort of operation or information is missing in such architecture? I am aware of of some results relating to certain graphs, but I do not think this would apply to a text describing a math problem, say.