3 ms·
GPT-4 doesn't really fail at simple arithmetic. But it's a harder task to learn because arithmetic doesn't encode information about its solution in preceding c
by famouswaffles 4y ago
GPT-4 doesn't really fail at simple arithmetic.
But it's a harder task to learn because arithmetic doesn't encode information about its solution in preceding context. "9383 + 3545" or "is any of the following numbers a prime, 96885, 66576, 4766 ?" doesn't actually tell you anything that would inform the answer. You go to school and you learn the required set of steps for solving these problems.
On the hand, for "John is smiling so he is _____", the preceding context screams happy as a very likely choice. Preceding context actually helps finding the solution rather than being the equivalent of deadweight.
And you simply didn't understand LLMs as well as you thought you did.
Language models trained on Code reason better, even on benchmarks that have nothing to do with code.
https://arxiv.org/abs/2210.07128 https://arxiv.org/abs/2210.07128
Encoding/Decoding Base64 is neat but not particularly mindblowing unless you have some serious misconceptions on what language models are capable of.
- sinuhe69 4y agoAlbeit a bit condescending, you answer didn’t actually answer the OP’s question. So how could GPT-4 encode and decode base64? Was it explicitly trained on this task or there are enough base64 text in the world so GPT could “automatically” pick it up? Or is it even an emergent property?
- famouswaffles 4y agoIf there was enough Base64 in the training set then it would pick it up. It's just more text to predict. The best prediction is the correct prediction. Maybe this answer seems a bit lacking but no one, not even the people training these models know exactly what it is these artificial neurons are learning or doing during inference so anything more is more speculation than not.
- beebmam 4y ago"Base64 conversions are simple arithmetic", I think is the missing piece to connecting their argument to the OP's question
- famouswaffles 4y agoThe problem with that is that there's no guarantee it Zero shots base64 enc/dec the same way people would do it. and there's no way to tell. for all we know, it treats it as just another "translate x to y" task.