3 ms·
I don't appreciate your snark at all. I made a mistake and didn't read the paper carefully again so I confused myself with what they mean by one- two- and three
by YeGoblynQueenne 6y ago
I don't appreciate your snark at all. I made a mistake and didn't read the
paper carefully again so I confused myself with what they mean by one- two-
and three-digit tasks, I accept that. I don't see what you get from pouncing
on my mistake, other than a few marks for an internet burn.
Now, the two- and three digit addition and subtraction tasks (operations on
numbers between 0 and 99 and 0 and 999, respectively) are both small enough
for the large, 175B parameter model to have memorised them exactly. Even if
there was a single parameter for each three-digit number, of which there are a
million, you could fit the entire set 175 thousand times in the 175 billion
model (assuming they mean "a billion" as "one thousand million", not "one
million million", which they don't clarify, but to be on the safe side let's
assume the smallest). There is plenty of room.
These four tasks are also the tasks that are most likely to be present in
their entirety in a corpus of natural language, as the one GPT-3 was trained
on, for example as records of common monetary transactions (especially the
two-digit ones). That is, yes, the training set can comfortably contain 800k
3-digit addition problems. Why not? It contained 410 billion tokens from the
Common Crawl dataset alone, plus a few extras.
In short, the almost perfect accuracy on this task is not impressive. The 25%
ish accuracy on the four-digit addition task is even less impressive. I don't
know what the baseline is here, but 25% accuracy on anything is not something
to write home about.
You ask me to provide evidence of my own to support the memorisation claim.
The claim is not memorisation. The claim is that GPT-3 has learned arithmetic
(not stated exacly like that in the paper). This claim flies in the face of
the commonly understood operation of language models, which are systems that
compute the probability of a token to follow a sequnce of tokens- and nothing
else. It's very hard to see how such a system should be able to perform
arithmetic operations, while it's very easy to see how it can instead memorise
their results. If the authors of the GPT-3 paper wish to claim that GPT-3 can
perform arithmetic, instead of the much simpler explanation, they have to
provide very strong evidence to back that up and refute the simpler
explanation.
And the "spot checks" that they performed are nowhere near such strong
evidence: I can fail to find anything I search for, if I search with the wrong
terms and the authors don't give much information about how they did their
"spot checks". I mean, did they use a regular expression? Which one? ("<NUM1>
+ <NUM2> =" is not a regular expression! But then - what is it?) Did they take
into account whitespace? Punctuation? Something else? What search terms they
used? They dont' say. Can we tell why they failed to find what they were
looking for? No.
Besides, why only "spot check" three-digit arithmetic? It would make a lot
more sense to spot-check two-digit problems, first, because these are the
most likely to be found more often in the dataset and consequently be
memorised. Indeed, the fact that they don't report "spot checks" for two-digit
arithmetic suggests that they did perform those spot checks and they found a
lot more overlap than for the three digit arithmetic, but chose not to report
it. And if their model was memorising two-digit arithmetic, and that explains
its performance on that type of task, it's safe to assume that it was
memorising the third-digit arithmetic task also and that their "spot checks"
were simply not very well put together to find the three-digit arithmetic
examples.
Note that section 4 goes in length over the possibility that the test set for
all tasks (not just arithmetic) was contaminated (i.e. that it containted
training examples from existing benchmarks, published on the internet). I
haven't read that one carefully but test set contamination is another
possibility. And, to be frank, any possibility is more possible than the
possibility that a langauge model has learned arithmetic- which is tantamount
to magick.
- jointpdf 6y agoI’m not attacking you, which is why I immediately offered you a salve for the burn by marking my comment as snarky. It did have a real point though—that learning (even something inherently logical like arithmetic) is not a binary outcome, as fossuser also pointed out. The claim that “GPT-3’s performance on arithmetic tasks is solely due to memorization / data leakage—it has no generalization ability on this type of task”, is easily attackable by...well, doing arithmetic and applying common sense. There are 2(10^5)^2 = 20 billion possible 5 digit problems (both addition and subtraction). The accuracy on those tasks is about 10%, so roughly 2 billion* 5-digit addition and subtraction problems would need to be represented in the training data (Common Crawl + books as you said). Each problem is at minimum 5 tokens (e.g. 99999 + 11111 = 111110). So is ~2.5% of the training corpus 5-digit addition and subtraction problems that eluded their filtration process? (assuming it’s ~400B tokens like you said). Seems exceedingly unlikely, so much so that memorization ceases to be the simplest explanation. That said, yes it is surprising that a language model can generalize in this way—that’s the point of the paper. How exactly this happens seems like a valuable thread to pull. Your critiques may help, but writing the results off as impossible magic does not.
- YeGoblynQueenne 6y agoI was a bit touchy yesterday, I guess - thanks for the salve. About the 5-digit problems- I didn't make this clear but I don't think those were memorised. I think the two- and three-digit problems (all three operations) were memorised, because those are the most likely to be represented in their entirety, or close, in GPT-3's training corpus, given that they are operations that are common to very common in daily life. I doubt that the four- and five-digit addition problems (and the single-digit, multi-op problem) were represented often enough in GPT-3's training corpus for them to be memorised. I think the low accuracy in these problems (less than 10% in the few-shot setting and near zero in the zero- and one-shot) is low enough that it doesn't require an explanation other than a mix of luck and overfitting that is common enough in machine learning algorithms that it's no surprise. e.g. we evaluate classifiers using diverse metrics, not just accuracy, because this is so common. It is this observation, that GPT-3 did well in problems that are likely to be well reprsented in its training corpus and badly in ones that aren't, that convinces me that no more complicated explanation is needed than memorisation. Something else. Like I say above, we evaluate classifiers not only by accuracy (the rate of correct answers), because accuracy can be misleading. e.g. a classifier can have 100% accuracy with 0% false positives and 100% false negatives. The GPT-3 authors only tested the ability of their model to give answers to problems stated as "x + y = ". They didn't test, e.g. what happens if they prompt it with "10 + 20 = 40, 38 + 25 = ". Testing for aberrant answers following from such confusing prompts has often showed that language models that appear to be answering questions correctly because of a deep understanding of language are in truth overfitting to surface statistical regularities. See for example [1,2] and many other references in [3]. Indeed, I could be wrong about rote memorisation and GPT-3 can still not be learning to perform arithmetic computations, given the tendency of language models to learn spurious correlations. There is an article about a mathemagician on the front page today, that shows how she found roots of huge numbers by finding shortcuts around expensive calculations. For instance, all sums between numbers ending in 5 end in 0, etc. I wouldn't find it magickal if a language model was finding such heuristics and that this is the "something else" that is said to be going on. However that would not be "generalisation" and it would not be learning to perform arithmetic. In the end, I don't understand how a model can be said to know how to add two-digit numbers perfectly but not five-digit numbers. If it's performing an incomplete computation in the latter case, then what kind of incomplete computation is it performing? If it "gets it wrong after three digits" then why does it get three digits right? What's the big difference between three- and four-digit numbers that causes performance to fall off a cliff - other than the chance of finding such numbers in a natural language corpus? As to magick- I'm writing off not the results, but the hand-waving presented in place of an explanation as magick. GPT-3 is a technological artifact designed to do one job, now reported to be doing another. This requires a thorough explanation but instead we got magickal thinking: the authors wish that their models could learn arithmetic, so they took its behaviour as proof that it learned arithmetic. ___________________ [1] Probing Neural Network Comprehension of Natural Language Arguments https://www.aclweb.org/anthology/P19-1459/ https://www.aclweb.org/anthology/P19-1459/ [2] Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference https://www.aclweb.org/anthology/P19-1334/ https://www.aclweb.org/anthology/P19-1334/ [3] https://www.technologyreview.com/2020/07/31/1005876/natural-language-processing-evaluation-ai-opinion/ https://www.technologyreview.com/2020/07/31/1005876/natural-... (try F9 if you 're over limit)