3 ms·
It would seem that this can be easily analysed scientifically. To give a simple example: if, hypothetically, someone thought that GPT-3 is good at basic arithm
by FartyMcFarter 6y ago
It would seem that this can be easily analysed scientifically.
To give a simple example: if, hypothetically, someone thought that GPT-3 is good at basic arithmetic (1 plus 1, 1000 times 3 etc.), they can provide a template for how to ask GPT-3 questions about arithmetic. Anyone can then verify that this template results in accurate answers, by asking randomly sampled questions using that template.
This verification method could be applied to pretty much any problem. Has anyone done anything like that?
- computerphage 6y agoThe original paper is full of comparisons and benchmarks. It includes a section on arithmetic.
- FartyMcFarter 6y agoThanks! The corresponding graphs in the paper show that it's OK at two-digit operations (except multiplication), but it doesn't generalize to bigger numbers. This would seem to support LeCun's statement that there's a lot of over-hyping going on.