3 ms·
Those tests don't have "construct validity" (i.e. they don't accurately test what they claim to test) with regards to LLMs. Word problems can be used to test hu
by lsy 4y ago
Those tests don't have "construct validity" (i.e. they don't accurately test what they claim to test) with regards to LLMs. Word problems can be used to test human abilities because humans use words to represent interior understanding. LLMs have no interior understanding, so "performing well" on a set of word problems only tells us that the LLM's training set contains enough form to predict the answer, but not that the LLM has any concept of what the words in the question mean or how they might apply to novel situations in the real world.
- moreice 4y agoYou can test this hypothesis by questioning GPT-4 until you hit a detail that it doesn't know. It will say something like "Without more context or a clearer reference to this detail in my training data, I can't provide a definitive answer.". Then you ask it to speculate about what the answer might be.