3 ms·
How do you quantify capability though? In coding or math there's an easy way to validate the output. That makes it easy to judge capability. But for many cognit
by kjcharles 23d ago
How do you quantify capability though? In coding or math there's an easy way to validate the output. That makes it easy to judge capability. But for many cognitive tasks the output is either subjective, not quantifiable, or requires deterministic results. No validation can tell you if an essay is good. Or if an argument will be persuasive to a specific audience. Or that the statistics an LLM pulled from a data source are accurate.
So how does an LLM learn to outperform humans when its output in a large number of tasks can't be validated? These sorts of tasks are a large part of cognitive work and intelligence to me.
- mdspan 23d agoQuantitative data for subjective output can come from opinion: contests, polls, reviews, A/B tests. A problem though is that LLM adoption has grown to the point that their output can alter people's preferences, like we've already seen with the overuse of em-dashes by agents. I think that might be one of the bigger obstacles AI companies will face when getting LLMs perform well on non-quantitative tasks.