3 ms·
A problem with summaries is that it is not possible to verify their correctness without creating a summary by a human from scratch. In other words validation o
by sorokod 3y ago
A problem with summaries is that it is not possible to verify their correctness without creating a summary by a human from scratch.
In other words validation of the task costs at least as much as the original task.
This is different from say code generation where, to first approximation at least, automated tools (compilers, unit tests) can do this cheaply.
- pixl97 3y agoI mean technically you have no idea how accurate the human summary is without multiple summaries.
- vsnf 3y agoBut I can incentivize, hold accountable, and hire people I trust to be competent. I cannot incentivize, hold accountable, or higher better quality LLMs.
- sorokod 3y agoTrue, we can and do make assumptions about quality based on our previous experience. I don't see an equivalent approach that can be applied to LLMs.