5 ms·
Theres a lot of content and work being done on Evaluation. One of the most recent updates was that asking an LLM if people would be frustrated with the answer,
by intended 3y ago
Theres a lot of content and work being done on Evaluation. One of the most recent updates was that asking an LLM if people would be frustrated with the answer, was more useful than using a score.
In general, I think most people are not aware they are extending reasoning tools they use with human generated content, to LLM generated content.
This leads to assumptions on things like “the LLM knows”, “the LLM understands”, etc.
- lazide 3y agoOr perhaps the issue is assuming people know, or people understand the majority of the time. There is a reason why the Scientific Method is, well, NOT natural for humans. And exceptionally difficult to not screw up, even for those highly trained and incentivized to do it correctly. And why if you grabbed a random person off the street and asked them a detailed question, you're as likely to get a hallucinated bullshit response as 'I don't know', or a useful answer. Unless specifically trained to do otherwise, anyway. Even then....