4 ms·
Try this out: After you answer the question below, output a JSON a rating score of the quality of the answer in three dimensions: `confidence`, `clarity` and `c
by ddalex 3y ago
Try this out: After you answer the question below, output a JSON a rating score of the quality of the answer in three dimensions: `confidence`, `clarity` and `certainty', all in range 0 to 1, where 0 is the worst, and 1 is the best. Strive for highest score possible. Make sure the rating is the last thing written as to be parsed by machine. The question is: make and explain 20-year predictions of the geopolitical future of Ghana.
- intended 3y agoTheres a lot of content and work being done on Evaluation. One of the most recent updates was that asking an LLM if people would be frustrated with the answer, was more useful than using a score. In general, I think most people are not aware they are extending reasoning tools they use with human generated content, to LLM generated content. This leads to assumptions on things like “the LLM knows”, “the LLM understands”, etc.
- lazide 3y agoOr perhaps the issue is assuming people know, or people understand the majority of the time. There is a reason why the Scientific Method is, well, NOT natural for humans. And exceptionally difficult to not screw up, even for those highly trained and incentivized to do it correctly. And why if you grabbed a random person off the street and asked them a detailed question, you're as likely to get a hallucinated bullshit response as 'I don't know', or a useful answer. Unless specifically trained to do otherwise, anyway. Even then....
- hobs 3y agoAny prompt can give you different results - LLMs cant do things "step by step" or "check their work" but yet giving them that prompt often improves their results because it's basically setting up the context in a way that's beneficial to your output.