3 ms·
Yes, I don't understand why they are using a search benchmark for these... it would be much better to have something like giving it a story up to the context le
by andersa 3y ago
Yes, I don't understand why they are using a search benchmark for these... it would be much better to have something like giving it a story up to the context length (from a book? how to find one that's not in the training data?) and have it write a new chapter/ending that is consistent with all prior text and introduces zero inconsistencies.
But how can you automatically evaluate whether it did this?
- ricw 3y agoBecause that’s how people use llms. You go to ChatGPT to ask a question and get an answer, rather than searching on Google, revising your search because you didn’t know a term, and then look at 3-5 different links to find the answer to what you were searching for
- loa_in_ 3y agoThat's how people use llms because they (llms) don't seem to be good at the more sophisticated thing
- sp332 3y agoBut also because search engines have gotten even worse at answering questions.
- zone411 3y agoIt seems you might be mixing up different types of "context" in LLM benchmarking. In this case, it refers to the input text directly provided to the model during evaluation by the user (as in in-context learning). This is separate from the text an LLM is trained on or can access via RAG methods.