4 ms·
Are there any objective benchmarks for rating and comparing the summarization performance of LLMs? I've had mixed results when using the latest versions of Chat
by tkgally 2y ago
Are there any objective benchmarks for rating and comparing the summarization performance of LLMs? I've had mixed results when using the latest versions of ChatGPT, Claude, and Gemini to summarize texts that I know well (books I have read, papers I wrote, transcripts of online meetings I attended). Often their summaries are as good as or maybe even better than what I could prepare myself, but sometimes they omit key points or entire sections. Other than failures clearly due to context length limitations, it's hard to judge which LLM is a better summarizer overall or whether a new LLM version is better than the previous one.