3 ms·

Interesting. I read through the ChatGPT summary first, it seemed very plausible, then I read the original and I kinda see the author's point. The ChatGPT summ

Interesting. I read through the ChatGPT summary first, it seemed very plausible, then I read the original and I kinda see the author's point. The ChatGPT summary basically did gloss over every important detail in the original - but then, the details weren't important for generating a summary. I think that one of the author's key premises is false: > To summarise, you need to understand what the paper is saying. A summary is not about the author or about the summarizer, it's about the reader. It's about picking up on the portions of the original work that will matter to the reader's estimation of whether it's important to read the full work. And that actually depends much more on context and how the work relates to other works than it does about the specific details contained in the work. For example, Betteridge's Law of Headlines [1] basically provides a summary of any article whose title ends in a question, that summary is just "No", and it does so by making an observation about the authors rather than the content of the articles (about which it's completely agnostic). It reminds me of the problems that plague AI sentiment analysis. Machine learning is actually very good at that task, but you top out at about 70% precision because humans top out at 70% precision on sentiment analysis. At best, people only agree with each other ~70% of the time when judging the sentiment of a piece. [1] https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headlines https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headline...

sebastiennight 2y ago
- You're assuming that the goal of a summary is to decide whether one needs to read the full work. That's certainly one niche usage of a summary, but is definitely not the only one and is likely a minority of usage. Most summaries I've needed, from school to my career, were about not having to read the full work. - I would say Betteridge's Law of Headlines does not provide "No" as a summary to all of these articles. The accurate summary would be "[Title]? No.", since it seems fairly obvious that the word "No" is very light in information conveyed. On a personal note, I have to say I can't see the point in this current craze about summarizing everything. I never saw the point of those subscription programs which promise you'll "read" a book a week/a month because they'll send you a 5-min audio about the book. I think you're better off choosing one really awesome book a year and actually reading that. So I can't see how a (flawed or not) ChatGPT summary will provide any epiphany on the level you'd get from consuming fewer works, thoroughly.
Animats 2y ago
What this sounds like is this: - The author is pushing a specific proposal (a "council of stakeholders") in the pension space. - To support that, he wrote a long paper with much supporting information. - The ChatGPT summary system didn't pick up on his proposal being the important thing. - The author doesn't like that.

sebastiennight 2y ago

- You're assuming that the goal of a summary is to decide whether one needs to read the full work. That's certainly one niche usage of a summary, but is definitely not the only one and is likely a minority of usage. Most summaries I've needed, from school to my career, were about not having to read the full work. - I would say Betteridge's Law of Headlines does not provide "No" as a summary to all of these articles. The accurate summary would be "[Title]? No.", since it seems fairly obvious that the word "No" is very light in information conveyed. On a personal note, I have to say I can't see the point in this current craze about summarizing everything. I never saw the point of those subscription programs which promise you'll "read" a book a week/a month because they'll send you a 5-min audio about the book. I think you're better off choosing one really awesome book a year and actually reading that. So I can't see how a (flawed or not) ChatGPT summary will provide any epiphany on the level you'd get from consuming fewer works, thoroughly.

Animats 2y ago

What this sounds like is this: - The author is pushing a specific proposal (a "council of stakeholders") in the pension space. - To support that, he wrote a long paper with much supporting information. - The ChatGPT summary system didn't pick up on his proposal being the important thing. - The author doesn't like that.