5 ms·

I had it summarize a recipe for me. It did a good job, but definitely editorialized. The URL I sent included just the recipe for split pea soup, but the summa

by nvr219 4y ago

I had it summarize a recipe for me. It did a good job, but definitely editorialized. The URL I sent included just the recipe for split pea soup, but the summary added that "Split pea soup is a hearty and comforting dish that is easy to make" (which is true, but wasn't in the original) and said that the soup "is sure to be a hit with the whole family" (again, true, but wasn't in the original). So I feel like "summary" isn't the right word exactly. Still amazing. I tested it with a bunch of different blog posts that I was very familiar with and the summary always turned out accurate.

Brian_K_White 4y ago
This should be more visible. Both in terms of "everyone should see this comment" and "maybe Kagi should be labelling this more truthfully"
- Semaphor 4y ago
  To be fair, it’s also under the `labs` subdomain, which is their experimental stuff.
visarga 4y ago
> So I feel like "summary" isn't the right word exactly. Might be a close cousin of hallucination.
rpastuszak 4y ago
“ Rafal Pastuszak is available throughout the week for a coffee and a chat about ideas big and small. He also has a bi-weekly 2h slot which he uses for pair programming and mentoring sessions with friends, old and new. He also offers a 3 minute rant session, where he listens to the other person rant and then they spend the remaining 7 minutes contemplating the pointlessness of their endeavors. Rafal is interested in a wide range of topics, from product development to mental health, and from photography to emerging technologies. He is also open to pair programming with anyone on their own problem. His goal is to get to know people and what they have been up to. It's an interesting offer for anyone looking for someone to bounce off an idea or learn something new.” Fair enough, there’s a bit of editorialising but the main gist is fine.
jll29 4y ago
It's not "editorialized" in the sense that there are no humans involved. There are two types of automatic document summarization methods, abstractive and extractive. Abstractive analyze the original and then synthesize a summary, whereas extractive summaries are concatenated projections (sequences of text spans) from the original text. The phenomenon you observed suggests this was produced by an abstractive summarizer. Pre-trained neural language models for abstractive summarizations that create pieces of new text that are untrue or make sense is a phenomenon often referred to as "hallucinating" models. In my research, I found extractive summaries are more suitable for professional applications; extractive models never hallucinate, and the worst mistakes are (1) leaving out an important part and (2) creating misunderstandings through left out context. The leaders in the field of summarization research do not believe there can be something like a "universal summarizer", because the same document can be summarized for very different purposes, even by the same person on different occasions). For example, a lawyer may read a case summary one time to get up to speed about what her client got accused of in a lower court trial, and ten years later she may refer so said case again for a trial procedural detail that re-occurred in another case on a completely different subject matter.

by nvr219 4y ago

Brian_K_White 4y ago

This should be more visible. Both in terms of "everyone should see this comment" and "maybe Kagi should be labelling this more truthfully"

Semaphor 4y ago
To be fair, it’s also under the `labs` subdomain, which is their experimental stuff.

visarga 4y ago

> So I feel like "summary" isn't the right word exactly. Might be a close cousin of hallucination.

rpastuszak 4y ago

“ Rafal Pastuszak is available throughout the week for a coffee and a chat about ideas big and small. He also has a bi-weekly 2h slot which he uses for pair programming and mentoring sessions with friends, old and new. He also offers a 3 minute rant session, where he listens to the other person rant and then they spend the remaining 7 minutes contemplating the pointlessness of their endeavors. Rafal is interested in a wide range of topics, from product development to mental health, and from photography to emerging technologies. He is also open to pair programming with anyone on their own problem. His goal is to get to know people and what they have been up to. It's an interesting offer for anyone looking for someone to bounce off an idea or learn something new.” Fair enough, there’s a bit of editorialising but the main gist is fine.

jll29 4y ago

It's not "editorialized" in the sense that there are no humans involved. There are two types of automatic document summarization methods, abstractive and extractive. Abstractive analyze the original and then synthesize a summary, whereas extractive summaries are concatenated projections (sequences of text spans) from the original text. The phenomenon you observed suggests this was produced by an abstractive summarizer. Pre-trained neural language models for abstractive summarizations that create pieces of new text that are untrue or make sense is a phenomenon often referred to as "hallucinating" models. In my research, I found extractive summaries are more suitable for professional applications; extractive models never hallucinate, and the worst mistakes are (1) leaving out an important part and (2) creating misunderstandings through left out context. The leaders in the field of summarization research do not believe there can be something like a "universal summarizer", because the same document can be summarized for very different purposes, even by the same person on different occasions). For example, a lawyer may read a case summary one time to get up to speed about what her client got accused of in a lower court trial, and ten years later she may refer so said case again for a trial procedural detail that re-occurred in another case on a completely different subject matter.