4 ms·

> And a big strength LLMs have is summarizing things - I’d like to see you summarize the latest 10 arxiv papers relating to prompt engineering and produce a rep

> And a big strength LLMs have is summarizing things - I’d like to see you summarize the latest 10 arxiv papers relating to prompt engineering and produce a report geared towards non-techies. And do this every 30 mins please. Also produce social media threads with that info. Is this a task you could do yourself, better than LLMs? I don't mean to nitpick, but how good do you really think the output of this would be? Papers are short and usually have many references, I would expect the LLM to basically miss the important subtleties on every paper it's given, and misunderstand and misattribute any terms of art it encounters. I mean, of course LLMs are good at summarizing: the summaries are probably mostly sort of good, and anything I'm summarizing I won't read myself. But for technical and specific texts, what's the point when you're getting a "maybe correct" retelling? Best case scenario you get a pretty paragraph that's maybe good for an introduction, and worst case you get incorrect information that misinforms you.

phantompeace 2y ago
The quality of the summary is only as good as the effort you put into writing your workflow. If you’re simply one shotting the paper into a message and saying “plz summarise this and I’ll reward you with $1m” then of course it’s gonna be shit. But if you semantically chunked along sections and do some RAG Q&A summaries before combining into a well formatted schema then it’s probably going to be better than the first way. I’m using the summaries as a juicier abstract. I’m not taking them as gospel. I’m working on following references to then add those papers to a vector db for RAG so it can actually go the step beyond. It’s fun!
- hatefulmoron 2y ago
  > I’m using the summaries as a juicier abstract. I’m not taking them as gospel. I'm not sure of the value of this. Papers already have abstracts, rewording them using LLMs is just playing with your food. If you're seeing use out of it that's awesome though.
  - phantompeace 2y ago
    You do have a point you know. I have actually been thinking about this recently and have decided to try and focus more on extracting value out of abstracts instead of summarising papers, and relying on embeddings of the paper in case the answer needs more context.

phantompeace 2y ago

The quality of the summary is only as good as the effort you put into writing your workflow. If you’re simply one shotting the paper into a message and saying “plz summarise this and I’ll reward you with $1m” then of course it’s gonna be shit. But if you semantically chunked along sections and do some RAG Q&A summaries before combining into a well formatted schema then it’s probably going to be better than the first way. I’m using the summaries as a juicier abstract. I’m not taking them as gospel. I’m working on following references to then add those papers to a vector db for RAG so it can actually go the step beyond. It’s fun!

hatefulmoron 2y ago
> I’m using the summaries as a juicier abstract. I’m not taking them as gospel. I'm not sure of the value of this. Papers already have abstracts, rewording them using LLMs is just playing with your food. If you're seeing use out of it that's awesome though.
- phantompeace 2y ago
  You do have a point you know. I have actually been thinking about this recently and have decided to try and focus more on extracting value out of abstracts instead of summarising papers, and relying on embeddings of the paper in case the answer needs more context.