9 ms·
ArxivGPT: Chrome extension that summarizes arxived research papers using ChatGPT
- aix1 4y agoIsn't this what abstracts are for?
- muzani 4y agoOne of my first serious uses for GPT is summarizing the abstracts too.
- elashri 4y agoI don't think it is seroud. Abstract are usually tends to be a short summary. In many cases it is a short paragraph. I was skimming on arxiv before I opened HN now and saw a couple of abstracts that are 4-5 sentences. Are ChatGPT going to write a one sentence summary or will the summary of the summary be comparable or longer?
- krageon 4y agoAbstracts can also be close to a full page long. While this doesn't have to be bad, it's usually more information you're looking for (especially if you were only going to read the abstract anyway).
- mtlmtlmtlmtl 4y agoI can skim several full pages in the time it takes just to send off the abstract and get the response back...
- visarga 4y agothen batch summarise your paper feed every morning
- mtlmtlmtlmtl 4y agoI would prefer to optimise things that actually take a lot of time. It often takes me just a few seconds to eliminate a paper as irrelevant to my search based on its abstract. If I can't, I'm gonna have to skim the paper anyway. But even that could be pretty quick. Quickly reading something to gauge relevance is something I can confidently say I do much faster than GPT can. And I don't have a paper feed. I look up papers relevant to what I'm working on at the time.
- krageon 4y agoCan you skim several full pages minus one and read the abstract last? If it's not effort you have to expend, you've won.
- muzani 4y agoLength isn't always the indicator. Abstracts can be information dense, and sometimes more words is simpler to understand, or sometimes it just has jargon. For example, this paper on GPT-3: "Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions - something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine- tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning. with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general." The summary: - If you train a computer on a lot of words, it can do things a lot better - In this case, the computer has learned how to translate languages, answer questions, and unscramble words - It still has trouble with some things - This is very interesting because it shows computers can do things which are very hard. So here, it drops a lot of information (cloze tasks), and it adds some (that last point). But now I know what the paper is about in 10 seconds. I go back and see that oh, "a lot of words" really means 10x the previous. I'm now hooked on what problems it has trouble with, and what problems it solves for humans. I didn't know a thing about LLMs when I first read this. If I tried to read it top to bottom, I'd get stuck on "task-agnostic, few-shot performance" then "state-of-the-art fine- tuning approaches" then "an autoregressive language model". They're big words, but turns out they're not the interesting parts, and understanding what the paper is excited about helped me to understand the basics.
- stevenrj 4y agoNot only that, it appears to only feed the abstract to ChatGPT anyway.
- shusaku 4y agoAs far as I can tell, it just rephrases the abstracts...
- goldemerald 4y agoThe example here is a bit worrying for the peer review process. I am not looking forward to my "peers" reviewing my paper by putting it through LLMs and blindly copy pasting the output. I can already imagine emailing the Area Chair and saying "While reviewer 2 is detailed, the questions show a severe lack of basic understanding. We believe the contents are AI generated." Then again, perhaps LLMs could simply be incorporated into the peer-review process, where after submitting your paper, you'd have to answer the AI's basic questions. As a reviewer, I could imagine a structured AI report for a paper being helpful in guiding discussion: "The paper compares to recent approaches X, Y, and Z. And the work is indeed novel."
- asabla 4y agoI like the way you flipped it around. I'm a big believer of using all these new AI services/programs as tools to enhance my workflows, not replacing them.
- tmalsburg2 4y ago"The best model was truthful on 58% of questions, while human performance was 94%." https://arxiv.org/abs/2109.07958 https://arxiv.org/abs/2109.07958
- sh4rks 4y agoI tried asking ChatGPT the knuckle cracking question that GPT-3 got wrong (what happens if you crack your knuckles a lot). The response was: The sound of cracking knuckles is actually a release of gas in the joint, but there is little scientific evidence to support the idea that it leads to arthritis or other joint problems. In fact, several studies have shown that there is no significant difference in the development of arthritis or other joint conditions between people who crack their knuckles and those who do not. I wonder if OpenAI used that paper to fine tune ChatGPT
- krona 4y agoWell, text-davinci-003 (ChatGPT) is an 'improved' version of the model at paper was based on (GPT-3). I don't think we know exactly what the improvement consists of.
- amitport 4y agoAlso works on Edge
- 2Gkashmiri 4y agofirefox extension dude
- ggm 4y agoDon't trust the output to reflect what's in the paper.
- tmalsburg2 4y agoIn my experience, it's almost safe to assume that the LLM summary will misrepresent the content in some way.
- Maken 4y agoDon't expect the paper to reflect its output either.
- amelius 4y agoEspecially if the paper is about LLMs.
- meghan_rain 4y agoWhat if the paper is longer than what fits in a prompt?
- voxelghost 4y agoYou can ask chatGPT about arxiv.org papers if you have the doi and the full title if the paper is from 2021 or earlier
- alexanderdavide 4y agoThis renders the extension pretty useless for older papers, doesn't it? There doesn't seem to be a fallback to prompt older papers in a whole because it exceeds the prompt limit.
- voxelghost 4y agoWell yes in a way, the extension settings include the prompt they're using, so you can formulate your own prompt along the same lines. I think the extension is only useful for someone who spends a lot of time doing research on arxiv.org, (and if the quality of the summary is good enough, the jury's still out on that one)
- fassssst 4y agobing.com/new doesn’t have a length restriction, and the Edge sidebar version can do it like this. Disclosure: I work at MSFT but not on Bing
- meghan_rain 4y agoIt must have some length restriction though, no? A GPT always has a max context window if I understand correctly. It's likely Bing has access to models with a much larger context window than what is accessibly through the OpenAI API.
- voxelghost 4y agoFor my paper, it just straight up invented most of the "relevant references" section
- iamflimflam1 4y agoWould love something that would translate papers from academic speech into something I can enjoy reading.
- voxelghost 4y agoThen https://www.explainpaper.com/ https://www.explainpaper.com/ is probably much more to your liking.
- herbst 4y agoIve used ChatGPT to make short factual videos for YouTube and honestly it's a bit worrying with supposed 'facts' I would not suggest anyone to use ChatGPT outputs for actual knowledge at this point.
- magicalhippo 4y agoI've not used ChatGPT yet (almost feel like a luddite), what happens when you tell it that it's wrong? Will it get it right after a retry or two?
- mepian 4y agoSometimes by accident it will, but most of the time it keeps making things up with no regard to factual and logical correctness. It has no notion of truth.
- aflag 4y agoYou can use it for that, but you have to test its claims. It's specially helpful with questions that would take a lot of research to find the answer, but which are easier to test. That could be a good thing overall. If people get used to always testing everything anyone tells them, people will be a lot better informed.
- visarga 4y agoI think LLM checking is going to be the most important research direction this year. Generative models are worthless without verification. It relates to deciding truth and dealing with fake information, synthetic text and spam. Google hasn't been able to solve it in the last decade, but some people have an idea: > "Discovering Latent Knowledge in Language Models Without Supervision" Existing techniques for training language models can be misaligned with the truth: if we train models with imitation learning, they may reproduce errors that humans make; if we train them to generate text that humans rate highly, they may output errors that human evaluators can't detect. We propose circumventing this issue by directly finding latent knowledge inside the internal activations of a language model in a purely unsupervised way. https://arxiv.org/abs/2212.03827 https://arxiv.org/abs/2212.03827 In other words the model already tries to predict the truth because it is useful in next token prediction, but we need to find a way to detect the 'truth alignment' in its activations.
- alexanderdavide 4y agoI'm wondering about two things related to development of customer-facing programs that use paid APIs in the background: Are you using your own API key and pay for the usage? How can you justify operation of programs that produce high costs but no income? Isn't the API publicly exposed to the client-side and possible subject of theft and abuse?
- juliangmp 4y agoMan if only the authors of such papers would write small summaries about the content and put them in the papers themselves or something
- throwaway37495 4y agoIf only the abstracts they write were more often than not accurate and not attention-grabbing problem-ignoring pieces of crap that do not reflect the content of the actual paper, hoping investors/newspapers won't actually read the paper. This is a conflict of incentives. Whereas ArxivGPT has no reason not to tell the problems first.
- Al-Khwarizmi 4y agoYou're right that abstracts often have a positive bias that can sometimes border on dishonesty, but for most papers it's not a matter of investors or newspapers, but rather of getting the paper accepted to the conference or journal, especially if it's a highly selective one. It happens even in fields where there is no direct industry interest.
- throwaway37495 4y agoConferences are often done just because it's a requirement to get grants/continue to be sponsored by an university. Investors are not just VC/for-profit industry, academia system also counts.
- krona 4y agoWouldn't that be breaking academic kayfabe?
- wonderfuly 4y agoGreat work! Glad to see innovations based on my work https://github.com/wong2/chatgpt-google-extension https://github.com/wong2/chatgpt-google-extension That's why I open sourced the code!
- dankle 4y agoIs it good?
- irthomasthomas 4y agoCaution. Language models do not know what is salient to a human. They also have a strong bias toward information that they have seen frequently. Research will contain a larger amount of new information and it's that new information which is most valuable to us, but least relevant according to the models.
- FartyMcFarter 4y agoThey're also known to be unreliable at simple logical inference: https://news.ycombinator.com/item?id=33859482 https://news.ycombinator.com/item?id=33859482
- JangoSteve 4y agoI think my favorite part of this prompt is that it starts with, "Please..." With this new class of products based on crafting prompts that best exploit a GPT's algorithm and training data, are we going to start seeing pull requests that tweak individual parts or words of the prompt. I'm also curious how the test suite for projects like this would look for specific facts or phrases to be contained in the responses for specific inputs.
- seaucre 4y agoThe related references don't appear to be real.
- inductive_magic 4y agoThis isn't viable because of bias and blatant lies in LLM-outputs. https://huggingface.co/ml6team/keyphrase-extraction-kbir-inspec https://huggingface.co/ml6team/keyphrase-extraction-kbir-ins... is a decent tool to explore the constant stream of publications. The last mile still is left to the human.
- m3kw9 4y agoThis is dangerous because people that has no knowledge of the science would blindly trust whatever it summarizes. There is no way to verify, an example is if you ask to summarize a book that you have understanding of the subject, at least you can sense some bs or open up the book at verify a few points. Here you would be at GPT3 mercy
- dekken_ 4y ago> This is dangerous because people that has no knowledge of the science would blindly trust whatever it summarizes. This says more about you than the hypothetical "people" you are talking about.
- m3kw9 4y agoRealy? The “You are projecting” replies. That’s cute
- dekken_ 4y agoYou're assuming, not projecting.
- kurisufag 4y agoI don't think so. The average person already thinks (Chat)GPT is an all-knowing AGI homework-solver, and the problem only worsens if you add the airs of "science" to the situation.
- dekken_ 4y agoI think we're in uncharted waters to some extent and should have some restraint about predictions.
- m3kw9 4y agoIf you have used the summary function of Gpt, it can be out right wrong but sound very plausible. With the amount of disinformation out there people that are interested in science but wants it the easy way is could make things worse. Imagine the summarized states the result show that certain meds give good results but without the right statistic context, it could be just marginally good, or even not statistically significant to the trained reader. Now they pass it off and start validating their own biases
- newswasboring 4y agoTo make it work in brave we need to turn off language based finger printing [1]. I wonder how's that related. Edit: btw, congratulations on the release. This is the kind of stuff I think should be explored more using LLMs. Great choice on making a chrome extension, it's great UI for this kind of thing. [1] https://github.com/hunkimForks/chatgpt-arxiv-extension#how-to-make-it-work-in-brave https://github.com/hunkimForks/chatgpt-arxiv-extension#how-t...
- f_allwein 4y agoI don’t understand. Don’t research papers usually have a summary, provided by the authors?