3 ms·

It will analyse what it is given, but it will not have the ability to say, "hang on, these results are interesting, I wonder what will happed if I pour a differ

It will analyse what it is given, but it will not have the ability to say, "hang on, these results are interesting, I wonder what will happed if I pour a different liquid into the drum and spin it at the same speed?" LLMs, especially the latest ones, are decent at analysis of input, but disappointing at producing creative output. I am running a series of experiments using Gemini 1.5 and found it capable of producing good results if you stay away from "write me an academic paper on subject X" or "write me a novel". On the other hand, if you ask it to summarise text, extract particular information, it is fast and arguably good, but not necessarily great. It will miss things and miscategorise them requiring a human being to check its output. At this point, you may just as well do the job yourself. LLMs are still not very good, despite what their fans are saying. They are clever, as in a "clever trick" not as in a "clever human being".

MattJ100 2y ago
Interestingly your comment is the very opposite of my experience with LLMs. You can rely on them for creative stuff (write a poem, short story, etc.), but you cannot depend on them for factual stuff. Time and time again they will state "facts" that turn out to be false, and so now I no longer trust it for anything without manual verification. And since I then need to do the research myself for the verification, I rarely find LLMs helpful, except occasionally for initial exploration of some topic. You used summarization as an example, but whether they are fundamentally good at that is even debatable, e.g. https://ea.rna.nl/2024/05/27/when-chatgpt-summarises-it-actually-does-nothing-of-the-kind/ https://ea.rna.nl/2024/05/27/when-chatgpt-summarises-it-actu...
- surfingdino 2y ago
  I revisit LLMs a couple of times a year to see if they have gotten any better and it's not a great experience, but I will admit they have gotten slightly better. Still lack and will lack ability to understand what they are processing.
viraptor 2y ago
Your experience contradicts the data from the paper. Sonnet generated almost entirely novel concepts, GPT achieved ~4/5 novel ideas. Maybe it's specific to the area of research and the way of prompting, but "it will not have the ability to say" seems to be proven wrong already.

MattJ100 2y ago

Interestingly your comment is the very opposite of my experience with LLMs. You can rely on them for creative stuff (write a poem, short story, etc.), but you cannot depend on them for factual stuff. Time and time again they will state "facts" that turn out to be false, and so now I no longer trust it for anything without manual verification. And since I then need to do the research myself for the verification, I rarely find LLMs helpful, except occasionally for initial exploration of some topic. You used summarization as an example, but whether they are fundamentally good at that is even debatable, e.g. https://ea.rna.nl/2024/05/27/when-chatgpt-summarises-it-actually-does-nothing-of-the-kind/ https://ea.rna.nl/2024/05/27/when-chatgpt-summarises-it-actu...

surfingdino 2y ago
I revisit LLMs a couple of times a year to see if they have gotten any better and it's not a great experience, but I will admit they have gotten slightly better. Still lack and will lack ability to understand what they are processing.

viraptor 2y ago

Your experience contradicts the data from the paper. Sonnet generated almost entirely novel concepts, GPT achieved ~4/5 novel ideas. Maybe it's specific to the area of research and the way of prompting, but "it will not have the ability to say" seems to be proven wrong already.