2 ms·

Every time somebody posts a negative result on LLMs' capabilities like that, somebody will go on HN, on Twitter, on Reddit, etc, and post an example where they

Every time somebody posts a negative result on LLMs' capabilities like that, somebody will go on HN, on Twitter, on Reddit, etc, and post an example where they get an LLM to produce a counter example of the negative result. Crucially, this follow-up "rebuttal" always seems to use a different prompt than the one in the original experiment. What can we conclude from that kind of sequence? We can conclude that neither the original assertion, that "LLMs can't do X", nor the rebuttal, "They can if you tweak the prompt", are really telling us anything about the capabilities of LLMs to do what their users ask them to do: they are only telling us something about the capability of a user to get an LLM to do what the user wants. In other words, we can see LLMs as random-access memories with imperfect recall. Much like a SQL query to a relational database, the right prompt can access the right piece of data, but unlike SQL nobody has a clue what "the right prompt" is in the general case. Seen another way, any prompt a user makes to an LLM has some probability to elicit the desired response from the LLM. There is no known way to maximise that probability. Until there is, we cannot draw any conclusions about the capabilities of LLMs just by poking them and checking the results. To be very clear about it: we can't conclude either that "LLMs can't do X", nor that "LLMs can do X'. All we can conclude is "a user can do X".

version_five 3y ago
Along the same lines, I read something (on here but I don't think I can find it) suggesting that llms + prompting are kind of overfitting, where if you know what you achieve, you can try few things and get it - which would be equivalent to optimizing on a train set. Whereas if you try to get it to do something you don't already know how to do, all bets are off.
lsy 3y ago
I agree and I think the reason it's sort of an indictment of LLMs is because there's no real way to guarantee performance over a range of input, which makes it unsuitable for many automation tasks. If you develop an LLM prompt that says "Add <user input> to 5" and prove that it works for a handful of cases, there's no guarantee it works for any other case. Whereas "(lambda (x) (+ x 5))" has some mechanical guarantee based on the implementation. And for the LLM, the negative result becomes more possible the higher the complexity of the prompt. This reduces LLMs to being suitable for tasks where reliability is not particularly important, or a supervisory person or system is checking for consistency and somehow can re-run the LLM to generate new results (which itself is not always reliable).

version_five 3y ago

Along the same lines, I read something (on here but I don't think I can find it) suggesting that llms + prompting are kind of overfitting, where if you know what you achieve, you can try few things and get it - which would be equivalent to optimizing on a train set. Whereas if you try to get it to do something you don't already know how to do, all bets are off.

lsy 3y ago

I agree and I think the reason it's sort of an indictment of LLMs is because there's no real way to guarantee performance over a range of input, which makes it unsuitable for many automation tasks. If you develop an LLM prompt that says "Add <user input> to 5" and prove that it works for a handful of cases, there's no guarantee it works for any other case. Whereas "(lambda (x) (+ x 5))" has some mechanical guarantee based on the implementation. And for the LLM, the negative result becomes more possible the higher the complexity of the prompt. This reduces LLMs to being suitable for tasks where reliability is not particularly important, or a supervisory person or system is checking for consistency and somehow can re-run the LLM to generate new results (which itself is not always reliable).