4 ms·

> Embeddings can be considered to be the thoughts of an LLM. That's a very unhelpful metaphor in a sea of misleading narrative. That narrative started the mome

> Embeddings can be considered to be the thoughts of an LLM. That's a very unhelpful metaphor in a sea of misleading narrative. That narrative started the moment we called it "an AI". Anthropomorphization chooses the words we use next. Thoughts, limitations, hallucinations, etc. all belong to thinkers. If we want to prove that an LLM is indeed a thinker, we must be careful to avoid these words, otherwise the narrative itself is a circular proposition: true because you said so. > LLMs are ‘just a next-token predictor’. The only problem with this phrasing is the implication of the use of the word ‘just’. I disagree. I think the problem word is "next". The structure of an LLM's model is not one dimensional, and the path of a continuation function is not linear. The LLM is also not alone in this story: there's also the original human-written text it was made from, and the human-written text it is prompted with. Those had to be given to the LLM. > Let’s consider a hypothetical situation...an API that performs next-token prediction...Ultimately, the best solution would be to place humans behind the API... Before I criticize this hypothetical, let's run with it and see where it takes us. You write a prompt, and send it to the large-language-group. The prompt gets divided into words, and each word gets assigned to a human. Each human presents a list of words they think are the most likely to come next, along with how confident they are in those predictions, and the system sorts all the responses into a continuation. What isn't happening in that story? Objectivity. The humans aren't thinking about what your prompt means. There is no categorization. There is no logic. Most of what comes to mind when I say "they are thinking" just isn't there. > there is no way to tell what process is behind it, similar to modern LLMs. Except that you did just explicitly answer that question. It's humans. The thing "behind" an LLM is human-written text. That doesn't put the LLM itself in the same category as human. > The humans behind the API possess real intelligence, and by extension, so does the API. Therefore, the intelligence implication falls apart, as something being a next-token predictor doesn’t preclude it from possessing real intelligence. That's all very nearly accurate, but it doesn't help the overall argument. All we are saying is that real intelligence is present, and that the LLM extends that intelligence. What we absolutely failed to prove is that the LLM is intelligent or that it does intelligence. Reason is a verb and a noun. An LLM does not do reason. An LLM does not contain reason. An LLM only presents reason.

Hugsun 2y ago
> the narrative itself is a circular proposition I only use the simile to make the text more approachable to leymen. I don't ever conclude that because embeddings are like thoughts, thoughts are like embeddings---or LLMs are intelligent because they have thoughts in the form of embeddings. That is not a rational inference to make IMO. I don't think we should avoid similes and metaphors because of the risk of this misunderstanding. A is like B does not mean A is the same as B. > I disagree. I think the problem word is "next". The structure of an LLM's model is not one dimensional, and the path of a continuation function is not linear. The LLM is also not alone in this story: there's also the original human-written text it was made from, and the human-written text it is prompted with. Those had to be given to the LLM. You can do many things with LLMs but when they are run during inference (like the commonly known case of ChatGPT), they are predicting the next token, so calling them next-token predictors is fine. The LLM is predicting the next token after the human-written text it is prompted with. Nowhere is it implied that that is not a part of the story. > The prompt gets divided into words, and each word gets assigned to a human. The article states that the API should be as human-like as possible. They would thus certainly not give each person a single word. They would give each prompt to a single person so they had the entire context. The divvying is amongst different prompts, not words. > The humans aren't thinking about what your prompt means. The point of the hypothetical is that to make this LLM-like API, you have to employ humans, and you have to have them think about the output. To make it as human-like as possible, you can just pay them to think about it. > Except that you did just explicitly answer that question. A person using the API is not able to tell if there is a human behind it, or an LLM. That is the point. > All we are saying is that real intelligence is present, and that the LLM extends that intelligence. There is no LLM in the hypothetical you are referring to. It happens in the year 2000. > What we absolutely failed to prove is that the LLM is intelligent or that it does intelligence. The segment you are referring to is just making the argument that a next-token predictor can be intelligent. Not that all next-token predictors are intelligent, or are the same. If you read this segment: https://www.arnaldur.be/writing/about/large-language-model-reasoning#our-reasoning-does-not-occur-in-a-vacuum https://www.arnaldur.be/writing/about/large-language-model-r... You will see the arguments for LLMs being able to reason, why they in some cases seem unable to, and how it should be expected in some cases.

Hugsun 2y ago

> the narrative itself is a circular proposition I only use the simile to make the text more approachable to leymen. I don't ever conclude that because embeddings are like thoughts, thoughts are like embeddings---or LLMs are intelligent because they have thoughts in the form of embeddings. That is not a rational inference to make IMO. I don't think we should avoid similes and metaphors because of the risk of this misunderstanding. A is like B does not mean A is the same as B. > I disagree. I think the problem word is "next". The structure of an LLM's model is not one dimensional, and the path of a continuation function is not linear. The LLM is also not alone in this story: there's also the original human-written text it was made from, and the human-written text it is prompted with. Those had to be given to the LLM. You can do many things with LLMs but when they are run during inference (like the commonly known case of ChatGPT), they are predicting the next token, so calling them next-token predictors is fine. The LLM is predicting the next token after the human-written text it is prompted with. Nowhere is it implied that that is not a part of the story. > The prompt gets divided into words, and each word gets assigned to a human. The article states that the API should be as human-like as possible. They would thus certainly not give each person a single word. They would give each prompt to a single person so they had the entire context. The divvying is amongst different prompts, not words. > The humans aren't thinking about what your prompt means. The point of the hypothetical is that to make this LLM-like API, you have to employ humans, and you have to have them think about the output. To make it as human-like as possible, you can just pay them to think about it. > Except that you did just explicitly answer that question. A person using the API is not able to tell if there is a human behind it, or an LLM. That is the point. > All we are saying is that real intelligence is present, and that the LLM extends that intelligence. There is no LLM in the hypothetical you are referring to. It happens in the year 2000. > What we absolutely failed to prove is that the LLM is intelligent or that it does intelligence. The segment you are referring to is just making the argument that a next-token predictor can be intelligent. Not that all next-token predictors are intelligent, or are the same. If you read this segment: https://www.arnaldur.be/writing/about/large-language-model-reasoning#our-reasoning-does-not-occur-in-a-vacuum https://www.arnaldur.be/writing/about/large-language-model-r... You will see the arguments for LLMs being able to reason, why they in some cases seem unable to, and how it should be expected in some cases.