4 ms·

> Why should I think that LLMs would be good at the task of analyzing a cloud incident and determining root cause? LLMs are good at predicting the next word in

by roenxi 2y ago

> Why should I think that LLMs would be good at the task of analyzing a cloud incident and determining root cause? LLMs are good at predicting the next word in written language. They are generative; they make new text given a prompt. Now explain what humans are doing and why it is different. RCA can be done fully remotely, so we know it can be modelled as a word-prediction model with context and a "how is all this context made consistent?" prompt. There is every reason to expect that an advanced text prediction model would be excellent at predicting the most-technically-correct response to that prompt. I've seen a few people make this argument as though text prediction is some specific sub-field that can be solved independently of having a world model and intelligence. That doesn't hold up at all, if we solve text prediction we've solved general intelligence - all aspects of human intelligence are less complex than being able to predict the most objectively correct next word in a sequence of words because all aspects of intelligence can be framed as a text-prediction problem. A system can't predict the next word in a sequence and fool a human without being at least as clever as a human.

crystal_revenge 2y ago
You don't even have to get deep into the internals of LLMs to see what's wrong with your reasoning. The problem lies with the basic mechanics of: > predict the most objectively correct next word in a sequence of words Currently all LLMs are only determining the most probable next token, but this means they are not aware of the probability of the entire sequence of tokens they are emitting. That is, they can only build sentences by picking the most probable next word, but can never choose the most probable sentence. In practice, there are a great many very likely sentences that are composed of a fairly unlikely words. When we use the output of an LLM we're thinking it of a sequence sampled from the set of all possible sequences, but that's not really what we're getting (at least as far as probability is concerned). There are approaches to address this: you can do multinomial sampling instead of greedy so that are casting a slightly large net or you can do beam search where your once again trying to search a broader set of possible sentences choosing by the most probable sequence. But all of these are fairly limited. Which gets to your first remark: > Now explain what humans are doing and why it is different. There's very little we really know about how humans reason, but we are certainly building linguistic expressions at with a more abstract form of composition. This comment for example was planned out in parts, not even sequentially, and then reworked to the whole thing makes some sense. But at the very least humans are clearly reasoning at the level of entire sequences as their probability rather than individual tokens at a time. The word "planning" almost tautologically implies thinking ahead of the next step. When humans write HN comments or code they're clearly planning rather than just thinking of the next most likely word over and over again with some noise to make it sound more interesting. No matter how powerful and sophisticated the mathematical models driving the core of LLMs are, we're fundamentally limited by the methods we use to sample from them.
- roenxi 2y ago
  That doesn't hold together. You seem to be arguing that LLMs produce text as a sequence of words. Which, fair enough, they obviously do. But then your argument seems to drift into humans not producing text as a series of words. I'm not sure how you type your comments but you should upload a YouTube video of it as it sounds like it'd be quite a spectacle! If your argument is that LLMs can't reason because they don't edit their comments, it'd be worth stopping and reflecting for a few moments about how weak a position that is. I wrote this comment linearly just to make a point with no editing except spellchecking.
  - pfsalter 2y ago
    Humans don't really generate text as a series of words. If you've ever known what you wanted to say but not been able to remember the word you can see this in practice. Although the analogy is probably a helpful one, LLMs are basically doing the word remembering bit of language, without any of the thought behind it.
    - deleted 2y ago
      [deleted]
    - roenxi 2y ago
      How do you generate your text? Do you write the middle of the sentence first, come back to the start then finish it? Or do you have a special keyboard where you drop sentences as fully formed input? As systems humans and LLMs behave in observably similar ways. You feed in some sort of prompt+context, there is a little bit of thinking done, a response is developed by some wildly black-box method, and then a series of words are generated as output. The major difference is that the black boxes presumably work differently but since they are both black boxes that doesn't matter much for which will do a better job at root cause analysis. People seem to go a bit crazy on this topic at the idea that complex systems can be built from primitives. Just because the LLM primitives are simple doesn't mean the overall model isn't capable of complex responses.
      - 2y ago
zahlman 2y ago
>the most objectively correct next word in a sequence of words Why should I believe, in the first place, that this is even a coherent concept?

by roenxi 2y ago

crystal_revenge 2y ago

You don't even have to get deep into the internals of LLMs to see what's wrong with your reasoning. The problem lies with the basic mechanics of: > predict the most objectively correct next word in a sequence of words Currently all LLMs are only determining the most probable next token, but this means they are not aware of the probability of the entire sequence of tokens they are emitting. That is, they can only build sentences by picking the most probable next word, but can never choose the most probable sentence. In practice, there are a great many very likely sentences that are composed of a fairly unlikely words. When we use the output of an LLM we're thinking it of a sequence sampled from the set of all possible sequences, but that's not really what we're getting (at least as far as probability is concerned). There are approaches to address this: you can do multinomial sampling instead of greedy so that are casting a slightly large net or you can do beam search where your once again trying to search a broader set of possible sentences choosing by the most probable sequence. But all of these are fairly limited. Which gets to your first remark: > Now explain what humans are doing and why it is different. There's very little we really know about how humans reason, but we are certainly building linguistic expressions at with a more abstract form of composition. This comment for example was planned out in parts, not even sequentially, and then reworked to the whole thing makes some sense. But at the very least humans are clearly reasoning at the level of entire sequences as their probability rather than individual tokens at a time. The word "planning" almost tautologically implies thinking ahead of the next step. When humans write HN comments or code they're clearly planning rather than just thinking of the next most likely word over and over again with some noise to make it sound more interesting. No matter how powerful and sophisticated the mathematical models driving the core of LLMs are, we're fundamentally limited by the methods we use to sample from them.

roenxi 2y ago
That doesn't hold together. You seem to be arguing that LLMs produce text as a sequence of words. Which, fair enough, they obviously do. But then your argument seems to drift into humans not producing text as a series of words. I'm not sure how you type your comments but you should upload a YouTube video of it as it sounds like it'd be quite a spectacle! If your argument is that LLMs can't reason because they don't edit their comments, it'd be worth stopping and reflecting for a few moments about how weak a position that is. I wrote this comment linearly just to make a point with no editing except spellchecking.
- pfsalter 2y ago
  Humans don't really generate text as a series of words. If you've ever known what you wanted to say but not been able to remember the word you can see this in practice. Although the analogy is probably a helpful one, LLMs are basically doing the word remembering bit of language, without any of the thought behind it.
  - deleted 2y ago
    [deleted]
  - roenxi 2y ago
    How do you generate your text? Do you write the middle of the sentence first, come back to the start then finish it? Or do you have a special keyboard where you drop sentences as fully formed input? As systems humans and LLMs behave in observably similar ways. You feed in some sort of prompt+context, there is a little bit of thinking done, a response is developed by some wildly black-box method, and then a series of words are generated as output. The major difference is that the black boxes presumably work differently but since they are both black boxes that doesn't matter much for which will do a better job at root cause analysis. People seem to go a bit crazy on this topic at the idea that complex systems can be built from primitives. Just because the LLM primitives are simple doesn't mean the overall model isn't capable of complex responses.
    - 2y ago

zahlman 2y ago

>the most objectively correct next word in a sequence of words Why should I believe, in the first place, that this is even a coherent concept?