3 ms·
> LLMs are good at predicting the next word in written language. They are generative; they make new text given a prompt. LLMs do not have base sets of facts abo
by xg15 2y ago
> LLMs are good at predicting the next word in written language. They are generative; they make new text given a prompt. LLMs do not have base sets of facts about how complex systems work, and do not attempt to reason over a corpus of evidence and facts.
Not wanting to advocate for using LLMs for RCA, which is a dumb and dangerous idea for all the reasons the OP mentioned - but I am getting allergic to the phrase "it just predicts the next word".
Yes, that is how the "API" of an LLM works, but on itself, it says nothing about how the LLM does the prediction and how complex the internal model is that it uses for that task.
It's obvious that the task of predicting has a huge variance in complexity, depending on which word has to be predicted.
E.g. the sentence "The apple does not fall far from the" could be completed by a decently trained Markov chain, whereas (correctly!) completing "sqrt(153847)=" would either require an impossibly large training set or an internal model that can parse integers and perform square root calculations.
Yet both are on the surface "predict the next word" tasks.
The actual complexity of LLMs' internal models seems to still be poorly understood. That's not to say the model has superhuman ability or even reaches human abilities. But the point is that it's still really hard to make predictions how complex the reasoning is that an LLM performs for a task.
In the OP's example, we can't really say if the LLM "has base sets of facts about how complex systems work", because we don't really know if and how "facts" would be represented inside the model. If enough in-domain examples were in the trainset, there is no fundamental reason why it wouldn't have learned such a set of facts.
Just saying "it can't reason at all because it's just a next word predictor" is mixing up different layers of meaning and, I believe, does not lead to more insight.