2 ms·
Except it's unlikely because they have a bit larger context and are modeling a bit more than one-dimensional probabilities of "what comes after this?". I'd say
by TeMPOraL 3y ago
Except it's unlikely because they have a bit larger context and are modeling a bit more than one-dimensional probabilities of "what comes after this?".
I'd say that LLMs are more resistant to GIGO, as long as the fraction of garbage in training data is small - it'll look like outliers to the larger model.