4 ms·
Is it the fundamental architecture of an LLM that makes this problematic, or the input? Based on how it's reported that they work, I assume that even without g
by anonymouskimmer 3y ago
Is it the fundamental architecture of an LLM that makes this problematic, or the input?
Based on how it's reported that they work, I assume that even without garbage in their statistical reasoning of the next word would occasionally come up with bad output. E.g. they might come up with 2 + 2 = 5 because of mathematical inputs such as "1 + 2 + 2 = 5".
- TeMPOraL 3y agoExcept it's unlikely because they have a bit larger context and are modeling a bit more than one-dimensional probabilities of "what comes after this?". I'd say that LLMs are more resistant to GIGO, as long as the fraction of garbage in training data is small - it'll look like outliers to the larger model.