4 ms·
I wonder if there has been any study of what happens to LLMs subject to bit rot, given the level of compression of facts on one hand and the “superposition of s
by harperlee 2y ago
I wonder if there has been any study of what happens to LLMs subject to bit rot, given the level of compression of facts on one hand and the “superposition of stored facts” on the other one ( I’m obviously a layman here, I don’t even have the correct vocabulary)
- sdenton4 2y agoLots of models are trained with dropout, which is kinda like bitrot at very high rates...
- clayhacks 2y agoBut that’s during training, not during inference. Also it’s more structured in where the dropout is happening. I do think that points to them being somewhat resilient but we haven’t had LLMs exist long enough for a good test