Its possible that the knowledge was never lost but covered up. If we imagine the neural net as code. As in the weights are the source, the fine tuning may effe

Its possible that the knowledge was never lost but covered up. If we imagine the neural net as code. As in the weights are the source, the fine tuning may effectively hack that code to not return certain things. Infact that is kinda what fine tuning is. Therefore you may have just built a firewall around certain outputs. But quantizing could make those recent edits disappear. They are too subtle to survive. Whereas quantizing doesn't destroy all knowledge as evidenced by popular quantized models. Also: @simonw incase he has alerts. Would be perfect topic for him to write up.