I know next to zero material science, but I would think there's not much motion going on, and that it's more that once you're past the yield strength, stress ca

I know next to zero material science, but I would think there's not much motion going on, and that it's more that once you're past the yield strength, stress causes nonlinear increases in strain, and you get hysteresis (i.e. permanent deformations). Then over time those deformations add up to a weaker structure that eventually fractures. As long as you stay within or close to the linear regime, there's little to no permanent damage. I imagine it's more of a chemical process (i.e. you're damaging the bonds/lattice structure of a material), but I don't know.