3 ms·
Cool I guess but not really surprising given previous work on mode connectivity ((1) and related) (1) https://arxiv.org/abs/1802.10026 https://arxiv.org/abs/18
by blackbear_ 4y ago
Cool I guess but not really surprising given previous work on mode connectivity ((1) and related)
(1) https://arxiv.org/abs/1802.10026 https://arxiv.org/abs/1802.10026
- version_five 4y agoRelated is this one https://arxiv.org/abs/1912.02757 https://arxiv.org/abs/1912.02757 which as I understand finds that there's little to be gained moving around locally in weight space, vs finding another mode. So it's not surprising that a model positioned between two modes would function sort of like an ensemble model
- xpe 4y agoWhich of course raises the question of quantifying when averaging models (or, more generally, staying within a convex hull) is likely to remain in the same low cost (high performing) basin.
- LChoshen 4y agoAll related, but why do any of it changes things about fusing models of "different" training, which is more interesting imo