Splitting knowledge out doesn't mean you need to drop weights. Just that it needs to become independent / attachable. To some extent we've already seen this wi

Splitting knowledge out doesn't mean you need to drop weights. Just that it needs to become independent / attachable. To some extent we've already seen this with MoE and Frankenstein models.