I proposed it similarly in an earlier HN discussion and my understanding from that discussion is that it's typically not any better than having a monolothic mod

I proposed it similarly in an earlier HN discussion and my understanding from that discussion is that it's typically not any better than having a monolothic model. I'm not entirely convinced as I think it would also be easier to finetune or re-train smaller model modules instead of needing to train the entire model again.