I’ve been thinking on this too, the real driver is just that it’s hard to scale models to that size, I think with the MoE work happening like GLAM it will hopef

I’ve been thinking on this too, the real driver is just that it’s hard to scale models to that size, I think with the MoE work happening like GLAM it will hopefully be easier to scale models in a distributed fashion.