"Yep, it would definitely be difficult to justify running it in production. Accuracy would need to be higher as you said or it would need to be applicable to mo

"Yep, it would definitely be difficult to justify running it in production. Accuracy would need to be higher as you said or it would need to be applicable to more tasks such that you can take other models out of production." Part of the justification is the MoE sparsity by design means only a small part of the model will be activated by a given query. Don't think of it as a single giant 1t model, think of it as 50 small models which happen to share a glue layer at the input. So at deployment, you could, for example, keep the gating-layer in RAM and only pull the necessary sub-model off disk as necessary. Or you could shard the sub-models over 51 GPUs and feed the master gating-layer+GPU lots of queries, and each query will be dispatched to a different expert+GPU pair. This could easily be competitive with running a lot of dense models in parallel trying to keep up with the same load.