I am not an expert or even particularly knowledgeable about this, so treat the following as “armchair speculation by an enthusiast”. Wikipedia says that as of

I am not an expert or even particularly knowledgeable about this, so treat the following as “armchair speculation by an enthusiast”. Wikipedia says that as of 2023, each expert is ~10 billion parameters, so they’re still much smaller than something like Deepseek Coder 70B. I’m not sure why they’re so small though. I don’t know whether there’s some kind of architectural issue or super linear scaling somewhere preventing them from growing, or if they’re just trying to keep inference costs down.