I think one of the biggest things missing from this point of view is mixture of experts. GPT-$VER does very well across a wide range of tasks because of the mi

I think one of the biggest things missing from this point of view is mixture of experts. GPT-$VER does very well across a wide range of tasks because of the mixture of experts approach. A "small" (relatively) startup implementing/deploying AI in the space targeting a specific use case (as one example) is often just finetuning a model on their dataset (in itself a moat). Given the amount of data and scale of compute for this task (relative to MAGMA) it can be done on VRAM limited cards or as we've seen from open source finetunes something like a cloud A100x8 for a few hours at a time. Many a startup has been successful targeting niches the juggernauts just don't care about (or understand) - what's insignificant and just not worth it to them is massive for anyone else. Sure these probably won't end up as unicorns (there's a reason they're called that) but many of us are perfectly happy with valuations in the seven-nine figure range or running a real business that is actually profitable - which is most successful startups. Or, for internal org use you can do something like "finetune Codellama on our entire codebase", or "finetune Llama on all of our data" which few would be willing to do with an OpenAI/Google finetune as it requires sending your entire codebase to them. An agreement or promise they won't use it isn't comforting enough to many of these orgs, especially in niches with significant regulatory issues.