For GPT-3, I think they mainly scraped stuff off the internet. Anyone with enough time and machines can replicate that, but it is a non zero effort. GPT is an e

For GPT-3, I think they mainly scraped stuff off the internet. Anyone with enough time and machines can replicate that, but it is a non zero effort. GPT is an exception here I think. It's harder to build open source MRI diagnosis tools for example when nobody gives you open source MRI scans with diagnosis data. It's more a lack of compute power problem plus a lack of people problem. The compute cost alone was estimated to be 4.6 million USD. As for the people, folks who can push SOTA are extremely sought after. So the individuals involved also pay a large opportunity cost by working on things for free or comparatively small open source pay. Also, I think many open source organizations/communities haven't recognized the value of open source ML models yet and don't invest in them as much as in parametric models. I think this will come with time. Last, anyone with a computer can write code, while few people have access to the hardware needed.