I’ve read OpenAI only used around 4000 high end GPUs ($30K each) for around 4 months to train GPT4. Yes there are some reinforcement tricks that are still uncle

I’ve read OpenAI only used around 4000 high end GPUs ($30K each) for around 4 months to train GPT4. Yes there are some reinforcement tricks that are still unclear but input data is basically known. I think people underestimate how many entities have the ability to easily acquire that amount of resources. It’s less than $200M to purchase that amount of hardware. Militaries spend $2-4M per missile. This is nothing for a lot of organizations. We already have self hosted LLM inference that reaches gpt 3.5 turbo level. Google and Microsoft are notoriously slow moving and probably less relevant than smaller faster operations like OpenAI.