3 ms·

You're massively overestimating the skills needed to build "competitive models". Researchers like Ilya were essential several years ago when OpenAI got started,

You're massively overestimating the skills needed to build "competitive models". Researchers like Ilya were essential several years ago when OpenAI got started, because at that time nobody even knew what to build or how to go about it. It was all research, and OpenAI was solely focused on RL. But these days, LLM code is 95%+ data and infrastructure engineering and very little novel research. Even things like LLM speedups via new attention mechanisms are more engineering than they are research. Above all else, what you need to build competitive models is a huge amount of money and access to compute. Money alone doesn't solve that either due to the global hardware shortage. You are competing for a globally limited pool of hardware with all other companies.

p1esk 3y ago
Google and Meta have more money and more compute than OpenAI. They have more data, better infrastructure, and more AI researchers. And they have been trying hard to catch up - so far unsuccessfully.
- fidotron 3y ago
  But they have the key problem of all established companies: they built the infrastructure for what is now the wrong thing. This means to do the right thing at Google would require fighting battles on all sorts of fronts as established mini empires within attempt to latch on to this new perceived opportunity to expand.
  - p1esk 3y ago
    to do the right thing at Google would require fighting battles Determining what the right thing is the hard part. Unfortunately that’s also the case right now at OpenAI.
- loveparade 3y ago
  Google and Meta have less compute infrastructure purpose-built for training and testing LLMs than OpenAI does. It's something that OpenAI has been investing in for several years, while Google/Meta/etc have just started playing catch up very recently.
varjag 3y ago
Researchers like Ilya were essential several years ago when OpenAI got started, because at that time nobody even knew what to build or how to go about it. Nah Sam will just use GPT4 to code his vision. That's how it works now, we've been told.

p1esk 3y ago

Google and Meta have more money and more compute than OpenAI. They have more data, better infrastructure, and more AI researchers. And they have been trying hard to catch up - so far unsuccessfully.

fidotron 3y ago
But they have the key problem of all established companies: they built the infrastructure for what is now the wrong thing. This means to do the right thing at Google would require fighting battles on all sorts of fronts as established mini empires within attempt to latch on to this new perceived opportunity to expand.
- p1esk 3y ago
  to do the right thing at Google would require fighting battles Determining what the right thing is the hard part. Unfortunately that’s also the case right now at OpenAI.
loveparade 3y ago
Google and Meta have less compute infrastructure purpose-built for training and testing LLMs than OpenAI does. It's something that OpenAI has been investing in for several years, while Google/Meta/etc have just started playing catch up very recently.

varjag 3y ago

Researchers like Ilya were essential several years ago when OpenAI got started, because at that time nobody even knew what to build or how to go about it. Nah Sam will just use GPT4 to code his vision. That's how it works now, we've been told.