4 ms·

Ask HN: Is it hopeless to try to build new foundational models now

1 points by nutanc 3y ago

Sama says in an answer to a question(What should Indian startups do to build foundational models?) that it is hopeless to try to build foundational models that can compete with OpenAI. What does the community think? Is OpenAI too far ahead? Is there a possibility of some new tech coming which can displace OpenAI? We are betting on the fact that such new tech is possible. Just wanted to get the feel of the community in this regard.

nutanc 3y ago
https://twitter.com/RajanAnandan/status/1666641010284449792?s=20 https://twitter.com/RajanAnandan/status/1666641010284449792?...
smoldesu 3y ago
It's not hopeless, but it requires so much money and time that you might as well surrender all hope. Competing with OpenAI is tough, finetuning llama to a point that it competed with GPT3.5 required several finetuning attempts and many months of research. There's still room for more AI models, but competitive ones are remarkably difficult to build. Competing with OpenAI's thousands of GPUs is not an easy feat.
- nutanc 3y ago
  What if we have an alternative to deep learning that does not need those thousands of GPUs?
  - smoldesu 3y ago
    Then it would have to compete with OpenAI's solution, which is no easy feat. I'd be interested in hearing more about this alternative, though.
    - nutanc 3y ago
      Have written a little about it here, https://gpt3experiments.substack.com/p/building-a-new-embeddingbit-vector https://gpt3experiments.substack.com/p/building-a-new-embedd... Will do a bigger blog post soon on the approach. Can mail you some details if you are interested.
raxxorraxor 3y ago
I think it is very much possible, especially in economies like India and China. If some business cases for Chat GPT are realized, I expect competitors to surface. The a difficult part is probably getting high quality training data. And there may very well be a barrier where improvement of the model stops and completely new strategies must be devised. Sure, you would need significant investments to start such a venture and success is not guaranteed. It is nothing a garage shop could do.

1 points by nutanc 3y ago

nutanc 3y ago

https://twitter.com/RajanAnandan/status/1666641010284449792?s=20 https://twitter.com/RajanAnandan/status/1666641010284449792?...

smoldesu 3y ago

It's not hopeless, but it requires so much money and time that you might as well surrender all hope. Competing with OpenAI is tough, finetuning llama to a point that it competed with GPT3.5 required several finetuning attempts and many months of research. There's still room for more AI models, but competitive ones are remarkably difficult to build. Competing with OpenAI's thousands of GPUs is not an easy feat.

nutanc 3y ago
What if we have an alternative to deep learning that does not need those thousands of GPUs?
- smoldesu 3y ago
  Then it would have to compete with OpenAI's solution, which is no easy feat. I'd be interested in hearing more about this alternative, though.
  - nutanc 3y ago
    Have written a little about it here, https://gpt3experiments.substack.com/p/building-a-new-embeddingbit-vector https://gpt3experiments.substack.com/p/building-a-new-embedd... Will do a bigger blog post soon on the approach. Can mail you some details if you are interested.

raxxorraxor 3y ago

I think it is very much possible, especially in economies like India and China. If some business cases for Chat GPT are realized, I expect competitors to surface. The a difficult part is probably getting high quality training data. And there may very well be a barrier where improvement of the model stops and completely new strategies must be devised. Sure, you would need significant investments to start such a venture and success is not guaranteed. It is nothing a garage shop could do.