3 ms·

> Given how useful GPT-4 is already. Maybe one more iteration would unlock the vast majority of practical use cases. Unless I'm misunderstanding, doesn't OpenA

> Given how useful GPT-4 is already. Maybe one more iteration would unlock the vast majority of practical use cases. Unless I'm misunderstanding, doesn't OpenAI have a very vested interest to keep making their products so good/so complex/so large that consumer hobbyists can't just `git clone` an alternative that's 95% as good running locally?

Frannyies 3y ago
They have a huge cost incentive to optimize it for runtime. The magic of openai is their training data and architecture. There is a real risk that a model gets leaked.
- slt2021 3y ago
  it is not really a moat if one engineer can leave openai with all the secret sauce in his head and replicate it elsewhere (anthropic?)
  - foobiekr 3y ago
    Name one software-based tech company where this isn’t true.
    - slt2021 3y ago
      microsoft? gogel? FB?
  - Frannyies 3y ago
    I only meant the trained model. You would need to steal it all over again as soon as the next model is trained.
    - slt2021 3y ago
      no need to steal the model if training process can be reliably replicated/adopted in clean room implementation with additional optimisations. startup as legal entity has close to 0 value, most value is in intellectual property which is stored and transmitted by meatbags.
chongli 3y ago
What is OpenAI's moat? Loads of people outside the company are working on alternative models. They may have a lead right now but will it last a few years? Will it even last 6 months?
- yumraj 3y ago
  > What is OpenAI's moat? There’s none. Which is why Sam Altman has been crying wolf, in hope of regulatory barriers which can provide it the moat.
- MuffinFlavored 3y ago
  > What is OpenAI's moat? From what I understand, if you take the absolute best cutting edge LLM with the most parameters and the most up to date model from GitHub/HuggingFace/whatever, it's very far off from the output you get from GPT-3.5 / GPT-4 aka full of hallucinations, not very useful I don't know if this is the right way to look at it but if what George Hotz said about GPT-4 simply being "8 220B parameter models glued together by something called a mixture-of-experts", from what I understand, OpenAI's moat is: their access/subsidiized cost to GPUs/infrastructure with Microsoft the 8 220B models they have are really good/I don't think anything open source matches them/nobody can download "all of Reddit/Twitter/Wikipedia/StackOverflow/whatever else they trained on" anymore like they could given how everybody wants to protect/monetize their content now and then the "router" / "MoE" piece seems to be something missing from open source offerings as well
  - easygenes 3y ago
    Depending on the task, the best open models will outperform GPT-3.5, but would be more expensive to run at comparable speed. GPT-4 is in a league of its own.
  - passion__desire 3y ago
    What if specialized smaller models is the best way ahead for community. I don't care if I am interacting with one big model which can do everything or I have to go to different websites to access specific models. All model sizes will be useful. Smaller models will be frequently used. Bigger less so.
reckless 3y ago
Indeed they do, however companies like Meta (altruistically or not) are preventing OpenAI from building 'moats' by releasing models and architecture details in a very public way.
- foobiekr 3y ago
  Commoditize your complement strategies can just as likely put a market into a zombie state in the long run.
- runjake 3y ago
  I think it's a safe bet to say it's not altruistic. And, if Meta were to wrestle away OpenAI's moat, they'd eagerly create their own, given the opportunity.
  - passion__desire 3y ago
    Meta doesn't interact with its users in very obvious ways which MS, Google do. All its models magic happen behind the scenes. Meta can continue to release 2nd best models to undercut others and them going far too ahead. And Open Source community will take it from there. Dall-E is dead.
    - bugglebeetle 3y ago
      And if all open source extends their models, they can accrue those benefits back to themselves. This is already how they’ve become such a huge player in machine learning (open sourcing amazing stuff).
    - astrange 3y ago
      I think Dall-E isn't actually dead, but was merely renamed Bing Image Creator.
  - sangnoir 3y ago
    > And, if Meta were to wrestle away OpenAI's moat, they'd eagerly create their own Meta is already capable of monetizing content generated by the models: these models complement their business and they could not care less which model you're using to earn them advertising dollars, as long as you keep the (preferably high quality) content coming.

Frannyies 3y ago

They have a huge cost incentive to optimize it for runtime. The magic of openai is their training data and architecture. There is a real risk that a model gets leaked.

slt2021 3y ago
it is not really a moat if one engineer can leave openai with all the secret sauce in his head and replicate it elsewhere (anthropic?)
- foobiekr 3y ago
  Name one software-based tech company where this isn’t true.
  - slt2021 3y ago
    microsoft? gogel? FB?
- Frannyies 3y ago
  I only meant the trained model. You would need to steal it all over again as soon as the next model is trained.
  - slt2021 3y ago
    no need to steal the model if training process can be reliably replicated/adopted in clean room implementation with additional optimisations. startup as legal entity has close to 0 value, most value is in intellectual property which is stored and transmitted by meatbags.

chongli 3y ago

What is OpenAI's moat? Loads of people outside the company are working on alternative models. They may have a lead right now but will it last a few years? Will it even last 6 months?

yumraj 3y ago
> What is OpenAI's moat? There’s none. Which is why Sam Altman has been crying wolf, in hope of regulatory barriers which can provide it the moat.
MuffinFlavored 3y ago
> What is OpenAI's moat? From what I understand, if you take the absolute best cutting edge LLM with the most parameters and the most up to date model from GitHub/HuggingFace/whatever, it's very far off from the output you get from GPT-3.5 / GPT-4 aka full of hallucinations, not very useful I don't know if this is the right way to look at it but if what George Hotz said about GPT-4 simply being "8 220B parameter models glued together by something called a mixture-of-experts", from what I understand, OpenAI's moat is: their access/subsidiized cost to GPUs/infrastructure with Microsoft the 8 220B models they have are really good/I don't think anything open source matches them/nobody can download "all of Reddit/Twitter/Wikipedia/StackOverflow/whatever else they trained on" anymore like they could given how everybody wants to protect/monetize their content now and then the "router" / "MoE" piece seems to be something missing from open source offerings as well
- easygenes 3y ago
  Depending on the task, the best open models will outperform GPT-3.5, but would be more expensive to run at comparable speed. GPT-4 is in a league of its own.
- passion__desire 3y ago
  What if specialized smaller models is the best way ahead for community. I don't care if I am interacting with one big model which can do everything or I have to go to different websites to access specific models. All model sizes will be useful. Smaller models will be frequently used. Bigger less so.

reckless 3y ago

Indeed they do, however companies like Meta (altruistically or not) are preventing OpenAI from building 'moats' by releasing models and architecture details in a very public way.

foobiekr 3y ago
Commoditize your complement strategies can just as likely put a market into a zombie state in the long run.
runjake 3y ago
I think it's a safe bet to say it's not altruistic. And, if Meta were to wrestle away OpenAI's moat, they'd eagerly create their own, given the opportunity.
- passion__desire 3y ago
  Meta doesn't interact with its users in very obvious ways which MS, Google do. All its models magic happen behind the scenes. Meta can continue to release 2nd best models to undercut others and them going far too ahead. And Open Source community will take it from there. Dall-E is dead.
  - bugglebeetle 3y ago
    And if all open source extends their models, they can accrue those benefits back to themselves. This is already how they’ve become such a huge player in machine learning (open sourcing amazing stuff).
  - astrange 3y ago
    I think Dall-E isn't actually dead, but was merely renamed Bing Image Creator.
- sangnoir 3y ago
  > And, if Meta were to wrestle away OpenAI's moat, they'd eagerly create their own Meta is already capable of monetizing content generated by the models: these models complement their business and they could not care less which model you're using to earn them advertising dollars, as long as you keep the (preferably high quality) content coming.