4 ms·

Where's the moat for OpenAI? How does the company not end up as just a feature stitched into Azure somewhere?

JofArnold 3y ago
In an exponential game where the prize is ASI, being just a couple of steps ahead may turn out to be enough.
spangry 3y ago
Training & individual user preference data - specifically the data being generated by people interacting with ChatGPT and DALLE. Maybe Google can out data them with all the data they currently hold, but they'd better hurry before the network effects get too strong in favour of OpenAI.
danielmarkbruce 3y ago
Try training and running really really big models.
- objektif 3y ago
  You mean now. Commodity soon.
  - empath-nirvana 3y ago
    Because of scaling advantages, it likely never will be. Even if you could run a GPT4 quality model on your laptop (you can't), by the time you can, Open Ai will be running a model that uses orders of magnitude more compute.
    - jjjjj55555 3y ago
      Stupid question, but why aren't they doing it already? What's their bottleneck right now?
      - danielmarkbruce 3y ago
        Try shoving a couple trill (estimate) parameters in memory. Or, shuttling it back and forth from disk as required to generate a single token. And then shuttling that into the chip. Assuming they are using FP16 it's 4 TB give or take. They aren't kidding when they call them "Large".
    - mindwok 3y ago
      This could easily end up like the PC revolution though. ChatGPT being the mainframe of AI, and when the hardware catches up and performance is “good enough” we’re all running our own models all over the place.
    - objektif 3y ago
      I dont think compute capacity will be the determining factor for moat. It will most likely be access to various types of data and I believe on that front Meta and google are way ahwad
  - danielmarkbruce 3y ago
    It's going the other way. The number of levers to pull around architecture, data amount/type/quality, hardware optimizations, decoding optimizations, evals, system architecture (distinct from model architecture), alignment tricks and various other things means the complexity is just getting started.
- hoc 3y ago
  The old industry question is whether it's complex and protected enough (or the separation wanted by the consumer, but that's an old guy's dream) that the owners of the production equipment can't do it on their own. Azure, AWS, Google are easy candidates, Nvidia might even be proof.
jasonjmcghee 3y ago
I'm surprised to see this comment. I think OpenAI has a huge moat. Companies like Google who are pouring insane resources into beating OpenAI are struggling to. I think they have the potential to be worth $1T. I think the moat will continue to grow. I haven't tried Gemini 1.5 yet, but every single model I have tried haven't come close to GPT-4. I don't care what benchmarks say - they really don't come close. If Gemini 1.5 is competitive, that was a full year to catch up. I'm sure OpenAI has other things cooking.
- bamboozled 3y ago
  I've read several reports Gemini 1.5 being excellent, I have on reason to doubt those reports because we can slag Google off all we like but there is nothing magic about what OpenAI is doing and it's only a short matter of time till others can do it too. Again, you might not like Google but I mean, they have a pretty kick as AI team and all the money and hardware in the world to catchup for lord's sake. People talk and there is zero chance anything OpenAI does will be kept secret forever.
  - flamedoge 3y ago
    I mean.. is there anything magic about what Nvidia's doing?
    - bamboozled 3y ago
      No, and what's your point?
  - jasonjmcghee 3y ago
    I hope the tone of my comment did not convey I thought poorly of anyone here. Google has done incredible work in AI. Which I think makes my point that it took a year to catch up a damn convincing argument that OpenAI has something special going on. And what are you talking about nothing magic about what OpenAI is doing? LLMs are about as damn near close to magic as we have. It's incredible it works at all. They are so damn cool. And Sora is crazy impressive too. 3 years ago, I wouldn't have guessed we would be anywhere near where we already are, and I was in the thick of it, doing ML research and engineering at that time. Eventually sure, but things are moving so fast!

In an exponential game where the prize is ASI, being just a couple of steps ahead may turn out to be enough.

Training & individual user preference data - specifically the data being generated by people interacting with ChatGPT and DALLE. Maybe Google can out data them with all the data they currently hold, but they'd better hurry before the network effects get too strong in favour of OpenAI.

danielmarkbruce 3y ago

Try training and running really really big models.

objektif 3y ago
You mean now. Commodity soon.
- empath-nirvana 3y ago
  Because of scaling advantages, it likely never will be. Even if you could run a GPT4 quality model on your laptop (you can't), by the time you can, Open Ai will be running a model that uses orders of magnitude more compute.
  - jjjjj55555 3y ago
    Stupid question, but why aren't they doing it already? What's their bottleneck right now?
    - danielmarkbruce 3y ago
      Try shoving a couple trill (estimate) parameters in memory. Or, shuttling it back and forth from disk as required to generate a single token. And then shuttling that into the chip. Assuming they are using FP16 it's 4 TB give or take. They aren't kidding when they call them "Large".
  - mindwok 3y ago
    This could easily end up like the PC revolution though. ChatGPT being the mainframe of AI, and when the hardware catches up and performance is “good enough” we’re all running our own models all over the place.
  - objektif 3y ago
    I dont think compute capacity will be the determining factor for moat. It will most likely be access to various types of data and I believe on that front Meta and google are way ahwad
- danielmarkbruce 3y ago
  It's going the other way. The number of levers to pull around architecture, data amount/type/quality, hardware optimizations, decoding optimizations, evals, system architecture (distinct from model architecture), alignment tricks and various other things means the complexity is just getting started.
hoc 3y ago
The old industry question is whether it's complex and protected enough (or the separation wanted by the consumer, but that's an old guy's dream) that the owners of the production equipment can't do it on their own. Azure, AWS, Google are easy candidates, Nvidia might even be proof.

jasonjmcghee 3y ago

I'm surprised to see this comment. I think OpenAI has a huge moat. Companies like Google who are pouring insane resources into beating OpenAI are struggling to. I think they have the potential to be worth $1T. I think the moat will continue to grow. I haven't tried Gemini 1.5 yet, but every single model I have tried haven't come close to GPT-4. I don't care what benchmarks say - they really don't come close. If Gemini 1.5 is competitive, that was a full year to catch up. I'm sure OpenAI has other things cooking.

bamboozled 3y ago
I've read several reports Gemini 1.5 being excellent, I have on reason to doubt those reports because we can slag Google off all we like but there is nothing magic about what OpenAI is doing and it's only a short matter of time till others can do it too. Again, you might not like Google but I mean, they have a pretty kick as AI team and all the money and hardware in the world to catchup for lord's sake. People talk and there is zero chance anything OpenAI does will be kept secret forever.
- flamedoge 3y ago
  I mean.. is there anything magic about what Nvidia's doing?
  - bamboozled 3y ago
    No, and what's your point?
- jasonjmcghee 3y ago
  I hope the tone of my comment did not convey I thought poorly of anyone here. Google has done incredible work in AI. Which I think makes my point that it took a year to catch up a damn convincing argument that OpenAI has something special going on. And what are you talking about nothing magic about what OpenAI is doing? LLMs are about as damn near close to magic as we have. It's incredible it works at all. They are so damn cool. And Sora is crazy impressive too. 3 years ago, I wouldn't have guessed we would be anywhere near where we already are, and I was in the thick of it, doing ML research and engineering at that time. Eventually sure, but things are moving so fast!