4 ms·

What is OpenAI's moat? Loads of people outside the company are working on alternative models. They may have a lead right now but will it last a few years? Will

What is OpenAI's moat? Loads of people outside the company are working on alternative models. They may have a lead right now but will it last a few years? Will it even last 6 months?

yumraj 3y ago
> What is OpenAI's moat? There’s none. Which is why Sam Altman has been crying wolf, in hope of regulatory barriers which can provide it the moat.
MuffinFlavored 3y ago
> What is OpenAI's moat? From what I understand, if you take the absolute best cutting edge LLM with the most parameters and the most up to date model from GitHub/HuggingFace/whatever, it's very far off from the output you get from GPT-3.5 / GPT-4 aka full of hallucinations, not very useful I don't know if this is the right way to look at it but if what George Hotz said about GPT-4 simply being "8 220B parameter models glued together by something called a mixture-of-experts", from what I understand, OpenAI's moat is: their access/subsidiized cost to GPUs/infrastructure with Microsoft the 8 220B models they have are really good/I don't think anything open source matches them/nobody can download "all of Reddit/Twitter/Wikipedia/StackOverflow/whatever else they trained on" anymore like they could given how everybody wants to protect/monetize their content now and then the "router" / "MoE" piece seems to be something missing from open source offerings as well
- easygenes 3y ago
  Depending on the task, the best open models will outperform GPT-3.5, but would be more expensive to run at comparable speed. GPT-4 is in a league of its own.
- passion__desire 3y ago
  What if specialized smaller models is the best way ahead for community. I don't care if I am interacting with one big model which can do everything or I have to go to different websites to access specific models. All model sizes will be useful. Smaller models will be frequently used. Bigger less so.
- nmfisher 3y ago
  This isn’t really true (or at least, doesn’t apply across the board). Qwen (Alibaba’s open source model) outperforms GPT4 on Chinese language tasks, and I can further finetune it for my own tasks (which I’ve done, and I confirm it’s produces more natural output than GPT4). Other benchmarks/anecdotes suggest fine-tuned code models are outperforming GPT4 too. The trend seems to be that smaller, fine-tuned task specific models outperform larger generalised models. It requires a lot of resources to pretrain the base model, but as we’ve seen, there’s no shortage of companies who are willing and able to do that. Not to mention, all those other companies are already profitable, whereas OpenAI is already burning investor cash.
ben_w 3y ago
OpenAI's "moat" is basically the same as Adobe's or Microsoft's, give or take a metaphor, for Photoshop or Office. Although see last week for previous responses: https://news.ycombinator.com/item?id=37333747 https://news.ycombinator.com/item?id=37333747
foobiekr 3y ago
Adoption and a mass of human feedback collected which is not available in the gleaned data sets. Here’s another way to think about it. Why does ISA matter in CPUs? There are minor issues around efficiencies of various kinds, but the real advantage of any mainstream ISA is, in part, the availability of tooling (hence this was a correct and heavy early focus for the RISCV effort) but also a lot of ecosystem things you don’t see: for example, Intel and Arm have truly mammoth test and verification suites that represent thousands++ of man years of investment. OpenAI almost certainly has a massive invisible accumulated value at this point. The actual models themselves are the output in the same way that a packaged CPU is the output. How you got there matters almost as much or more.
- AnthonyMouse 3y ago
  > Here’s another way to think about it. Why does ISA matter in CPUs? Honestly the answer is that it mostly doesn't. An ISA isn't viable without tooling, but that's why it's the first thing they all get. The only ISA with any significant moat is x86, and that's because there is so much legacy closed source software for it that people still need but would have to be emulated on any other architecture. And even that only works as long as x86 processors are competitive; if they fell behind then customers would just eat the emulation overhead on something else. Other ISAs don't even have that. Would anybody actually be surprised if RISC-V took a huge chunk of out ARM's market share in the not too distant future?
  - foobiekr 3y ago
    That's literally my point. The problem is that there's a massive amount of hidden infrastructure behind those that you don't see and that "oh look everyone has a big model" isn't as impressive as it sounds.
    - AnthonyMouse 3y ago
      But the open source infrastructure is getting built too. And the infrastructure is mostly independent of the model. This is Falcon 180B running using the code from llama.cpp.

What is OpenAI's moat? Loads of people outside the company are working on alternative models. They may have a lead right now but will it last a few years? Will it even last 6 months?

yumraj 3y ago

> What is OpenAI's moat? There’s none. Which is why Sam Altman has been crying wolf, in hope of regulatory barriers which can provide it the moat.

MuffinFlavored 3y ago

> What is OpenAI's moat? From what I understand, if you take the absolute best cutting edge LLM with the most parameters and the most up to date model from GitHub/HuggingFace/whatever, it's very far off from the output you get from GPT-3.5 / GPT-4 aka full of hallucinations, not very useful I don't know if this is the right way to look at it but if what George Hotz said about GPT-4 simply being "8 220B parameter models glued together by something called a mixture-of-experts", from what I understand, OpenAI's moat is: their access/subsidiized cost to GPUs/infrastructure with Microsoft the 8 220B models they have are really good/I don't think anything open source matches them/nobody can download "all of Reddit/Twitter/Wikipedia/StackOverflow/whatever else they trained on" anymore like they could given how everybody wants to protect/monetize their content now and then the "router" / "MoE" piece seems to be something missing from open source offerings as well

easygenes 3y ago
Depending on the task, the best open models will outperform GPT-3.5, but would be more expensive to run at comparable speed. GPT-4 is in a league of its own.
passion__desire 3y ago
What if specialized smaller models is the best way ahead for community. I don't care if I am interacting with one big model which can do everything or I have to go to different websites to access specific models. All model sizes will be useful. Smaller models will be frequently used. Bigger less so.
nmfisher 3y ago
This isn’t really true (or at least, doesn’t apply across the board). Qwen (Alibaba’s open source model) outperforms GPT4 on Chinese language tasks, and I can further finetune it for my own tasks (which I’ve done, and I confirm it’s produces more natural output than GPT4). Other benchmarks/anecdotes suggest fine-tuned code models are outperforming GPT4 too. The trend seems to be that smaller, fine-tuned task specific models outperform larger generalised models. It requires a lot of resources to pretrain the base model, but as we’ve seen, there’s no shortage of companies who are willing and able to do that. Not to mention, all those other companies are already profitable, whereas OpenAI is already burning investor cash.

ben_w 3y ago

OpenAI's "moat" is basically the same as Adobe's or Microsoft's, give or take a metaphor, for Photoshop or Office. Although see last week for previous responses: https://news.ycombinator.com/item?id=37333747 https://news.ycombinator.com/item?id=37333747

foobiekr 3y ago

Adoption and a mass of human feedback collected which is not available in the gleaned data sets. Here’s another way to think about it. Why does ISA matter in CPUs? There are minor issues around efficiencies of various kinds, but the real advantage of any mainstream ISA is, in part, the availability of tooling (hence this was a correct and heavy early focus for the RISCV effort) but also a lot of ecosystem things you don’t see: for example, Intel and Arm have truly mammoth test and verification suites that represent thousands++ of man years of investment. OpenAI almost certainly has a massive invisible accumulated value at this point. The actual models themselves are the output in the same way that a packaged CPU is the output. How you got there matters almost as much or more.

AnthonyMouse 3y ago
> Here’s another way to think about it. Why does ISA matter in CPUs? Honestly the answer is that it mostly doesn't. An ISA isn't viable without tooling, but that's why it's the first thing they all get. The only ISA with any significant moat is x86, and that's because there is so much legacy closed source software for it that people still need but would have to be emulated on any other architecture. And even that only works as long as x86 processors are competitive; if they fell behind then customers would just eat the emulation overhead on something else. Other ISAs don't even have that. Would anybody actually be surprised if RISC-V took a huge chunk of out ARM's market share in the not too distant future?
- foobiekr 3y ago
  That's literally my point. The problem is that there's a massive amount of hidden infrastructure behind those that you don't see and that "oh look everyone has a big model" isn't as impressive as it sounds.
  - AnthonyMouse 3y ago
    But the open source infrastructure is getting built too. And the infrastructure is mostly independent of the model. This is Falcon 180B running using the code from llama.cpp.