3 ms·

> What is OpenAI's moat? From what I understand, if you take the absolute best cutting edge LLM with the most parameters and the most up to date model from Git

> What is OpenAI's moat? From what I understand, if you take the absolute best cutting edge LLM with the most parameters and the most up to date model from GitHub/HuggingFace/whatever, it's very far off from the output you get from GPT-3.5 / GPT-4 aka full of hallucinations, not very useful I don't know if this is the right way to look at it but if what George Hotz said about GPT-4 simply being "8 220B parameter models glued together by something called a mixture-of-experts", from what I understand, OpenAI's moat is: their access/subsidiized cost to GPUs/infrastructure with Microsoft the 8 220B models they have are really good/I don't think anything open source matches them/nobody can download "all of Reddit/Twitter/Wikipedia/StackOverflow/whatever else they trained on" anymore like they could given how everybody wants to protect/monetize their content now and then the "router" / "MoE" piece seems to be something missing from open source offerings as well

easygenes 3y ago
Depending on the task, the best open models will outperform GPT-3.5, but would be more expensive to run at comparable speed. GPT-4 is in a league of its own.
passion__desire 3y ago
What if specialized smaller models is the best way ahead for community. I don't care if I am interacting with one big model which can do everything or I have to go to different websites to access specific models. All model sizes will be useful. Smaller models will be frequently used. Bigger less so.
nmfisher 3y ago
This isn’t really true (or at least, doesn’t apply across the board). Qwen (Alibaba’s open source model) outperforms GPT4 on Chinese language tasks, and I can further finetune it for my own tasks (which I’ve done, and I confirm it’s produces more natural output than GPT4). Other benchmarks/anecdotes suggest fine-tuned code models are outperforming GPT4 too. The trend seems to be that smaller, fine-tuned task specific models outperform larger generalised models. It requires a lot of resources to pretrain the base model, but as we’ve seen, there’s no shortage of companies who are willing and able to do that. Not to mention, all those other companies are already profitable, whereas OpenAI is already burning investor cash.

easygenes 3y ago

Depending on the task, the best open models will outperform GPT-3.5, but would be more expensive to run at comparable speed. GPT-4 is in a league of its own.

passion__desire 3y ago

What if specialized smaller models is the best way ahead for community. I don't care if I am interacting with one big model which can do everything or I have to go to different websites to access specific models. All model sizes will be useful. Smaller models will be frequently used. Bigger less so.

nmfisher 3y ago

This isn’t really true (or at least, doesn’t apply across the board). Qwen (Alibaba’s open source model) outperforms GPT4 on Chinese language tasks, and I can further finetune it for my own tasks (which I’ve done, and I confirm it’s produces more natural output than GPT4). Other benchmarks/anecdotes suggest fine-tuned code models are outperforming GPT4 too. The trend seems to be that smaller, fine-tuned task specific models outperform larger generalised models. It requires a lot of resources to pretrain the base model, but as we’ve seen, there’s no shortage of companies who are willing and able to do that. Not to mention, all those other companies are already profitable, whereas OpenAI is already burning investor cash.