7 ms·

Why do you think OpenAPI is so far out in front? It's not like there's a lot of secret sauce here - most of this stuff (transformers,etc.) is all out there in p

Why do you think OpenAPI is so far out in front? It's not like there's a lot of secret sauce here - most of this stuff (transformers,etc.) is all out there in papers. And places like Google & Meta must have a lot more computing resources to train on that OpenAI does thus they should be able to train faster. Do you think OpenAI has discovered something they haven't been open about?

nicpottier 4y ago
Until we see otherwise, don't we have to assume there's some secret sauce? Bard doesn't match GPT4 and it isn't for a lack of trying. (though perhaps that will change, so far that's the case)
- xnx 4y ago
  Google is going slow. They might be behind, but we haven't seen their best effort yet. Google has a 540 billion parameter model.
  - nicpottier 4y ago
    So they say, and I agree this seems likely. But until they release it that's all talk, and they have plenty of incentive to release it.
    - nr2x 4y ago
      And plenty of reason not to: search ads.
  - nr2x 4y ago
    Google also has Sundar and Ruth who’d rather focus on how to get another ad on the SERP than kill the golden goose. They’re not going slow, they just don’t have the leadership for the moment.
- saynay 4y ago
  Bard would not trick anyone into thinking it was sentient, yet something they have supposedly did. I just think Google has far more to lose than Bing, so they are being more cautious.
  - dragonwriter 4y ago
    > Bard would not trick anyone into thinking it was sentient, yet something they have supposedly did Bard is overtly a reduced-resources model compared to the best version of the same technology (which, if true, is probably a boneheadedly bad choice for a public demo when everyone is already wowed by the people who got theirs out first, but easily explains that disparity. Though so does “guy who wanted public attention made stuff up well-calibrated to that goal.”)
    - Animats 4y ago
      > Bard is overtly a reduced-resources model compared to the best version of the same technology There's a scaling problem. ChatGPT/LLM systems cost far more to run per query than the Google search engine. Google can't afford to make those the first line query reply. A big business model question is whether Google will insist you be logged in to get to the large language model. At Google scale, these things are going to have to be a hierarchy. Not everything needs to go to a full LLM system. Most Google queries by volume can be answered from a a cache.
      - jacquesm 4y ago
        > Most Google queries by volume can be answered from a a cache. And given how aggressively they limit the number of search results (in spite of listing some ridiculous number of results on page #1) that percentage may well be very large.
  - deleted 4y ago
    [deleted]
- mewpmewp2 4y ago
  Maybe it is just time to train and also good training data for prompts which openai has gathered for so long already? E.g. there is a bottleneck on how fast you can train and also gather good data.
  - UncleOxidant 4y ago
    Possibly, but wouldn't Google and Meta have access to way more compute resources and data than OpenAI? Google has been touting their TPUs for several years now.
    - Robotbeat 4y ago
      OpenAI has access to Microsoft and Azure. That’s bigger than Meta, roughly on par with Google in terms of capability and higher in terms of market cap.
    - chillfox 4y ago
      Google has the compute, from the comparisons I have seen Bard smokes GPT-3.5-Turbo on response times. So my guess is that internal politics prevents them from putting out something better. There would have to be immense pressure from the search division to not make them obsolete.
      - int_19h 4y ago
        Bard is also a fair bit worse than GPT-3.5, though, so that can be a function of model size.
    - nr2x 4y ago
      Without Nadella footing the compute bills, nobody would be taking about OpenAI. He’s brilliant, he let the start up take on huge risk to quietly claim the gains for m$.
fdgsdfogijq 4y ago
I dont work at Google, I think other FAANGS underinvested in this area as they didnt think it was promising. But I will admit, I am suspicious that Google is incompetent. Probably they can come back given how much money they will be forced to throw at it. But Bard is clearly behind and I dont believe their "abundance of caution" arguments for why Bard cant even code.
- UncleOxidant 4y ago
  The impression I get is that they purposely limited Bard in certain ways. It does not seem to be willing to code and makes a lot of excuses when you ask it to (at least as of a couple of weeks ago when I last tried it). Did they put this limitation on it because it's not so good at coding or because they don't want it to be abused in some way? In my experience I had to trick it into coding by telling it to pretend that it had created it's own programming language and then implement an algorithm in it. It seemed to think that it had created Python. > I am suspicious that Google is incompetent. Google has put a lot of effort and investment into AI. With Bard I get the feeling they're not showing us what they really have - it's like for some reason they're holding back the good stuff, at least that's my suspicion.
  - pixl97 4y ago
    I am still wondering if Google is experiencing it Kodak moment. They have the dominant product that makes them billions and billions of dollars at 'relatively' low cost. The new dominant product is on its way, but it costs far more to operate and will net them far less money, so... um no one wants to kill the goose that is still laying golden eggs, even though its days are numbered already.
    - xyzzy123 4y ago
      I think it's not just run cost - there's also a risk of ad revenue destruction, that has to be worrying.
      - iamleppert 4y ago
redox99 4y ago
I don't know about GPT4, but GPT3.5 I'd bet is pretty traditional and boring. It's power comes from a really good, properly curated dataset (including the RLHF). GPT3.5 turbo is much more interesting probably, because they seem to have found out how to make it much more efficient (some kind of distillation?). GPT4 if I had to make a very rough guess, probably flash attention, 100% of the (useful) internet/books for it's dataset, and highly optimized hyperparameters. I'd say with GPT4 they probably reached the limit of how big the dataset can be, because they are already using all the data that exists. Thus for GPT5 they'll have to scale in other ways.
- brendamn 4y ago
  > I'd say with GPT4 they probably reached the limit of how big the dataset can be I’m curious about this too; not just on the dataset size, but also the model size. My hunch is that the rapid improvements of the underlying model by making it bigger/giving it more data will slow, and there’ll be more focus on shrinking the models/other optimisations.
  - int_19h 4y ago
    I don't think we're anywhere close to the limit of sheer hardware scalability on this. Returns are diminishing, but if GPT-4 (with its 8+ k context window) is any indication, even those diminishing returns are still very worthwhile. If anything, I wonder if the actual limit that'll be hit first will be the global manufacturing capacity for relevant hardware. Check out the stock price of NVDA since last October.
- flangola7 4y ago
  According to financial reports they are building a $225 million supercomputer for AI. What we can probably expect is the same dataset with even more compute ran on it.
- theturtletalks 4y ago
  Is there a limit on how big the context size can be?
diego 4y ago
If you try Bard or Claude or character.ai they are not far behind GPT4. They might even be on par in terms of raw LLM capabilities. ChatGPT has better marketing and in some cases better UX. A lot of this is self-fulfilling. We think it's far ahead, so it appears to be far ahead.
- dragonwriter 4y ago
  > If you try Bard or Claude or character.ai they are not far behind GPT4 Bard is way behind ChatGPT with GPT-3.5, much less GPT-4. Haven’t tried the others, though. OTOH, that’s way behind qualitatively, not in terms of time-of-progress. So I don’t think it is at all an insurmountable lead, as much as it is a big utility gap.
- sashank_1509 4y ago
  In my experiments: GPT4>ChatGPT>Claude>Character AI> Bard Claude and Character AI are great at holding a conversation but they lack the ability to do anything specialized that really makes these LLM’s useful in my day to day life. I ask GPT-4 and ChatGPT questions I would ask in stackoverflow, I can’t do that with Claude or Character AI. Bard actually seems behind even conversationally to the rest
JumpCrisscross 4y ago
> Why do you think OpenAPI is so far out in front? There is a network effect forming around its models. The strengths of its kit speak for themselves. (It also cannot be understated how making ChatGPT public, something its competitors were too feeble, incompetent and behind the curve to do, dealt OpenAI a massive first-mover advantage.) But as others note, other models are in the ballpark. Where OpenAI is different is in the ecosystem of marketing literature, contracts, code and e.g. prompt engineers being written and trained with GPT in mind. That introduces a subtle switching cost, and not-so-subtle platform advantage, that–barring a Google-scale bout of incompetence–OpenAI is set to retain for some time.
- paulryanrogers 4y ago
  Doubtful "feeble and incompetent" are fitting considering much of the research behind OpenAI was pioneered and published by those predecessors. More like "unwilling", and perhaps for good reason. Time will tell.
  - deleted 4y ago
    [deleted]
  - nr2x 4y ago
    Google is where ideas go to die.
- sberens 4y ago
  > other models are in the ballpark How true is this? From playing around with Bard and Claude, GPT-4 seems to be significantly better, especially around code generation / understanding.
  - letitgo12345 4y ago
    People really don't seem to understand just how far ahead OAI is in its reasoning abilities (https://crfm.stanford.edu/helm/v0.2.2/?group=reasoning https://crfm.stanford.edu/helm/v0.2.2/?group=reasoning) Maybe PaLM is near there (it's not evaluated on that page) but nothing else even comes close at all
black3r 4y ago
For a little more than a year I worked in an AI startup doing basically everything other than AI (APIs, webapps, devops...), but from what I've seen there the "secret sauce" to AI success is the training process (dataset, parameters, fine-tuning steps, ...). And OpenAI isn't open about theirs since their beginnings.
logicallee 4y ago
All of the neural network architecture for human level thinking and processing, including vision, speech, emotion, abstract thought, balance and fine motor skills, everything was publicly released in April 2003, twenty years ago this month. It's a 700 megabyte tarball and sets up an 80b parameter neural network. What? Huh? Yes the human genome encodes all human level thought.[1] Clearly it does because the only difference between humans that have abstract thought as well as language capabilities and primates that don't is slightly different DNA. In other words: those slight differences matter. To anyone who has used GPT since ChatGPT's public release in November and who pays to use GPT 4 now, it is clear that GPT 4 is a lot smarter than 3 was. However, to the select few who see an ocean in a drop of water, the November release already showed glimmers of abstract thought, many other people dismiss it as an illusion. To a select few, it is apparent that OpenAI have found the magic parameters. Everything after that is just fine tuning. Is it any surprise that without OpenAI releasing their weights, models, or training data, Google can't just come up with its own? Why should they when without turning it into weights and models, the human neural network architecture itself is still unmatched (even by OpenAI) despite being digitized twenty years ago? No, it's no surprise. OpenAI performed what amounts to a miracle, ten years ahead of schedule, and didn't tell anyone how they did it. If you work for another company, such as Google, don't be surprised that you are ten years behind. After all, the magic formula had been gathering dust on a CD-ROM for 20 years (human DNA which encodes the human neural network architecture), and nobody made the slightest tangible progress toward it until OpenAI brute forced a solution using $1 billion of Azure GPU's that Microsoft poured into OpenAI in 2019. Is your team using $1 billion of GPU's for 3 years? If not, don't expect to catch up with OpenAI's November miracle. p.s. two months after the November miracle, Microsoft closed a $10 billion follow-on investment in OpenAI. [1] https://en.m.wikipedia.org/wiki/Human_Genome_Project https://en.m.wikipedia.org/wiki/Human_Genome_Project
letitgo12345 4y ago
They have been collecting human feedback data for 2 years + probably have a lot of data from Copilot + are training with large context models + have invested a ridiculous amount in curating pretraining data -- the kind of stuff that won't get you a ton of pubs (so you won't see Google researchers having focused on it a lot) but apparently turns out to be super important for a good LLM
sangnoir 4y ago
> Do you think OpenAI has discovered something they haven't been open about? They have not, which makes me curious about which company gp works for because the "F" and "G" in FAANG are publicly known to already have LLMs. Not sure about Amazon, but I'm guessing they do too. As an outsider, the amazing thing about ML/AI research is that you get a revolutionary discovery of a technique or refinement that changes everything, and a few months later another seminal paper is published[0]. My bet is ChatGPT is not the last word in AI, and OpenAI will not have a monopoly on upcoming discoveries that will improve the state of the art. They will have to contend with the fact that Google, Meta & Amazon own their datacenters and can likely train models for cheaper[1] than what Microsoft is paying itself via their investment in OpenAI. 0. In no particular order: Deep learning, GANs, Transformers, transfer learning, Style Transfer, auto-encoders, BERT, LLMs. Betting the farm on LLMs doesn't sound like a reasonable thing to do - not saying that's what OpenAI is doing, but there are a lot of folk on HN who are treating LLMs as the holy grail. 1. OpenAI may get a discount, but my prediction when they burn through Microsoft, they'll end up being "owned" by Microsoft for all intents and purposes.
- letitgo12345 4y ago
  Issue is the data moat OAI is building. They'll have hundreds of millions of high quality user interactions with ChatGPT they can use to finetune their models. What will anyone else including Google have?
  - Teever 4y ago
    As people make chat bots with openAI and tie them into existing chat services, organizations that offer these chat services will get their hands on that kind of data too. Discord comes to mind.
    - letitgo12345 4y ago
      That's possible ya. Hope it happens
  - nmfisher 4y ago
andsoitis 4y ago
Having a model does not a platform or a product make. You also need users and mindshare. OpenAI is enjoying first mover advantage around the platformication and product-ification of LLMs. For instance, why has G not yet exposed some next-level capabilities in mail, in docs, and many of their other properties? Why do Google Assistant and Amazon Alexa and Apple Siri still suck?
hackerlight 4y ago
It probably is the secret sauce which remains undisclosed. Differences that seem small can lead to large differences in model quality.
majormajor 4y ago
A lot of FAANG data folks aren't on the teams that were doing research into this stuff and weren't using the latest fruits of that research. OpenAI has released a ton more easy-to-use-for-everyone stuff that has really leapfrogged what a lot of "applied" folks everywhere else were trying to build themselves, despite being on-the-face-of-it more "general."
- nr2x 4y ago
  Exactly!
rqtwteye 4y ago
I think it’s the way things go usually. The big players have a business to run so they can’t focus much on innovation. OpenAI has the only purpose right now to push AI and nothing else. Once they have a real business they will also slow down.

nicpottier 4y ago

Until we see otherwise, don't we have to assume there's some secret sauce? Bard doesn't match GPT4 and it isn't for a lack of trying. (though perhaps that will change, so far that's the case)

xnx 4y ago
Google is going slow. They might be behind, but we haven't seen their best effort yet. Google has a 540 billion parameter model.
- nicpottier 4y ago
  So they say, and I agree this seems likely. But until they release it that's all talk, and they have plenty of incentive to release it.
  - nr2x 4y ago
    And plenty of reason not to: search ads.
- nr2x 4y ago
  Google also has Sundar and Ruth who’d rather focus on how to get another ad on the SERP than kill the golden goose. They’re not going slow, they just don’t have the leadership for the moment.
saynay 4y ago
Bard would not trick anyone into thinking it was sentient, yet something they have supposedly did. I just think Google has far more to lose than Bing, so they are being more cautious.
- dragonwriter 4y ago
  > Bard would not trick anyone into thinking it was sentient, yet something they have supposedly did Bard is overtly a reduced-resources model compared to the best version of the same technology (which, if true, is probably a boneheadedly bad choice for a public demo when everyone is already wowed by the people who got theirs out first, but easily explains that disparity. Though so does “guy who wanted public attention made stuff up well-calibrated to that goal.”)
  - Animats 4y ago
    > Bard is overtly a reduced-resources model compared to the best version of the same technology There's a scaling problem. ChatGPT/LLM systems cost far more to run per query than the Google search engine. Google can't afford to make those the first line query reply. A big business model question is whether Google will insist you be logged in to get to the large language model. At Google scale, these things are going to have to be a hierarchy. Not everything needs to go to a full LLM system. Most Google queries by volume can be answered from a a cache.
    - jacquesm 4y ago
      > Most Google queries by volume can be answered from a a cache. And given how aggressively they limit the number of search results (in spite of listing some ridiculous number of results on page #1) that percentage may well be very large.
- deleted 4y ago
  [deleted]
mewpmewp2 4y ago
Maybe it is just time to train and also good training data for prompts which openai has gathered for so long already? E.g. there is a bottleneck on how fast you can train and also gather good data.
- UncleOxidant 4y ago
  Possibly, but wouldn't Google and Meta have access to way more compute resources and data than OpenAI? Google has been touting their TPUs for several years now.
  - Robotbeat 4y ago
    OpenAI has access to Microsoft and Azure. That’s bigger than Meta, roughly on par with Google in terms of capability and higher in terms of market cap.
  - chillfox 4y ago
    Google has the compute, from the comparisons I have seen Bard smokes GPT-3.5-Turbo on response times. So my guess is that internal politics prevents them from putting out something better. There would have to be immense pressure from the search division to not make them obsolete.
    - int_19h 4y ago
      Bard is also a fair bit worse than GPT-3.5, though, so that can be a function of model size.
  - nr2x 4y ago
    Without Nadella footing the compute bills, nobody would be taking about OpenAI. He’s brilliant, he let the start up take on huge risk to quietly claim the gains for m$.

fdgsdfogijq 4y ago

I dont work at Google, I think other FAANGS underinvested in this area as they didnt think it was promising. But I will admit, I am suspicious that Google is incompetent. Probably they can come back given how much money they will be forced to throw at it. But Bard is clearly behind and I dont believe their "abundance of caution" arguments for why Bard cant even code.

UncleOxidant 4y ago
The impression I get is that they purposely limited Bard in certain ways. It does not seem to be willing to code and makes a lot of excuses when you ask it to (at least as of a couple of weeks ago when I last tried it). Did they put this limitation on it because it's not so good at coding or because they don't want it to be abused in some way? In my experience I had to trick it into coding by telling it to pretend that it had created it's own programming language and then implement an algorithm in it. It seemed to think that it had created Python. > I am suspicious that Google is incompetent. Google has put a lot of effort and investment into AI. With Bard I get the feeling they're not showing us what they really have - it's like for some reason they're holding back the good stuff, at least that's my suspicion.
- pixl97 4y ago
  I am still wondering if Google is experiencing it Kodak moment. They have the dominant product that makes them billions and billions of dollars at 'relatively' low cost. The new dominant product is on its way, but it costs far more to operate and will net them far less money, so... um no one wants to kill the goose that is still laying golden eggs, even though its days are numbered already.
  - xyzzy123 4y ago
    I think it's not just run cost - there's also a risk of ad revenue destruction, that has to be worrying.
    - iamleppert 4y ago

redox99 4y ago

I don't know about GPT4, but GPT3.5 I'd bet is pretty traditional and boring. It's power comes from a really good, properly curated dataset (including the RLHF). GPT3.5 turbo is much more interesting probably, because they seem to have found out how to make it much more efficient (some kind of distillation?). GPT4 if I had to make a very rough guess, probably flash attention, 100% of the (useful) internet/books for it's dataset, and highly optimized hyperparameters. I'd say with GPT4 they probably reached the limit of how big the dataset can be, because they are already using all the data that exists. Thus for GPT5 they'll have to scale in other ways.

brendamn 4y ago
> I'd say with GPT4 they probably reached the limit of how big the dataset can be I’m curious about this too; not just on the dataset size, but also the model size. My hunch is that the rapid improvements of the underlying model by making it bigger/giving it more data will slow, and there’ll be more focus on shrinking the models/other optimisations.
- int_19h 4y ago
  I don't think we're anywhere close to the limit of sheer hardware scalability on this. Returns are diminishing, but if GPT-4 (with its 8+ k context window) is any indication, even those diminishing returns are still very worthwhile. If anything, I wonder if the actual limit that'll be hit first will be the global manufacturing capacity for relevant hardware. Check out the stock price of NVDA since last October.
flangola7 4y ago
According to financial reports they are building a $225 million supercomputer for AI. What we can probably expect is the same dataset with even more compute ran on it.
theturtletalks 4y ago
Is there a limit on how big the context size can be?

diego 4y ago

If you try Bard or Claude or character.ai they are not far behind GPT4. They might even be on par in terms of raw LLM capabilities. ChatGPT has better marketing and in some cases better UX. A lot of this is self-fulfilling. We think it's far ahead, so it appears to be far ahead.

dragonwriter 4y ago
> If you try Bard or Claude or character.ai they are not far behind GPT4 Bard is way behind ChatGPT with GPT-3.5, much less GPT-4. Haven’t tried the others, though. OTOH, that’s way behind qualitatively, not in terms of time-of-progress. So I don’t think it is at all an insurmountable lead, as much as it is a big utility gap.
sashank_1509 4y ago
In my experiments: GPT4>ChatGPT>Claude>Character AI> Bard Claude and Character AI are great at holding a conversation but they lack the ability to do anything specialized that really makes these LLM’s useful in my day to day life. I ask GPT-4 and ChatGPT questions I would ask in stackoverflow, I can’t do that with Claude or Character AI. Bard actually seems behind even conversationally to the rest

JumpCrisscross 4y ago

> Why do you think OpenAPI is so far out in front? There is a network effect forming around its models. The strengths of its kit speak for themselves. (It also cannot be understated how making ChatGPT public, something its competitors were too feeble, incompetent and behind the curve to do, dealt OpenAI a massive first-mover advantage.) But as others note, other models are in the ballpark. Where OpenAI is different is in the ecosystem of marketing literature, contracts, code and e.g. prompt engineers being written and trained with GPT in mind. That introduces a subtle switching cost, and not-so-subtle platform advantage, that–barring a Google-scale bout of incompetence–OpenAI is set to retain for some time.

paulryanrogers 4y ago
Doubtful "feeble and incompetent" are fitting considering much of the research behind OpenAI was pioneered and published by those predecessors. More like "unwilling", and perhaps for good reason. Time will tell.
- deleted 4y ago
  [deleted]
- nr2x 4y ago
  Google is where ideas go to die.
sberens 4y ago
> other models are in the ballpark How true is this? From playing around with Bard and Claude, GPT-4 seems to be significantly better, especially around code generation / understanding.
- letitgo12345 4y ago
  People really don't seem to understand just how far ahead OAI is in its reasoning abilities (https://crfm.stanford.edu/helm/v0.2.2/?group=reasoning https://crfm.stanford.edu/helm/v0.2.2/?group=reasoning) Maybe PaLM is near there (it's not evaluated on that page) but nothing else even comes close at all

black3r 4y ago

For a little more than a year I worked in an AI startup doing basically everything other than AI (APIs, webapps, devops...), but from what I've seen there the "secret sauce" to AI success is the training process (dataset, parameters, fine-tuning steps, ...). And OpenAI isn't open about theirs since their beginnings.

logicallee 4y ago

All of the neural network architecture for human level thinking and processing, including vision, speech, emotion, abstract thought, balance and fine motor skills, everything was publicly released in April 2003, twenty years ago this month. It's a 700 megabyte tarball and sets up an 80b parameter neural network. What? Huh? Yes the human genome encodes all human level thought.[1] Clearly it does because the only difference between humans that have abstract thought as well as language capabilities and primates that don't is slightly different DNA. In other words: those slight differences matter. To anyone who has used GPT since ChatGPT's public release in November and who pays to use GPT 4 now, it is clear that GPT 4 is a lot smarter than 3 was. However, to the select few who see an ocean in a drop of water, the November release already showed glimmers of abstract thought, many other people dismiss it as an illusion. To a select few, it is apparent that OpenAI have found the magic parameters. Everything after that is just fine tuning. Is it any surprise that without OpenAI releasing their weights, models, or training data, Google can't just come up with its own? Why should they when without turning it into weights and models, the human neural network architecture itself is still unmatched (even by OpenAI) despite being digitized twenty years ago? No, it's no surprise. OpenAI performed what amounts to a miracle, ten years ahead of schedule, and didn't tell anyone how they did it. If you work for another company, such as Google, don't be surprised that you are ten years behind. After all, the magic formula had been gathering dust on a CD-ROM for 20 years (human DNA which encodes the human neural network architecture), and nobody made the slightest tangible progress toward it until OpenAI brute forced a solution using $1 billion of Azure GPU's that Microsoft poured into OpenAI in 2019. Is your team using $1 billion of GPU's for 3 years? If not, don't expect to catch up with OpenAI's November miracle. p.s. two months after the November miracle, Microsoft closed a $10 billion follow-on investment in OpenAI. [1] https://en.m.wikipedia.org/wiki/Human_Genome_Project https://en.m.wikipedia.org/wiki/Human_Genome_Project

letitgo12345 4y ago

They have been collecting human feedback data for 2 years + probably have a lot of data from Copilot + are training with large context models + have invested a ridiculous amount in curating pretraining data -- the kind of stuff that won't get you a ton of pubs (so you won't see Google researchers having focused on it a lot) but apparently turns out to be super important for a good LLM

sangnoir 4y ago

> Do you think OpenAI has discovered something they haven't been open about? They have not, which makes me curious about which company gp works for because the "F" and "G" in FAANG are publicly known to already have LLMs. Not sure about Amazon, but I'm guessing they do too. As an outsider, the amazing thing about ML/AI research is that you get a revolutionary discovery of a technique or refinement that changes everything, and a few months later another seminal paper is published[0]. My bet is ChatGPT is not the last word in AI, and OpenAI will not have a monopoly on upcoming discoveries that will improve the state of the art. They will have to contend with the fact that Google, Meta & Amazon own their datacenters and can likely train models for cheaper[1] than what Microsoft is paying itself via their investment in OpenAI. 0. In no particular order: Deep learning, GANs, Transformers, transfer learning, Style Transfer, auto-encoders, BERT, LLMs. Betting the farm on LLMs doesn't sound like a reasonable thing to do - not saying that's what OpenAI is doing, but there are a lot of folk on HN who are treating LLMs as the holy grail. 1. OpenAI may get a discount, but my prediction when they burn through Microsoft, they'll end up being "owned" by Microsoft for all intents and purposes.

letitgo12345 4y ago
Issue is the data moat OAI is building. They'll have hundreds of millions of high quality user interactions with ChatGPT they can use to finetune their models. What will anyone else including Google have?
- Teever 4y ago
  As people make chat bots with openAI and tie them into existing chat services, organizations that offer these chat services will get their hands on that kind of data too. Discord comes to mind.
  - letitgo12345 4y ago
    That's possible ya. Hope it happens
- nmfisher 4y ago

andsoitis 4y ago

Having a model does not a platform or a product make. You also need users and mindshare. OpenAI is enjoying first mover advantage around the platformication and product-ification of LLMs. For instance, why has G not yet exposed some next-level capabilities in mail, in docs, and many of their other properties? Why do Google Assistant and Amazon Alexa and Apple Siri still suck?

hackerlight 4y ago

It probably is the secret sauce which remains undisclosed. Differences that seem small can lead to large differences in model quality.

majormajor 4y ago

A lot of FAANG data folks aren't on the teams that were doing research into this stuff and weren't using the latest fruits of that research. OpenAI has released a ton more easy-to-use-for-everyone stuff that has really leapfrogged what a lot of "applied" folks everywhere else were trying to build themselves, despite being on-the-face-of-it more "general."

nr2x 4y ago
Exactly!

rqtwteye 4y ago

I think it’s the way things go usually. The big players have a business to run so they can’t focus much on innovation. OpenAI has the only purpose right now to push AI and nothing else. Once they have a real business they will also slow down.