3 ms·

Ask HN: Why did all major companies release AI at the same time?

• shortrounddev2 3y ago
  I think OpenAI got there first, and Microsoft had been working with them behind the scenes. Google and Facebook got caught off guard and so sort of rushed to release their products; Google released bard in a somewhat incomplete state, and Facebook simply open sourced the work they had been doing internally. Google engineers actually created the technology (transformers[1]) which enabled these LLMs, but OpenAI simply released first. [1] https://en.wikipedia.org/wiki/Transformer_(machine_learning_model) https://en.wikipedia.org/wiki/Transformer_(machine_learning_...
• PaulHoule 3y ago
  There really was a breakthrough in chatbots. Happens all the time in science. People get a really great new idea, all the smart people jump on it, the field is burned out in six months. This revolution in science https://en.wikipedia.org/wiki/Renormalization_group https://en.wikipedia.org/wiki/Renormalization_group didn't last long enough to finish grad school. That idea of "build a language model by predicting the next token" was floating around for a while, I was in two startups that were trying similar things long before it was really feasible. First we were trying to make something that could write fake abstracts for clinical reports from PubMed as a proxy for clinical notes with LSTM, GRUs and such. Later we were training CNNs to complete a battery of classification tasks and believed this could make a "foundation model" that could be trained to new tasks easily. I had a long list of serious problems that doomed many lines of development related to what I was doing and when BERT came it was clear to me that subword tokens were essential (e.g. word vectors are doomed because half the time the most important word isn't in the dictionary!) and also that attention addressed the problem nobody really wanted to face with LSTM which is that in a story about a person you have to copy that person's name and variations over and over again. It wasn't clear to me that transformers were going to be as successful as they were. It helps also that there was a big investment in training technology. Often technological revolutions happen when there is a surplus in the base technologies. For instance, experts in 1970 anticipated we were going to get something like the WWW in 1980 but it was really 15 years late because there was no single company that was going to make the end-to-end investment to bring it to the mass market, excerpt in France https://en.wikipedia.org/wiki/Minitel https://en.wikipedia.org/wiki/Minitel (The TRS-80 Color Computer wasn't really intended to be a computer to teach kids BASIC, it was intended to be a terminal for teleagriculture!) When the web came out, however, the cost of investment in making it work had dropped so low that it didn't take one company to invest in an optimized end-to-end system, rather a bunch of little investments (like the one people made in a 32-bit PC) added up to a network that looked like the computer on the bridge of the Enterprise.
• smoldesu 3y ago
  > Was it the case that AI was already a thing and kept under wraps until someone pulled the trigger? There's transparent evidence that transformer-based LLMs existed before ChatGPT or Llama. There was originally GPT-2, which was not a public release but did get reverse engineered into GPT-Neo and GPT-J (among others). Besides that, there was BERT and T5 from Google, Longformers, XLNet, and dozens of other LLMs that predate the modern ones. > So what happened? OpenAI reduced the friction of using AI to a Google login. Other models existed, but were largely not perceived as powerful enough to compete with a hosted solution. The combination of GPT-3's power and relatively-cheap usage fees is what drove it's adoption. Local AI solutions were being developed before OpenAI stepped into the arena, and are continuing to get developed. Many FAANG companies are heavily invested in the openness of AI architectures.
  • filterfiber 3y ago
    > There's transparent evidence that transformer-based LLMs existed before ChatGPT or Llama. Transformers were first published about in "Attention is all you need" in 2017 by google https://arxiv.org/abs/1706.03762 https://arxiv.org/abs/1706.03762 > There was originally GPT-2, which was not a public release but did get reverse engineered into GPT-Neo and GPT-J (among others). GPT 2 had a proper release in 2019 with the code, structure, checkpoints, etc. - https://openai.com/research/gpt-2-1-5b-release https://openai.com/research/gpt-2-1-5b-release > Besides that, there was BERT and T5 from Google, Longformers, XLNet, and dozens of other LLMs that predate the modern ones. Yes all the big players had various research models. We simply crossed the line into large-scale usefulness just recently (outside of classification/summerization), and like you said OpenAI was the first to really make it accessible to the public.
• wmf 3y ago
  Google had Bard (under a different name) internally for years but didn't release it until ChatGPT forced their hand. Facebook was doing ML research for years (they created PyTorch) so it looks like they were able to create llama pretty rapidly.
• filterfiber 3y ago
  The research for LLMs has been ongoing for years, and all of these company's contributed bleeding edge research using models they internally trained but did not release. For a while it was largely academic and not crazy useful (GPT2ish era). After this they had internal LLM's that could potentially be used to make products. This is where the companies split in their approaches. Google was overly cautious from what I've heard, they were concerned about "safety". Then openai released GPT3.5 and soon after GPT4. During this time Meta took the approach of just releasing their research model outright without first building it into a product. Google then scrambled to build products from their LLMs. tl;dr - everyone of those are big players in the ML/AI space and had internally trained models for their research. We just recently crossed the line where these models became useful thus this scramble to release products (or the models themself). EDIT: It's worth noting that the big breakthrough paper about transformers (Attention is All you Need) was released in 2017. People started scaling up the parameter count, optimizing training time, inference cost, etc. since then.
• streetcat1 3y ago
  I think its about internet real estate. I.e. eyeballs / user to track. If chatgpt can steal eyeballs from google or facebook (something that Bing has tried with no success for the last 10 years), it is existential treat to google or facebook, so they must react. The same thing would have happen with VR. As soon as some company will succeed with VR, your would suddenly see a lot of VR companies.
  • smoldesu 3y ago
    > it is existential treat to google or facebook, so they must react. That may well be true, but both Google and Facebook have been writing Open source AI software longer than OpenAI has been publishing papers. I find it hard to believe that everyone else is playing catch-up with OpenAI right now. I don't think a product like ChatGPT could even exist unless it was massively subsidized.
    • dragonwriter 3y ago
      Everyone else is paying catchup, but only because incumbents had no incentive to productize research and cannibalize their own industry-leading offerings, until a non-incumbent was already doing the first and threatening the second. And mostly what they are catching up on is productization, not basic research.