4 ms·

In other news (https://archive.ph/WWKQ3 https://archive.ph/WWKQ3): > “I would advocate developing these types of AGI technologies in a cautious manner using th

In other news (https://archive.ph/WWKQ3 https://archive.ph/WWKQ3): > “I would advocate developing these types of AGI technologies in a cautious manner using the scientific method, where you try and do very careful controlled experiments to understand what the underlying system does,” Mr. Hassabis said. Myabe that's why Google was (seemingly) left in the dust?

mach1ne 3y ago
OpenAI minmaxed LLM development, DeepMind minmaxed other avenues of research, such as game-playing AI development. LLMs proved themselves first. Ockham’s Razor suggests there’s nothing more to it.
- raincole 3y ago
  Yes. Today LLM seems like an "obvious" choice, but before ChatGPT, one could easily argue that game-playing AI is with much higher business value (to "play" stock market or even fighters), and chatbots are only good for customer service.
  - kriro 3y ago
    I'd argue both areas are successfull because they found good ways to solve the data problem and go "big data" and the use case while nice and shiney now is not as relevant. Big models trained on text work, because there's a lot of text availabe and the task of predicting the next tokens is nice because you get autolabeled data just by collecting a lot of text. Similarily RL solves this issue by simulations of sorts (supersimplified play a lot of games against eachother, judge somehow cleverly what was better, update stuff, rinse, repeat). I distinctly remember this lightbulb going off in my head when I first played with NLP. I came from ML for computer vision and trained ULMFiT on the entire Wikipedia and was thinking...oh neat no labeling that's magical, if I had more compute I could basically feed this infinite data. I'm not following synthetic data closely (I've used it to train models before) but I see a bright future for this space (like game engines used to simulate big worlds simply to generate data or training in said worlds and transfering).
HarHarVeryFunny 3y ago
Google certainly appear to be a bit behind, although it's hard to know what they have internally that they've chosen not to release. It seems that Google have certainly been cautious about releasing this type of tech, but again hard to tell the exact mix of motivations - e.g. safety vs business risks. Almost all the authors of the Transformer paper have since left Google, and it seems a common theme is that they all thought Google was under-utilizing the tech - maybe just not knowing what to do with it as much as being worried about it ? It seems things are now changing at Google, with the Brain + DeepMind merger and what appears to be a new productization focus. It seems competition may now be trumping safety concerns, even if they are still paying lip service to it!

mach1ne 3y ago

OpenAI minmaxed LLM development, DeepMind minmaxed other avenues of research, such as game-playing AI development. LLMs proved themselves first. Ockham’s Razor suggests there’s nothing more to it.

raincole 3y ago
Yes. Today LLM seems like an "obvious" choice, but before ChatGPT, one could easily argue that game-playing AI is with much higher business value (to "play" stock market or even fighters), and chatbots are only good for customer service.
- kriro 3y ago
  I'd argue both areas are successfull because they found good ways to solve the data problem and go "big data" and the use case while nice and shiney now is not as relevant. Big models trained on text work, because there's a lot of text availabe and the task of predicting the next tokens is nice because you get autolabeled data just by collecting a lot of text. Similarily RL solves this issue by simulations of sorts (supersimplified play a lot of games against eachother, judge somehow cleverly what was better, update stuff, rinse, repeat). I distinctly remember this lightbulb going off in my head when I first played with NLP. I came from ML for computer vision and trained ULMFiT on the entire Wikipedia and was thinking...oh neat no labeling that's magical, if I had more compute I could basically feed this infinite data. I'm not following synthetic data closely (I've used it to train models before) but I see a bright future for this space (like game engines used to simulate big worlds simply to generate data or training in said worlds and transfering).

HarHarVeryFunny 3y ago

Google certainly appear to be a bit behind, although it's hard to know what they have internally that they've chosen not to release. It seems that Google have certainly been cautious about releasing this type of tech, but again hard to tell the exact mix of motivations - e.g. safety vs business risks. Almost all the authors of the Transformer paper have since left Google, and it seems a common theme is that they all thought Google was under-utilizing the tech - maybe just not knowing what to do with it as much as being worried about it ? It seems things are now changing at Google, with the Brain + DeepMind merger and what appears to be a new productization focus. It seems competition may now be trumping safety concerns, even if they are still paying lip service to it!