7 ms·

Strangest thing about this situation is that the "Attention Is All You Need" paper which, from what I read was crucial to give birth to the technology that fuel

by mrfinn 3y ago

Strangest thing about this situation is that the "Attention Is All You Need" paper which, from what I read was crucial to give birth to the technology that fuels OpenAI's GPT actually came from Google Labs. So in theory they had the very upper hand implementing it, back in 2018. Did they put it on hold on purpose?

skyhlar 3y ago
Well the paper only talks about translation, the authors never seem to have considered generalizing the architecture to other problems.
- mrfinn 3y ago
  Really? Because LaMDA looks like an effort in that direction but Google never made it publicly available AFAIK. Security concerns maybe? PS. Just read the Lemoine's transcriptions. Just wow.
  - akomtu 3y ago
    Big corps are risk averse. When you're at the top of the pyramid, there's nowhere else to climb, but there are plently of opportunities to fall, so you move very carefully and defensively. The lowly climbers at the bottom have nowhere else to fall, and have plenty of reasons to climb, so they climb in the most careless and brazen manner. Eventually they dislodge the lazy ass at the top and become risk averse. The winner of this show is the owner of the pyramid who sells food to the climbers.
add-sub-mul-div 3y ago
In Minority Report we saw a guy controlling a computer interface by waving his arms in the air, and ever since we've had people talking about how cool it would be to have that in real life. Well, actually, no it wouldn't. In practice your arms would get tired as hell pretty quickly. A mouse on a table surface is actually a pretty great solution, which is why it's lasted. People are drawn to things that look cool and innovative for the sake of it, but we're not good at thinking ahead to how well they'd work in practice. I've been searching the web for so long that it's like breathing. I type a few keywords, hop between the first few results very fast, get what I need and get out. Maybe I have to repeat. Few searches are more complicated than that. I've tried replacing that flow with AI. It's slower to input, slower to get results, the results are opaque with regards to where they came from and how trustworthy they are. And the results aren't better, in my experience as someone for who finds search to have been solved and painless for many years. Even asking one question of an LLM takes more time and effort than a traditional search, and the problem is compounded when I have to iterate on a search. (Also it sucks to cede the kind of power to search providers that gives them leverage to force us to be logged in and monetize what's always been free, but those are just random complaints.) So, going back to your question. My guess is that Google has always known this isn't a better overall experience, once you get over the initial wow factor. But maybe they underestimated how much that wow factor would allow true competition to materialize before the rest of us would figure it out and lose interest.
- boringuser2 3y ago
  You're right. So far, the only use-case I've seen for LLMs are basically code completion.
  - anshumankmr 3y ago
    It is usually good for auto complete in general, plus summarizing large blocks of non technical text. Probably once ChatGPT is open to fine tuning, we will be able to see how that can be leveraged as well.
    - boringuser2 3y ago
      The biggest failure of these LLMs it seems is the restricted context. It would need the ability to contain infinite context to be genuinely useful in my opinion.
      - falcor84 3y ago
        Do you think that your fellow man has infinite context about your situation?
        boringuser2 3y ago
        Non-sequitur. I'm talking about what it would take for these machines to be truly useful.
    - mrfinn 3y ago
      I said here a few weeks ago that, not ChatGPT neither GPT4, the Davinci model asked me one year ago one deep question about God trough a Bible verse. I kind of knew the answer... or MY answer, because personally I've been questioning spiritual knowledge along all my life. And the community largely ignored the issue. Really, I don't mean to incommode anyone but I really think that GPT is much, really much more than a text auto-completer.
wmf 3y ago
Yes, PaLM and Bard have existed internally for years but Google just didn't release them. Imagen, which is better than Midjourney or Stable Diffusion, also hasn't been released.
williamstein 3y ago
All but one of the eight authors of that paper left Google, as laid out in this thread: https://twitter.com/Contrary_Res/status/1621203016903954438?lang=en https://twitter.com/Contrary_Res/status/1621203016903954438?... For example, "(1) Aidan Gomez (@aidangomezzz) left google and founded @CohereAI in September 2019".

by mrfinn 3y ago

skyhlar 3y ago

Well the paper only talks about translation, the authors never seem to have considered generalizing the architecture to other problems.

mrfinn 3y ago
Really? Because LaMDA looks like an effort in that direction but Google never made it publicly available AFAIK. Security concerns maybe? PS. Just read the Lemoine's transcriptions. Just wow.
- akomtu 3y ago
  Big corps are risk averse. When you're at the top of the pyramid, there's nowhere else to climb, but there are plently of opportunities to fall, so you move very carefully and defensively. The lowly climbers at the bottom have nowhere else to fall, and have plenty of reasons to climb, so they climb in the most careless and brazen manner. Eventually they dislodge the lazy ass at the top and become risk averse. The winner of this show is the owner of the pyramid who sells food to the climbers.

add-sub-mul-div 3y ago

In Minority Report we saw a guy controlling a computer interface by waving his arms in the air, and ever since we've had people talking about how cool it would be to have that in real life. Well, actually, no it wouldn't. In practice your arms would get tired as hell pretty quickly. A mouse on a table surface is actually a pretty great solution, which is why it's lasted. People are drawn to things that look cool and innovative for the sake of it, but we're not good at thinking ahead to how well they'd work in practice. I've been searching the web for so long that it's like breathing. I type a few keywords, hop between the first few results very fast, get what I need and get out. Maybe I have to repeat. Few searches are more complicated than that. I've tried replacing that flow with AI. It's slower to input, slower to get results, the results are opaque with regards to where they came from and how trustworthy they are. And the results aren't better, in my experience as someone for who finds search to have been solved and painless for many years. Even asking one question of an LLM takes more time and effort than a traditional search, and the problem is compounded when I have to iterate on a search. (Also it sucks to cede the kind of power to search providers that gives them leverage to force us to be logged in and monetize what's always been free, but those are just random complaints.) So, going back to your question. My guess is that Google has always known this isn't a better overall experience, once you get over the initial wow factor. But maybe they underestimated how much that wow factor would allow true competition to materialize before the rest of us would figure it out and lose interest.

boringuser2 3y ago
You're right. So far, the only use-case I've seen for LLMs are basically code completion.
- anshumankmr 3y ago
  It is usually good for auto complete in general, plus summarizing large blocks of non technical text. Probably once ChatGPT is open to fine tuning, we will be able to see how that can be leveraged as well.
  - boringuser2 3y ago
    The biggest failure of these LLMs it seems is the restricted context. It would need the ability to contain infinite context to be genuinely useful in my opinion.
    - falcor84 3y ago
      Do you think that your fellow man has infinite context about your situation?
      - boringuser2 3y ago
        Non-sequitur. I'm talking about what it would take for these machines to be truly useful.
  - mrfinn 3y ago
    I said here a few weeks ago that, not ChatGPT neither GPT4, the Davinci model asked me one year ago one deep question about God trough a Bible verse. I kind of knew the answer... or MY answer, because personally I've been questioning spiritual knowledge along all my life. And the community largely ignored the issue. Really, I don't mean to incommode anyone but I really think that GPT is much, really much more than a text auto-completer.

wmf 3y ago

Yes, PaLM and Bard have existed internally for years but Google just didn't release them. Imagen, which is better than Midjourney or Stable Diffusion, also hasn't been released.

williamstein 3y ago

All but one of the eight authors of that paper left Google, as laid out in this thread: https://twitter.com/Contrary_Res/status/1621203016903954438?lang=en https://twitter.com/Contrary_Res/status/1621203016903954438?... For example, "(1) Aidan Gomez (@aidangomezzz) left google and founded @CohereAI in September 2019".