5 ms·

Google had every opportunity to capitalize on the architectural innovation and failed tremendously.

by captaintobs 3y ago

riku_iki 3y ago
they shared and opened innovations, "Open"AI grabbed, built on top, and keeps closed.
- ikekkdcjkfke 3y ago
  How much ai tech does google keep for itself though. Nerfing search also
  - riku_iki 3y ago
    It is not clear if they have any other secret ai tech inside besides everything they published: tensorflow, t5.
jimsimmons 3y ago
People severely underestimate how much OpenAI moved the field forward. If OpenAI didn’t exist, we’d be talking about AGI as a taboo word. Just because someone had a significant paper along the way doesn’t mean that they failed to capitalize. You can and should respect the contributions made by others since.
- junon 3y ago
  AGI is still very far away and even then was never taboo.
  - flangola7 3y ago
    GPT is highly general
    - junon 3y ago
      No, it's not. It's a transformation model. It cannot prioritize, remember things, create novel ideas, nor does it have an online processing model of any kind. It is bounded to the data on which it was trained, which as of GPT 3.5 caps out at a few years ago. We have a _very_ long way to go to AGI. Diluting down the meaning of the term "AGI" just to say we've reached it is, almost literally, moving the goalposts.
      - meltyness 3y ago
        Please describe what you mean by "an online processing model of any kind" ... not that I don't disagree with the rest of what you think you've said here, but I think that this part of your claim is egregiously paucious.
        junon 3y ago
        Meaning it's not going to be trained as it's being used. It can respond to context but it's not going to be continuously improving itself.
        meltyness 3y ago
        Let's estimate that it's current input limit is 20 or so paragraphs, and that it's capable of zero-shot learning as observed by many of the experimenters on the system. For every 20 paragraph thesis about the world around us, how often do you think someone meaningfully innovates and changes meaningfully our understanding of the world around us, and it's fundamentals? Consider, like, classical logic. It's pretty much done. You have the universal gate sets, and any novel functions are just going to be an mxn-mux. Most other such general concepts are essentially figured out and have had their applications recorded, explained, demonstrated. Sure, biographical information and history changes on a whim, but no one is requesting artificial clairvoyant intelligence. I suppose you could have the model pontificate and assess the truth of its own argumentation (ask it to generate outer-product concepts, ask it to reassess assumptions in current models) but it will quickly learn its own language that will be mutually unintelligible with the corpus of current human knowledge it has been trained on. Why does the underlying model need to change on a whim, in your opinion?... on what whim?
  - hackinthebochs 3y ago
    I was saying less than a decade before ChatGPT was released. I'm down to 5 years if someone wanted to really throw money at the problem.
    - YeGoblynQueenne 3y ago
      Less than 5 years to AGI? An "AGI" that has to be retrained from scratch, with more data and more compute, and with a modified architecture, every time we want to improve its performance? An "AGI" that forgets every interaction as soon as it's over? Doesn't sound too general, or too practical, to me, and certainly not very much of an "intelligence". If OpenAI or anyone else finds a way to get their models to train themselves over and over at minimal cost, like humans do, then we can maybe talk about a "general" "AI". Until then all they have is a system with a static performance that does not improve with experience, that is not even machine learning anymore and that is very far from whatever people imagine when they say "AGI".
      - JeremyBanks 3y ago
        [dead]
      - YeGoblynQueenne 3y ago
        Hey, JerremyBanks [dead], check out my comment history. Should help answer your question ;)
      - hackinthebochs 3y ago
        I agree with all that. I don't think LLMs are proto-AGI, in the sense that simply scaling parameters/data/compute will result in a qualitative change towards AGI. But I think LLMs are a significant component of an AGI. What they lack, feedback control, addressable online memory, planning, and recurrent execution, are fairly straightforward engineering problems. Transformers/self-attention solve the problem of scaling capabilities to the available data/compute. The big problem left is to have the right architecture such that AGI is in the solution-space.
        3y ago

Google had every opportunity to capitalize on the architectural innovation and failed tremendously.

riku_iki 3y ago

they shared and opened innovations, "Open"AI grabbed, built on top, and keeps closed.

ikekkdcjkfke 3y ago
How much ai tech does google keep for itself though. Nerfing search also
- riku_iki 3y ago
  It is not clear if they have any other secret ai tech inside besides everything they published: tensorflow, t5.

jimsimmons 3y ago

People severely underestimate how much OpenAI moved the field forward. If OpenAI didn’t exist, we’d be talking about AGI as a taboo word. Just because someone had a significant paper along the way doesn’t mean that they failed to capitalize. You can and should respect the contributions made by others since.

junon 3y ago
AGI is still very far away and even then was never taboo.
- flangola7 3y ago
  GPT is highly general
  - junon 3y ago
    No, it's not. It's a transformation model. It cannot prioritize, remember things, create novel ideas, nor does it have an online processing model of any kind. It is bounded to the data on which it was trained, which as of GPT 3.5 caps out at a few years ago. We have a _very_ long way to go to AGI. Diluting down the meaning of the term "AGI" just to say we've reached it is, almost literally, moving the goalposts.
    - meltyness 3y ago
      Please describe what you mean by "an online processing model of any kind" ... not that I don't disagree with the rest of what you think you've said here, but I think that this part of your claim is egregiously paucious.
      - junon 3y ago
        Meaning it's not going to be trained as it's being used. It can respond to context but it's not going to be continuously improving itself.
        meltyness 3y ago
        Let's estimate that it's current input limit is 20 or so paragraphs, and that it's capable of zero-shot learning as observed by many of the experimenters on the system. For every 20 paragraph thesis about the world around us, how often do you think someone meaningfully innovates and changes meaningfully our understanding of the world around us, and it's fundamentals? Consider, like, classical logic. It's pretty much done. You have the universal gate sets, and any novel functions are just going to be an mxn-mux. Most other such general concepts are essentially figured out and have had their applications recorded, explained, demonstrated. Sure, biographical information and history changes on a whim, but no one is requesting artificial clairvoyant intelligence. I suppose you could have the model pontificate and assess the truth of its own argumentation (ask it to generate outer-product concepts, ask it to reassess assumptions in current models) but it will quickly learn its own language that will be mutually unintelligible with the corpus of current human knowledge it has been trained on. Why does the underlying model need to change on a whim, in your opinion?... on what whim?
- hackinthebochs 3y ago
  I was saying less than a decade before ChatGPT was released. I'm down to 5 years if someone wanted to really throw money at the problem.
  - YeGoblynQueenne 3y ago
    Less than 5 years to AGI? An "AGI" that has to be retrained from scratch, with more data and more compute, and with a modified architecture, every time we want to improve its performance? An "AGI" that forgets every interaction as soon as it's over? Doesn't sound too general, or too practical, to me, and certainly not very much of an "intelligence". If OpenAI or anyone else finds a way to get their models to train themselves over and over at minimal cost, like humans do, then we can maybe talk about a "general" "AI". Until then all they have is a system with a static performance that does not improve with experience, that is not even machine learning anymore and that is very far from whatever people imagine when they say "AGI".
    - JeremyBanks 3y ago
      [dead]
    - YeGoblynQueenne 3y ago
      Hey, JerremyBanks [dead], check out my comment history. Should help answer your question ;)
    - hackinthebochs 3y ago
      I agree with all that. I don't think LLMs are proto-AGI, in the sense that simply scaling parameters/data/compute will result in a qualitative change towards AGI. But I think LLMs are a significant component of an AGI. What they lack, feedback control, addressable online memory, planning, and recurrent execution, are fairly straightforward engineering problems. Transformers/self-attention solve the problem of scaling capabilities to the available data/compute. The big problem left is to have the right architecture such that AGI is in the solution-space.
      - 3y ago