3 ms·

Show me the AI chat system from a decade ago that could hallucinate a git repo. Edit: let me be more careful: Show me the AI chat system from a decade ago tha

Show me the AI chat system from a decade ago that could hallucinate a git repo. Edit: let me be more careful: Show me the AI chat system from a decade ago that wasn't specifically designed to hallucinate git repos, but would hallucinate a git repo in response a detailed natural language prompt. It should have been good enough in its output to cause the user to write more prompts in an attempt to gauge whether the AI system is querying the internet in realtime to generate its response.

jchw 4y ago
Decade ago, definitely not. But the answer to that question, time aside, is, well, GPT-3. No doubt ChatGPT is better, but it does seem like a decent bit of the improvement was UX related, since with the right prompts people have shown similarly miraculous output from GPT-3. Of course, I'm pretty sure the quoted person is referring to the fact that Google and others likely have similarly capable LLMs already that they simply won't release the models for. I can see the argument that new architectures and designs can be innovative, but that simply being the one to throw a million dollars of compute time into training is much less so, if that's the kind of argument.
- hnfong 4y ago
  Yes, but along that reasoning the criticism that "ChatGPT is not particularly innovative" is a bit disingenuous in the sense that the general public has never had easy access to these models until ChatGPT arrived. Besides, even though the newer GPT models presumably just extended earlier ones with similar techniques but better hardware and data, it wasn't a foregone conclusion how powerful (from the user perspective) the models were going to be. Somebody had to put the resources into training the models. It's like saying human brains aren't particularly powerful because even the simplest animals have neurons and it's obvious that a more powerful brain could be built with more of them.
  - jchw 4y ago
    I don't think the point here is to downplay the significance of ChatGPT as a product, but to point out that OpenAI is likely not uniquely resourced in this regard. If anything, it sounds like what they're trying to say is that ChatGPT is only the tip of the iceberg, and the reason why it came out of OpenAI and not Meta or Google is more a function of other factors like reputational damage, etc. The headline does hit as a bit sensationalist or dismissive if you read it literally, as if it's claiming the work that went into ChatGPT was all easy and no big deal. Rather though, it seems like it's just trying to point out to the zeitgeist currently enamored with ChatGPT and all of the new possibilities, that other companies also have this capability today and that transformers have been more of an evolutionary process than a revolutionary process. Maybe I'm reading it too leniently, but that was my takeaway.
    - visarga 4y ago
      Meta should RLHF or RLAIF their model Galactica and put it up again. It was interesting to use to sample ideas. Just tune it down a bit with the confident language.
    - hnfong 4y ago
      That's probably what they're trying to say on the surface. I'm a bit less charitable and can't help but feel that there's also a saltiness factor that somebody else got all the attention -- but, even to your point, "other companies also have this capability today and that transformers have been more of an evolutionary process than a revolutionary process" - we probably should ask why game changing research like this has been placed under the veil of secrecy and kept hidden from the public for so long. It seems like AI researchers have a collective tendency to invent reasons to hide the latest advances from the public. Then when ChatGPT is released publicly, of course the public is astounded by its capabilities. I can't help recalling the incident where Blake Lemoine claimed Google's LaMDA was sentient. I don't necessarily agree with his claims, but people were generally quicker to dismiss his claims as nonsense than I expected. It is quite clear that no matter whether the sentient claims were valid, Google did have an AI that's on par with ChatGPT today. That should have been news itself, but all I recall seeing was experts shooting down (aka "debunking") Lemoine without recognizing that there was something more to the story. FWIW I don't think ChatGPT as-is is sentient (if only because OpenAI probably tweaked it until it couldn't talk about itself), but this technology is still hugely impactful. Maybe the AI researchers who thinks ChatGPT is overhyped are just a bit out of touch with the general public after years of having access to private models.

jchw 4y ago

Decade ago, definitely not. But the answer to that question, time aside, is, well, GPT-3. No doubt ChatGPT is better, but it does seem like a decent bit of the improvement was UX related, since with the right prompts people have shown similarly miraculous output from GPT-3. Of course, I'm pretty sure the quoted person is referring to the fact that Google and others likely have similarly capable LLMs already that they simply won't release the models for. I can see the argument that new architectures and designs can be innovative, but that simply being the one to throw a million dollars of compute time into training is much less so, if that's the kind of argument.

hnfong 4y ago
Yes, but along that reasoning the criticism that "ChatGPT is not particularly innovative" is a bit disingenuous in the sense that the general public has never had easy access to these models until ChatGPT arrived. Besides, even though the newer GPT models presumably just extended earlier ones with similar techniques but better hardware and data, it wasn't a foregone conclusion how powerful (from the user perspective) the models were going to be. Somebody had to put the resources into training the models. It's like saying human brains aren't particularly powerful because even the simplest animals have neurons and it's obvious that a more powerful brain could be built with more of them.
- jchw 4y ago
  I don't think the point here is to downplay the significance of ChatGPT as a product, but to point out that OpenAI is likely not uniquely resourced in this regard. If anything, it sounds like what they're trying to say is that ChatGPT is only the tip of the iceberg, and the reason why it came out of OpenAI and not Meta or Google is more a function of other factors like reputational damage, etc. The headline does hit as a bit sensationalist or dismissive if you read it literally, as if it's claiming the work that went into ChatGPT was all easy and no big deal. Rather though, it seems like it's just trying to point out to the zeitgeist currently enamored with ChatGPT and all of the new possibilities, that other companies also have this capability today and that transformers have been more of an evolutionary process than a revolutionary process. Maybe I'm reading it too leniently, but that was my takeaway.
  - visarga 4y ago
    Meta should RLHF or RLAIF their model Galactica and put it up again. It was interesting to use to sample ideas. Just tune it down a bit with the confident language.
  - hnfong 4y ago
    That's probably what they're trying to say on the surface. I'm a bit less charitable and can't help but feel that there's also a saltiness factor that somebody else got all the attention -- but, even to your point, "other companies also have this capability today and that transformers have been more of an evolutionary process than a revolutionary process" - we probably should ask why game changing research like this has been placed under the veil of secrecy and kept hidden from the public for so long. It seems like AI researchers have a collective tendency to invent reasons to hide the latest advances from the public. Then when ChatGPT is released publicly, of course the public is astounded by its capabilities. I can't help recalling the incident where Blake Lemoine claimed Google's LaMDA was sentient. I don't necessarily agree with his claims, but people were generally quicker to dismiss his claims as nonsense than I expected. It is quite clear that no matter whether the sentient claims were valid, Google did have an AI that's on par with ChatGPT today. That should have been news itself, but all I recall seeing was experts shooting down (aka "debunking") Lemoine without recognizing that there was something more to the story. FWIW I don't think ChatGPT as-is is sentient (if only because OpenAI probably tweaked it until it couldn't talk about itself), but this technology is still hugely impactful. Maybe the AI researchers who thinks ChatGPT is overhyped are just a bit out of touch with the general public after years of having access to private models.