4 ms·

Been shouting this for over a year now. We're training AI to be convincing, not to be actually helpful. We're sampling the wrong distributions.

by __loam 2y ago

danielbln 2y ago
I find them very helpful, personally.
- sussmannbaka 2y ago
  Understandable, they have been trained to convince you of their helpfulness.
  - danielbln 2y ago
    If they convinced me of their helpfulness, and their output is actually helpful in solving my problems.. well, if it walks like a duck and quacks like a duck, and all that.
    - tpoacher 2y ago
      if it walks like a duck and it quacks like a duck, then it lacks strong typing.
    - Nullabillity 2y ago
      "Appears helpful" and "is helpful" are two very different properties, as it turns out.
      - snapcaster 2y ago
        Sometimes, but that's an edge case that doesn't seem to impact the productivity boosts from LLMs
        __loam 2y ago
        It doesn't until it does. Productivity isn't the only or even the most important metric, at least in software dev.
        snapcaster 2y ago
        Can you be more specific with like examples or something?
  - exe34 2y ago
    https://xkcd.com/810/ https://xkcd.com/810/
  - djeastm 2y ago
    This is true, but part of that convincing is actually providing at least some amount of response that is helpful and moving you forward. I have to use coding as an example, because that's 95% of my use cases. I type in a general statement of the problem I'm having and within seconds, I get back a response that speaks my language and provides me with some information to ingest. Now, I don't know for sure if everything sentence I read in the response is correct, but let's say that 75% of what I read aligns with what I currently know to be true. If I were to ask a real expert, I'd possibly understand or already know 75% of what they're telling me, as well, with the other 25% still to be understood and thus trusting the expert. But either with AI or a real expert, for coding at least, that 25% will be easily testable. I go and implement and see if it passes my test. If it does, great. If not, at least I have tried something and gotten farther down the road in my problem solving. Since AI generally does that for me, I am convinced of their helpfulness because it moves me along.
dgb23 2y ago
Depends on who you ask. Advertisement and propaganda is not necessarily helpful for consumers, but just needs to be convincing in order to be helpful for producers.
- khafra 2y ago
  It would be interesting to see RL on a chatbot that's the last stage of a sales funnel for some high-volume item--it'd have fast, real-world feedback on how convincing it is, in the form of a purchase decision.
iamatoool 2y ago
Sideways eye look at leetcode culture
tpoacher 2y ago
s / AI / Marketing|Ads|Consultants|Experts|Media|Politicians|...
HarHarVeryFunny 2y ago
If what you want is auto-complete (e.g. CoPilot, or natural language search) then LLMs are built for that, and useful. If what you want it AGI then design an architecture with the necessary moving parts! Current approach reminds of the joke of the drunk looking for his dropped cars keys under the street lamp because "it's bright here", rather than near where he actually dropped them. It seems folk have spent years trying to come up with alternate learning mechanisms to gradient descent (or RL), and having failed are now trying to use SGD/pre-training for AGI "because it's what we've got", as opposed to doing the hard work of designing the type of always-on online learning algorithm that AGI actually requires.
- adroniser 2y ago
  Isn't RL the algorithm we want basically?
  - HarHarVeryFunny 2y ago
    Want for what? RL is one way to implement goal directed behavior (making decisions now that hopefully will lead towards a later reward), but I doubt this is the actual mechanism at play when we exhibit goal directed behavior ourselves. Something more RL-like may potentially be used in our cerebellum (not cortex) to learn fine motor skills. Some of the things that are clearly needed for human-like AGI are things like the ability to learn incrementally and continuously (the main ways we learn are by trial and error, and by copying), as opposed to pre-training with SGD, things like working memory, ability to think to arbitrary depth before acting, innate qualities like curiosity and boredom to drive learning and exploration, etc. The Transformer architecture underlying all of today's LLMs have none of the above, not surprising since it was never intended as a cognitive architecture - it was designed for seq2seq use such as language models (LLMs). So, no, I don't think RL is the answer to AGI, and note that DeepMind who had previously believed that have since largely switched to LLMs in the pursuit of AGI, and are mostly using RL as part of more specialized machine learning applications such as AlphaGo and AlphaFold.
    - adroniser 2y ago

Been shouting this for over a year now. We're training AI to be convincing, not to be actually helpful. We're sampling the wrong distributions.

by __loam 2y ago

Been shouting this for over a year now. We're training AI to be convincing, not to be actually helpful. We're sampling the wrong distributions.

danielbln 2y ago

I find them very helpful, personally.

sussmannbaka 2y ago
Understandable, they have been trained to convince you of their helpfulness.
- danielbln 2y ago
  If they convinced me of their helpfulness, and their output is actually helpful in solving my problems.. well, if it walks like a duck and quacks like a duck, and all that.
  - tpoacher 2y ago
    if it walks like a duck and it quacks like a duck, then it lacks strong typing.
  - Nullabillity 2y ago
    "Appears helpful" and "is helpful" are two very different properties, as it turns out.
    - snapcaster 2y ago
      Sometimes, but that's an edge case that doesn't seem to impact the productivity boosts from LLMs
      - __loam 2y ago
        It doesn't until it does. Productivity isn't the only or even the most important metric, at least in software dev.
        snapcaster 2y ago
        Can you be more specific with like examples or something?
- exe34 2y ago
  https://xkcd.com/810/ https://xkcd.com/810/
- djeastm 2y ago
  This is true, but part of that convincing is actually providing at least some amount of response that is helpful and moving you forward. I have to use coding as an example, because that's 95% of my use cases. I type in a general statement of the problem I'm having and within seconds, I get back a response that speaks my language and provides me with some information to ingest. Now, I don't know for sure if everything sentence I read in the response is correct, but let's say that 75% of what I read aligns with what I currently know to be true. If I were to ask a real expert, I'd possibly understand or already know 75% of what they're telling me, as well, with the other 25% still to be understood and thus trusting the expert. But either with AI or a real expert, for coding at least, that 25% will be easily testable. I go and implement and see if it passes my test. If it does, great. If not, at least I have tried something and gotten farther down the road in my problem solving. Since AI generally does that for me, I am convinced of their helpfulness because it moves me along.

dgb23 2y ago

Depends on who you ask. Advertisement and propaganda is not necessarily helpful for consumers, but just needs to be convincing in order to be helpful for producers.

khafra 2y ago
It would be interesting to see RL on a chatbot that's the last stage of a sales funnel for some high-volume item--it'd have fast, real-world feedback on how convincing it is, in the form of a purchase decision.

iamatoool 2y ago

Sideways eye look at leetcode culture

tpoacher 2y ago

HarHarVeryFunny 2y ago

If what you want is auto-complete (e.g. CoPilot, or natural language search) then LLMs are built for that, and useful. If what you want it AGI then design an architecture with the necessary moving parts! Current approach reminds of the joke of the drunk looking for his dropped cars keys under the street lamp because "it's bright here", rather than near where he actually dropped them. It seems folk have spent years trying to come up with alternate learning mechanisms to gradient descent (or RL), and having failed are now trying to use SGD/pre-training for AGI "because it's what we've got", as opposed to doing the hard work of designing the type of always-on online learning algorithm that AGI actually requires.

adroniser 2y ago
Isn't RL the algorithm we want basically?
- HarHarVeryFunny 2y ago
  Want for what? RL is one way to implement goal directed behavior (making decisions now that hopefully will lead towards a later reward), but I doubt this is the actual mechanism at play when we exhibit goal directed behavior ourselves. Something more RL-like may potentially be used in our cerebellum (not cortex) to learn fine motor skills. Some of the things that are clearly needed for human-like AGI are things like the ability to learn incrementally and continuously (the main ways we learn are by trial and error, and by copying), as opposed to pre-training with SGD, things like working memory, ability to think to arbitrary depth before acting, innate qualities like curiosity and boredom to drive learning and exploration, etc. The Transformer architecture underlying all of today's LLMs have none of the above, not surprising since it was never intended as a cognitive architecture - it was designed for seq2seq use such as language models (LLMs). So, no, I don't think RL is the answer to AGI, and note that DeepMind who had previously believed that have since largely switched to LLMs in the pursuit of AGI, and are mostly using RL as part of more specialized machine learning applications such as AlphaGo and AlphaFold.
  - adroniser 2y ago