7 ms·

> “So it tells you wrong answers? Sounds utterly pointless”. I was flabbergasted. Could you tell a bit more about why this objection surprised you so much? I a

> “So it tells you wrong answers? Sounds utterly pointless”. I was flabbergasted. Could you tell a bit more about why this objection surprised you so much? I am often seeing this in the same tech circles that rejected the web3 craze; and I have to confess, it does sound reasonable to me. There is a recent PR opened in the MDN docs repository after MDN added the "AI explain" button, pointing out how utterly useless a AI guide is if it gives you incorrect answers and you do not have enough knowledge to catch it [0]. [0] - https://github.com/mdn/yari/issues/9208 https://github.com/mdn/yari/issues/9208

mannykannot 3y ago
There is nothing wrong about the specific objection but, to borrow an analogy from Scott Aaronson, it is rather like dismissing the Wrights’ early airplanes as being slower than a train and only carrying one passenger. What is missing is an appreciation for how intractable the problem seemed five years before.
- rsynnott 3y ago
  Bear in mind that for every Wright Brother, there’s a Ferdinand Von Zeppelin. Most heavily hyped technologies don’t have a huge long-term impact.
- ajkjk 3y ago
  It's not really like that? A small plane with one passenger is a concept that you can extrapolate to a bigger, faster plane. Afaik an often-wrong generative AI is _not_ a concept that extrapolates to a never-wrong generative AI: that's just not how it works at a fundamental level. Although presumably very smart folks are working on it.
  - mannykannot 3y ago
    I see that the analogy is having the unintended side-effect of apparently being predicated on a supposed utility of ChatGPT’s direct descendants. The point I want to get across is how difficult anything like it seemed before, regardless of whether it begets anything markedly better afterwards.
  - thaw13579 3y ago
    For me, it’s already a jetliner in the context of coding assist. It’s correct more often than top hits provided by a top search engine (and any coworker), and it is a very enjoyable user experience (no ads and SEO garbage to filter out). I’d say the wright brothers version was something like BERT or earlier GPTs.
- mikhmha 3y ago
  Can we stop making analogies like this. It’s so bizarre. The circumstances around each invention are completely different. To always reference the conditions of some past technology is to be stuck in this bizarre form of thinking. A few days ago some poster tried to compare denying LLM’s to being a naysayer in Gallileos time. Is this all we can do? Make references to past events and fail to evaluate the present properly?
  - mannykannot 3y ago
    Of course every case of anything is different in detail. On the other hand, identifying patterns is often useful in understanding human reactions to change. Do you disagree with this, or do you think there is something special about inventions? My analysis here is that the young person in question appeared to focus only on current utility. You have not yet explained why you say it is a failed analysis of why that response may seem surprising.
- nerdbert 3y ago
  Except that people could explain how you could make an airplane faster and larger. Nobody can explain how you can make an LLM understand what it's saying.
  - bondarchuk 3y ago
    A lot depends on the definition of "understand" you want to use. According to some definitions there is already some kind of understanding going on in the current generation of LLMs, according to others true understanding requires a soul in the Christian sense and thus it is unattainable an LLM, and of course there is a spectrum of opinions in-between.
    - parineum 3y ago
      If you want to limit your range of opinions to credible domain experts speaking scientifically, the range goes from "definitely not" to "maybe a little if you have a very loose definition".
      - bondarchuk 3y ago
        Which is not that far from what I was saying. And if you take the domain to be philosophy or psychology rather than deep learning or CS, I'm sure the range will be much wider.
demarq 3y ago
I explained all it could do, and I felt I was doing great. But the moment I mentioned that it can get things wrong and sound totally confident, pretty much the conversation was me trying to justify why it’s still useful and I didn’t get anywhere beyond that. I was shocked because I felt that being occasionally wrong would be a foot note, but to the person I was talking to its entire usefulness hinged on it being fully reliable. For the capabilities we’re getting surely, the occasional wrong answer is barely a sacrifice!
- cmrdporcupine 3y ago
  So; I think many people in our sector have a progress-bias from years of Moore's law etc. and run under the assumption that all things tech that display problems (in this case "incorrect information") will just resolve with time and progress. People outside of tech don't necessarily think this way. So hearing that "it produces incorrect answers" is kind of a deal breaker, no? Who is right in this case? I actually think that the LLM approach has limits that we could hit the wall of, and for that technique at least never get past the "I'm just making shit up" problem, and in fact the skeptics are quite right. LLMs are exciting as an automatic language content generation tool, they chain words together in ways that sound like humans, and extract reasoning patterns that look like human reasoning. But they're not reasoning, it doesn't take much to trip them up in basic argumentation. But because they look like they're "thinking" some people with a tech-optimist bias get excited and just assume that the problems will resolve themselves. They could be very wrong, there could in fact be very strong limits to this approach. ... More worrisome is if LLMs because omnipresent despite having this flaw, and we just accept bullshit from computers the way we seemingly now accept complete bullshit from politicians and businessmen....
  - azangru 3y ago
    > People outside of tech don't necessarily think this way. So hearing that "it produces incorrect answers" is kind of a deal breaker, no? I think it's a deal-breaker for many people inside tech either :-) It looks like it may be useful in doing grunt work in the field where you are an expert and can check and correct anything that it produces. Where people expect it to be useful though is in providing them with information they do not already know; and the fact that you cannot trust anything it says makes it unusable for this case.

mannykannot 3y ago

There is nothing wrong about the specific objection but, to borrow an analogy from Scott Aaronson, it is rather like dismissing the Wrights’ early airplanes as being slower than a train and only carrying one passenger. What is missing is an appreciation for how intractable the problem seemed five years before.

rsynnott 3y ago
Bear in mind that for every Wright Brother, there’s a Ferdinand Von Zeppelin. Most heavily hyped technologies don’t have a huge long-term impact.
ajkjk 3y ago
It's not really like that? A small plane with one passenger is a concept that you can extrapolate to a bigger, faster plane. Afaik an often-wrong generative AI is _not_ a concept that extrapolates to a never-wrong generative AI: that's just not how it works at a fundamental level. Although presumably very smart folks are working on it.
- mannykannot 3y ago
  I see that the analogy is having the unintended side-effect of apparently being predicated on a supposed utility of ChatGPT’s direct descendants. The point I want to get across is how difficult anything like it seemed before, regardless of whether it begets anything markedly better afterwards.
- thaw13579 3y ago
  For me, it’s already a jetliner in the context of coding assist. It’s correct more often than top hits provided by a top search engine (and any coworker), and it is a very enjoyable user experience (no ads and SEO garbage to filter out). I’d say the wright brothers version was something like BERT or earlier GPTs.
mikhmha 3y ago
Can we stop making analogies like this. It’s so bizarre. The circumstances around each invention are completely different. To always reference the conditions of some past technology is to be stuck in this bizarre form of thinking. A few days ago some poster tried to compare denying LLM’s to being a naysayer in Gallileos time. Is this all we can do? Make references to past events and fail to evaluate the present properly?
- mannykannot 3y ago
  Of course every case of anything is different in detail. On the other hand, identifying patterns is often useful in understanding human reactions to change. Do you disagree with this, or do you think there is something special about inventions? My analysis here is that the young person in question appeared to focus only on current utility. You have not yet explained why you say it is a failed analysis of why that response may seem surprising.
nerdbert 3y ago
Except that people could explain how you could make an airplane faster and larger. Nobody can explain how you can make an LLM understand what it's saying.
- bondarchuk 3y ago
  A lot depends on the definition of "understand" you want to use. According to some definitions there is already some kind of understanding going on in the current generation of LLMs, according to others true understanding requires a soul in the Christian sense and thus it is unattainable an LLM, and of course there is a spectrum of opinions in-between.
  - parineum 3y ago
    If you want to limit your range of opinions to credible domain experts speaking scientifically, the range goes from "definitely not" to "maybe a little if you have a very loose definition".
    - bondarchuk 3y ago
      Which is not that far from what I was saying. And if you take the domain to be philosophy or psychology rather than deep learning or CS, I'm sure the range will be much wider.

demarq 3y ago

I explained all it could do, and I felt I was doing great. But the moment I mentioned that it can get things wrong and sound totally confident, pretty much the conversation was me trying to justify why it’s still useful and I didn’t get anywhere beyond that. I was shocked because I felt that being occasionally wrong would be a foot note, but to the person I was talking to its entire usefulness hinged on it being fully reliable. For the capabilities we’re getting surely, the occasional wrong answer is barely a sacrifice!

cmrdporcupine 3y ago
So; I think many people in our sector have a progress-bias from years of Moore's law etc. and run under the assumption that all things tech that display problems (in this case "incorrect information") will just resolve with time and progress. People outside of tech don't necessarily think this way. So hearing that "it produces incorrect answers" is kind of a deal breaker, no? Who is right in this case? I actually think that the LLM approach has limits that we could hit the wall of, and for that technique at least never get past the "I'm just making shit up" problem, and in fact the skeptics are quite right. LLMs are exciting as an automatic language content generation tool, they chain words together in ways that sound like humans, and extract reasoning patterns that look like human reasoning. But they're not reasoning, it doesn't take much to trip them up in basic argumentation. But because they look like they're "thinking" some people with a tech-optimist bias get excited and just assume that the problems will resolve themselves. They could be very wrong, there could in fact be very strong limits to this approach. ... More worrisome is if LLMs because omnipresent despite having this flaw, and we just accept bullshit from computers the way we seemingly now accept complete bullshit from politicians and businessmen....
- azangru 3y ago
  > People outside of tech don't necessarily think this way. So hearing that "it produces incorrect answers" is kind of a deal breaker, no? I think it's a deal-breaker for many people inside tech either :-) It looks like it may be useful in doing grunt work in the field where you are an expert and can check and correct anything that it produces. Where people expect it to be useful though is in providing them with information they do not already know; and the fact that you cannot trust anything it says makes it unusable for this case.