3 ms·

I explained all it could do, and I felt I was doing great. But the moment I mentioned that it can get things wrong and sound totally confident, pretty much the

by demarq 3y ago

I explained all it could do, and I felt I was doing great. But the moment I mentioned that it can get things wrong and sound totally confident, pretty much the conversation was me trying to justify why it’s still useful and I didn’t get anywhere beyond that. I was shocked because I felt that being occasionally wrong would be a foot note, but to the person I was talking to its entire usefulness hinged on it being fully reliable. For the capabilities we’re getting surely, the occasional wrong answer is barely a sacrifice!

cmrdporcupine 3y ago
So; I think many people in our sector have a progress-bias from years of Moore's law etc. and run under the assumption that all things tech that display problems (in this case "incorrect information") will just resolve with time and progress. People outside of tech don't necessarily think this way. So hearing that "it produces incorrect answers" is kind of a deal breaker, no? Who is right in this case? I actually think that the LLM approach has limits that we could hit the wall of, and for that technique at least never get past the "I'm just making shit up" problem, and in fact the skeptics are quite right. LLMs are exciting as an automatic language content generation tool, they chain words together in ways that sound like humans, and extract reasoning patterns that look like human reasoning. But they're not reasoning, it doesn't take much to trip them up in basic argumentation. But because they look like they're "thinking" some people with a tech-optimist bias get excited and just assume that the problems will resolve themselves. They could be very wrong, there could in fact be very strong limits to this approach. ... More worrisome is if LLMs because omnipresent despite having this flaw, and we just accept bullshit from computers the way we seemingly now accept complete bullshit from politicians and businessmen....
- azangru 3y ago
  > People outside of tech don't necessarily think this way. So hearing that "it produces incorrect answers" is kind of a deal breaker, no? I think it's a deal-breaker for many people inside tech either :-) It looks like it may be useful in doing grunt work in the field where you are an expert and can check and correct anything that it produces. Where people expect it to be useful though is in providing them with information they do not already know; and the fact that you cannot trust anything it says makes it unusable for this case.
  - cmrdporcupine 3y ago
    Yep. CoPilot for example is great for making grunt code, filling in the blanks. But it's really crappy at anything that requires reasoning through a problem. It is our responsibility as tech professionals to recognize this and explain it to laymen otherwise we're in trouble. I've said it before, and I'll say it again: It's really really bad that these systems are made to speak in first person, that they're often given "names", that they use human voices, and that they present authority. This is irresponsible engineering from a social and ethics POV, and our "profession" such as it is, should be held to task for it.
ajkjk 3y ago
Well, 90% of the applications people are shouting about and raising money over are complete non-starters as long as 'being wrong sometimes' is a feature. Not that there isn't a good use for a generative system that lies, but.. Everyone is just going around pretending it doesn't, because it's convenient for them, and it feels like a big dumb joke.
mattlondon 3y ago
Computers are not supposed to get stuff wrong. Being right some of the time just isn't good enough for people who expect their calculator to always give them the right answers. I think we can all agree that LLMs, as they stand today, are useless for anything apart from "creative tasks" where accuracy and facts simply don't matter. So things like writing fan fiction it is great at, but for anything meaningful it is utter tripe. No one can deny this.
- thaw13579 3y ago
  They’re useful for tasks that are easy to verify but hard to generate, and code is one meaningful example. It seems trite to say after so many others, but I built a web app in one day by pair coding with GPT-4, which I am sure would have taken weeks, mostly learning the quirks of the various frameworks involved by sifting through the noise of the web. The LLM wasn’t correct 100% of the time, but I could iterate with it when there was a problem and debug. It probably helps that I was using hugely popular libraries, but I think this means there’s a ton of promise in fine-tuning these models on internal codebases/docs.
thaw13579 3y ago
Yes, the goal post seems to move quickly with LLMs. Search engines provide relevant documents, not truthful ones, and I don’t know a single person who isn’t occasionally wrong, often confidently. I suppose it’s good to set a high bar with these models though, since it gives some incentive to improve them.

by demarq 3y ago

cmrdporcupine 3y ago

So; I think many people in our sector have a progress-bias from years of Moore's law etc. and run under the assumption that all things tech that display problems (in this case "incorrect information") will just resolve with time and progress. People outside of tech don't necessarily think this way. So hearing that "it produces incorrect answers" is kind of a deal breaker, no? Who is right in this case? I actually think that the LLM approach has limits that we could hit the wall of, and for that technique at least never get past the "I'm just making shit up" problem, and in fact the skeptics are quite right. LLMs are exciting as an automatic language content generation tool, they chain words together in ways that sound like humans, and extract reasoning patterns that look like human reasoning. But they're not reasoning, it doesn't take much to trip them up in basic argumentation. But because they look like they're "thinking" some people with a tech-optimist bias get excited and just assume that the problems will resolve themselves. They could be very wrong, there could in fact be very strong limits to this approach. ... More worrisome is if LLMs because omnipresent despite having this flaw, and we just accept bullshit from computers the way we seemingly now accept complete bullshit from politicians and businessmen....

azangru 3y ago
> People outside of tech don't necessarily think this way. So hearing that "it produces incorrect answers" is kind of a deal breaker, no? I think it's a deal-breaker for many people inside tech either :-) It looks like it may be useful in doing grunt work in the field where you are an expert and can check and correct anything that it produces. Where people expect it to be useful though is in providing them with information they do not already know; and the fact that you cannot trust anything it says makes it unusable for this case.
- cmrdporcupine 3y ago
  Yep. CoPilot for example is great for making grunt code, filling in the blanks. But it's really crappy at anything that requires reasoning through a problem. It is our responsibility as tech professionals to recognize this and explain it to laymen otherwise we're in trouble. I've said it before, and I'll say it again: It's really really bad that these systems are made to speak in first person, that they're often given "names", that they use human voices, and that they present authority. This is irresponsible engineering from a social and ethics POV, and our "profession" such as it is, should be held to task for it.

ajkjk 3y ago

Well, 90% of the applications people are shouting about and raising money over are complete non-starters as long as 'being wrong sometimes' is a feature. Not that there isn't a good use for a generative system that lies, but.. Everyone is just going around pretending it doesn't, because it's convenient for them, and it feels like a big dumb joke.

mattlondon 3y ago

Computers are not supposed to get stuff wrong. Being right some of the time just isn't good enough for people who expect their calculator to always give them the right answers. I think we can all agree that LLMs, as they stand today, are useless for anything apart from "creative tasks" where accuracy and facts simply don't matter. So things like writing fan fiction it is great at, but for anything meaningful it is utter tripe. No one can deny this.

thaw13579 3y ago
They’re useful for tasks that are easy to verify but hard to generate, and code is one meaningful example. It seems trite to say after so many others, but I built a web app in one day by pair coding with GPT-4, which I am sure would have taken weeks, mostly learning the quirks of the various frameworks involved by sifting through the noise of the web. The LLM wasn’t correct 100% of the time, but I could iterate with it when there was a problem and debug. It probably helps that I was using hugely popular libraries, but I think this means there’s a ton of promise in fine-tuning these models on internal codebases/docs.

thaw13579 3y ago

Yes, the goal post seems to move quickly with LLMs. Search engines provide relevant documents, not truthful ones, and I don’t know a single person who isn’t occasionally wrong, often confidently. I suppose it’s good to set a high bar with these models though, since it gives some incentive to improve them.