4 ms·

I don’t understand why this talking point keeps surfacing. LLMs are in significant usage already in many production environments and are already improving peopl

I don’t understand why this talking point keeps surfacing. LLMs are in significant usage already in many production environments and are already improving peoples’ lives and productivity. I can have a dig for more examples but I’d urge you to do a bit of research and try out the gpt-4 API for yourself. Or make some more qualified statements with specific bugbears. Equating it all to homeopathy is so confusing. - Image descriptions eg. Be My Eyes: https://www.bemyeyes.com/ https://www.bemyeyes.com/ - Summarisation and content distillation, question answering Etc: literally everywhere. This is now a solved problem (to the point of it being dull) thanks to LLMs. - Customer service chat triage. - helping students learn- eg khan academy, tutor Lily - Helping create and debug software. If you don’t think is happening then you’re either living under a rock or just close to believe people are lying about how they’ve used it?

skepticATX 3y ago
I'd push back on summarization being solved. I have extensively used GPT 3.5 and 4 for this in production, and it works very well but it still often just ignores critical aspects of the original text, and it will rarely even completely hallucinate details. And this is with a lot of iteration on the prompt. We still have to throw out > 50% of its output because a human can summarize the text much better.
dns_snek 3y ago
> - Summarisation and content distillation, question answering Etc: literally everywhere. This is now a solved problem (to the point of it being dull) thanks to LLMs. Calling this problem domain "solved" only undermines whatever point you're trying to make, unless you really believe that current LLMs which fabricate information, flip flop between answers with follow up questions, and even gaslight the user, can be called "solutions". LLMs have proven themselves to be such untrustworthy "sources" of information and knowledge that I'm struggling to understand why anyone would even try to make this particular claim. It's trivial to refute by anyone who's used them and has been thoroughly refuted by StackOverflow Developer survey which reported that only 2.8% of developers "highly trust" the output of AI tools. https://survey.stackoverflow.co/2023/#section-developer-tools-accuracy-of-ai-tools https://survey.stackoverflow.co/2023/#section-developer-tool...
- famouswaffles 3y ago
  Summarization, language translation and tasks in that manner (morphing tasks?) get much lower hallucinations. Not sure what that survey has to do with anything.
  - krainboltgreene 3y ago
    > Summarization, language translation and tasks in that manner (morphing tasks?) get much lower hallucinations. Source?
    - famouswaffles 3y ago
      There are no benchmarks that test that. It's my experience and the experience of others who use LLMs for such tasks. Down below there's a comment pushing back on saying Summarization has been solved. Even he/she is saying hallucination is rare.
pdntspa 3y ago
> and are already improving peoples’ lives and productivity. Don't forget this is coming at the cost of jobs that the AI is replacing. I would say, for those people, it is hurting, if not destroying, their lives. You would all do well to remember this.
- TheBrokenRail 3y ago
  So does all forms of automation. By your logic we should've never invented computers because they took jobs from human computers[0]. Improved technology and automation reduces jobs for people doing things the old way and makes the job easier for everyone else. No job is necessary forever, that's just how the world works. [0]: https://en.m.wikipedia.org/wiki/Computer_(occupation) https://en.m.wikipedia.org/wiki/Computer_(occupation)
  - pdntspa 3y ago
    And we've always had a shitty answer to dealing with the fallout. Unskilled people are running out of things they can do or retrain to. Sending everyone to college has only destroyed the meaning of a college degree and created a surplus of postgrads who can't earn anywhere near their potential for lack of demand. If we keep overfishing this lake, at some point (I am guessing in the near future) we will hit an inflection point where the number of people vastly outnumbered the availability of jobs, and social services will be strained to the point of collapse. That is when the torches and pitchforks will come out, and these folks had better hope to whatever God they think exists that AI will save them. Because my bet is it won't. What are YOU going to be doing to help these people? I advocate for UBI wherever feasible, I simply don't see a way to put these folks back to work without laying waste to whatever industry they decide to enter.
YeGoblynQueenne 3y ago
>> - Image descriptions eg. Be My Eyes: https://www.bemyeyes.com/ https://www.bemyeyes.com/ Not working yet. From the website: Starting today, you can register for the waitlist in the Be My Eyes app. >> - Customer service chat triage. More specifically? Which company is currently using LLMs for this purpose? >> - helping students learn- eg khan academy, tutor Lily Helping them learn, but what? Per wikipedia: Statements made in certain mathematics and physics videos have been questioned for their technical accuracy.[43] Sounds like using LLMs will just generate more garbage teaching material. >> - Helping create and debug software. If you don’t think is happening then you’re either living under a rock or just close to believe people are lying about how they’ve used it? Oh, absolutely. People are absolutely lying to themselves. OpenAI's and DeepMind's systematic testing of their code-geneator LLMs make it very clear that those systems produce incorrect code the majority of the time. The best results reported are 28.8% for Codex (in perpetual preprint: https://arxiv.org/abs/2107.03374 https://arxiv.org/abs/2107.03374) and 29.6% for AlphaCode (preprint: https://arxiv.org/abs/2203.07814 https://arxiv.org/abs/2203.07814 Science, paywalled: https://www.science.org/doi/10.1126/science.abq1158 https://www.science.org/doi/10.1126/science.abq1158) and the latter is with their special 10@k metric which basically means the LLM gets 10 guesses. I have definitely observed people convince themselves online that CoPilot or ChatGPT even is helping them "improve their productivity" or some such. It is obvious that they are fooling themselves, badly. If you push them, they immediately say "oh yeah, it makes mistakes, but I can correct them" etc. So they just feel like it's useful, even when it's just making them do more work. In fact, that's exactly like homeopathic self-delusions: people use it because it makes them feel better, not because it has any measurable benefit. Btw: >> ... you’re either living under a rock ... Stop being a jerk.
- famouswaffles 3y ago
  >I have definitely observed people convince themselves online that CoPilot or ChatGPT even is helping them "improve their productivity" or some such. It is obvious that they are fooling themselves, badly. If you push them, they immediately say "oh yeah, it makes mistakes, but I can correct them" etc. So they just feel like it's useful, even when it's just making them do more work. That mistakes can be made doesn't mean time isn't being saved. Everybody makes mistakes as is and first try code isn't typically being pushed human or not. The presence of mistakes means nothing. Frankly, you're the one who comes off delusional here. "Most people are telling me one thing but they can't be right because i believe so so they must be lying to themselves" isn't normal behavior, especially when your biggest argument that they must all be wrong and you right is that the machine makes mistakes. That's very weak. Take a step back and really look at what you're saying.

skepticATX 3y ago

I'd push back on summarization being solved. I have extensively used GPT 3.5 and 4 for this in production, and it works very well but it still often just ignores critical aspects of the original text, and it will rarely even completely hallucinate details. And this is with a lot of iteration on the prompt. We still have to throw out > 50% of its output because a human can summarize the text much better.

dns_snek 3y ago

> - Summarisation and content distillation, question answering Etc: literally everywhere. This is now a solved problem (to the point of it being dull) thanks to LLMs. Calling this problem domain "solved" only undermines whatever point you're trying to make, unless you really believe that current LLMs which fabricate information, flip flop between answers with follow up questions, and even gaslight the user, can be called "solutions". LLMs have proven themselves to be such untrustworthy "sources" of information and knowledge that I'm struggling to understand why anyone would even try to make this particular claim. It's trivial to refute by anyone who's used them and has been thoroughly refuted by StackOverflow Developer survey which reported that only 2.8% of developers "highly trust" the output of AI tools. https://survey.stackoverflow.co/2023/#section-developer-tools-accuracy-of-ai-tools https://survey.stackoverflow.co/2023/#section-developer-tool...

famouswaffles 3y ago
Summarization, language translation and tasks in that manner (morphing tasks?) get much lower hallucinations. Not sure what that survey has to do with anything.
- krainboltgreene 3y ago
  > Summarization, language translation and tasks in that manner (morphing tasks?) get much lower hallucinations. Source?
  - famouswaffles 3y ago
    There are no benchmarks that test that. It's my experience and the experience of others who use LLMs for such tasks. Down below there's a comment pushing back on saying Summarization has been solved. Even he/she is saying hallucination is rare.

pdntspa 3y ago

> and are already improving peoples’ lives and productivity. Don't forget this is coming at the cost of jobs that the AI is replacing. I would say, for those people, it is hurting, if not destroying, their lives. You would all do well to remember this.

TheBrokenRail 3y ago
So does all forms of automation. By your logic we should've never invented computers because they took jobs from human computers[0]. Improved technology and automation reduces jobs for people doing things the old way and makes the job easier for everyone else. No job is necessary forever, that's just how the world works. [0]: https://en.m.wikipedia.org/wiki/Computer_(occupation) https://en.m.wikipedia.org/wiki/Computer_(occupation)
- pdntspa 3y ago
  And we've always had a shitty answer to dealing with the fallout. Unskilled people are running out of things they can do or retrain to. Sending everyone to college has only destroyed the meaning of a college degree and created a surplus of postgrads who can't earn anywhere near their potential for lack of demand. If we keep overfishing this lake, at some point (I am guessing in the near future) we will hit an inflection point where the number of people vastly outnumbered the availability of jobs, and social services will be strained to the point of collapse. That is when the torches and pitchforks will come out, and these folks had better hope to whatever God they think exists that AI will save them. Because my bet is it won't. What are YOU going to be doing to help these people? I advocate for UBI wherever feasible, I simply don't see a way to put these folks back to work without laying waste to whatever industry they decide to enter.

YeGoblynQueenne 3y ago

>> - Image descriptions eg. Be My Eyes: https://www.bemyeyes.com/ https://www.bemyeyes.com/ Not working yet. From the website: Starting today, you can register for the waitlist in the Be My Eyes app. >> - Customer service chat triage. More specifically? Which company is currently using LLMs for this purpose? >> - helping students learn- eg khan academy, tutor Lily Helping them learn, but what? Per wikipedia: Statements made in certain mathematics and physics videos have been questioned for their technical accuracy.[43] Sounds like using LLMs will just generate more garbage teaching material. >> - Helping create and debug software. If you don’t think is happening then you’re either living under a rock or just close to believe people are lying about how they’ve used it? Oh, absolutely. People are absolutely lying to themselves. OpenAI's and DeepMind's systematic testing of their code-geneator LLMs make it very clear that those systems produce incorrect code the majority of the time. The best results reported are 28.8% for Codex (in perpetual preprint: https://arxiv.org/abs/2107.03374 https://arxiv.org/abs/2107.03374) and 29.6% for AlphaCode (preprint: https://arxiv.org/abs/2203.07814 https://arxiv.org/abs/2203.07814 Science, paywalled: https://www.science.org/doi/10.1126/science.abq1158 https://www.science.org/doi/10.1126/science.abq1158) and the latter is with their special 10@k metric which basically means the LLM gets 10 guesses. I have definitely observed people convince themselves online that CoPilot or ChatGPT even is helping them "improve their productivity" or some such. It is obvious that they are fooling themselves, badly. If you push them, they immediately say "oh yeah, it makes mistakes, but I can correct them" etc. So they just feel like it's useful, even when it's just making them do more work. In fact, that's exactly like homeopathic self-delusions: people use it because it makes them feel better, not because it has any measurable benefit. Btw: >> ... you’re either living under a rock ... Stop being a jerk.

famouswaffles 3y ago
>I have definitely observed people convince themselves online that CoPilot or ChatGPT even is helping them "improve their productivity" or some such. It is obvious that they are fooling themselves, badly. If you push them, they immediately say "oh yeah, it makes mistakes, but I can correct them" etc. So they just feel like it's useful, even when it's just making them do more work. That mistakes can be made doesn't mean time isn't being saved. Everybody makes mistakes as is and first try code isn't typically being pushed human or not. The presence of mistakes means nothing. Frankly, you're the one who comes off delusional here. "Most people are telling me one thing but they can't be right because i believe so so they must be lying to themselves" isn't normal behavior, especially when your biggest argument that they must all be wrong and you right is that the machine makes mistakes. That's very weak. Take a step back and really look at what you're saying.