5 ms·

somewhat unrelated to what the article is about but: LLMs can't do anything reliable AFIK they are too much probabilistic to be fully "reliable" at least if

somewhat unrelated to what the article is about but: LLMs can't do anything reliable AFIK they are too much probabilistic to be fully "reliable" at least if we take the definition of reliable from well written programs or even proof based programs if we take the definition of reliable from idk. how reliable a underpayed and not well treated employee is then ... it might be a different thing

cjk2 3y ago
LLMs are "close enough" solutions. Which should never be used where "close enough" is not good enough. Like Turing machines. Or most things really.
- mrbombastic 3y ago
  I am already starting to hear from product managers that they want to replace simple deterministic code with llm solutions. There are certainly domains where it makes sense to use these tools but I am afraid they are good enough to essentially make a shiny sales demo for anything that we are gonna be inundated soon with unreliable services.
  - cjk2 3y ago
    As it always goes: Monkey get shiny hammer. Monkey hit toes.
  - jvanderbot 3y ago
    What about summaries? Maybe LLMs are really good human approximates if you need to mashup two or more deterministically computed results into a coherent message. A RAG that pulls meta data about the system and takes as input some plain old computer output could ELI5 just about any result. This is more "project this result onto a context -specific human like space of answers" than "compute it all in LLM"
    - cjk2 3y ago
      This is actually one of my principal objections. A summary of a deterministic computation requires careful thought around how it is explained so that it is interpreted as intended. That requires understanding of the domain, understanding of the semantics of the data being presented and understanding the audience. A failure in this particular space in an observability product we use leads people wildly astray. While the data is correct, the summary is definitely not because it does not have enough understanding of the context.
    - bluefirebrand 3y ago
      I have been working on a report building webapp for the past few months. Recently a PM floated the idea of using LLMs for generating a summary portion of the report I said to him "Would you really take a report into a meeting without knowing 100% what it was going to say on it beforehand? Even in the summary, do you want to risk presenting some wildly inaccurate LLM output to your boss and peers?" Essentially: Accuracy is only one thing we want out of software. Replicability is another one. LLMs are terrible at the latter and only "good enough" at the former
      - HarHarVeryFunny 3y ago
        Yeah, things like summarization and general chatbot use are some of the most impressive things LLMs can do, yet maybe the last things they should be used for in a business setting where the accuracy of the output matters. Air Canada found this out by being dumb enough to put an LLM-based chatbot into production. Better use cases for LLMs are either where: a) There is a non-judging human in the loop who is using the LLM as a tool, and accepts it's errors/limitations. It's a bit like asking a co-worker for advice - they might have something useful on occasion, but you're not just going to take what they say unchecked and put it on the company web site or send it to your boss as accepted truth. b) You can verify the output of the LLM in someway - maybe easier if it's a structured output of some sort (SQL query perhaps) rather than free-form text.
        cjk2 3y ago
        a) it is more effort to rationalise something that is written than it is to write something rationalised. b) if it was that deterministic, then you could generate it without an LLM.
HarHarVeryFunny 3y ago
There are a number of reasons why LLMs are "unreliable" (prone to hallucination and bullshitting, as well as just being stochastic). I think the core issue is perhaps that they don't have any personal experience, nor know the trustworthiness or core competencies of their sources. Their training material is just an undifferentiated mass of text - some might be the writings of Richard Feynman, some the writing of someone on 4chan. Us humans know the value of personal experience and expertise. If we have seen or done something ourselves, applying our own learned skepticism and knowledge, then we treat that as much more trustworthy than something posted on 4chan. If we read something in a book by a well known expert, or hear it from an acquaintance we have learned to trust, then we also treat that as likely to be true, although perhaps still not as much as if we've seen/tried it for ourselves (for which there is little substitute, since only then can we experimentally test any surrounding assumptions for the transmitted knowledge to put it in context).
- sulam 3y ago
  You're anthropomorphizing way too much here. All LLMs are is a somewhat more mature Markov chain. There is no persistent intelligence in there that is simply lacking experience. The only "change" you see from run to run is what context you provide it.
  - HarHarVeryFunny 3y ago
    I understand what a transformer is, but even without architectural changes they could probably utilize source-trustworthiness (if it was provided during training) to give better responses. During training they would learn that trusted sources help better predict factual answers where that is called for (i.e. in context of a WikiPedia page). Personal experience would require architectural changes, not least online learning. No need to anthropomorphize - our brain is also just a machine, and we can use it as inspiration for what is missing to build more intelligent systems.
    - bingbingbing777 3y ago
      So for LLMs to be more trustworthy... they can just link to a Wikipedia page. The power of AI!
amelius 3y ago
Can other DL algorithms do things reliably, see e.g. self driving cars?
geor9e 3y ago
>LLMs can't do anything reliable AFIK Are you speaking just out of anecdotal experiences? Try something simple like +1 to an integer. Or swapping an ascii character. Neural nets are layers of function approximations, and plenty of those weights were trained beyond the precision needed to map the entire set of syntactically correct inputs to the correct output, for a given well-written traditional function. Remember to set the temperature to 0 (deterministic). Of course, as you ask more complex things you quickly dip below the threshold of perfection. But to say they can't do anything reliably isn't true.
- sorokod 3y ago
  Using llama2:13b -641114711832495315269166494840 + 1 = This expression is a sum of two large numbers, and the result is a very large number. Here's the calculation: -641114711832495315269166494840 + 1 = -641114711832495315269166494840 + 1 = -641114711832495315269166494840 + 1 The result of this calculation is: -641114711832495315269166494840 + 1 = 641114711832495315269166494841
  - geor9e 3y ago
    Nice. Some LLMs still have a ways to go before they learn such a basic function. Can you stump Claude with that? https://i.imgur.com/g2HCGNY.png https://i.imgur.com/g2HCGNY.png

cjk2 3y ago

LLMs are "close enough" solutions. Which should never be used where "close enough" is not good enough. Like Turing machines. Or most things really.

mrbombastic 3y ago
I am already starting to hear from product managers that they want to replace simple deterministic code with llm solutions. There are certainly domains where it makes sense to use these tools but I am afraid they are good enough to essentially make a shiny sales demo for anything that we are gonna be inundated soon with unreliable services.
- cjk2 3y ago
  As it always goes: Monkey get shiny hammer. Monkey hit toes.
- jvanderbot 3y ago
  What about summaries? Maybe LLMs are really good human approximates if you need to mashup two or more deterministically computed results into a coherent message. A RAG that pulls meta data about the system and takes as input some plain old computer output could ELI5 just about any result. This is more "project this result onto a context -specific human like space of answers" than "compute it all in LLM"
  - cjk2 3y ago
    This is actually one of my principal objections. A summary of a deterministic computation requires careful thought around how it is explained so that it is interpreted as intended. That requires understanding of the domain, understanding of the semantics of the data being presented and understanding the audience. A failure in this particular space in an observability product we use leads people wildly astray. While the data is correct, the summary is definitely not because it does not have enough understanding of the context.
  - bluefirebrand 3y ago
    I have been working on a report building webapp for the past few months. Recently a PM floated the idea of using LLMs for generating a summary portion of the report I said to him "Would you really take a report into a meeting without knowing 100% what it was going to say on it beforehand? Even in the summary, do you want to risk presenting some wildly inaccurate LLM output to your boss and peers?" Essentially: Accuracy is only one thing we want out of software. Replicability is another one. LLMs are terrible at the latter and only "good enough" at the former
    - HarHarVeryFunny 3y ago
      Yeah, things like summarization and general chatbot use are some of the most impressive things LLMs can do, yet maybe the last things they should be used for in a business setting where the accuracy of the output matters. Air Canada found this out by being dumb enough to put an LLM-based chatbot into production. Better use cases for LLMs are either where: a) There is a non-judging human in the loop who is using the LLM as a tool, and accepts it's errors/limitations. It's a bit like asking a co-worker for advice - they might have something useful on occasion, but you're not just going to take what they say unchecked and put it on the company web site or send it to your boss as accepted truth. b) You can verify the output of the LLM in someway - maybe easier if it's a structured output of some sort (SQL query perhaps) rather than free-form text.
      - cjk2 3y ago
        a) it is more effort to rationalise something that is written than it is to write something rationalised. b) if it was that deterministic, then you could generate it without an LLM.

HarHarVeryFunny 3y ago

There are a number of reasons why LLMs are "unreliable" (prone to hallucination and bullshitting, as well as just being stochastic). I think the core issue is perhaps that they don't have any personal experience, nor know the trustworthiness or core competencies of their sources. Their training material is just an undifferentiated mass of text - some might be the writings of Richard Feynman, some the writing of someone on 4chan. Us humans know the value of personal experience and expertise. If we have seen or done something ourselves, applying our own learned skepticism and knowledge, then we treat that as much more trustworthy than something posted on 4chan. If we read something in a book by a well known expert, or hear it from an acquaintance we have learned to trust, then we also treat that as likely to be true, although perhaps still not as much as if we've seen/tried it for ourselves (for which there is little substitute, since only then can we experimentally test any surrounding assumptions for the transmitted knowledge to put it in context).

sulam 3y ago
You're anthropomorphizing way too much here. All LLMs are is a somewhat more mature Markov chain. There is no persistent intelligence in there that is simply lacking experience. The only "change" you see from run to run is what context you provide it.
- HarHarVeryFunny 3y ago
  I understand what a transformer is, but even without architectural changes they could probably utilize source-trustworthiness (if it was provided during training) to give better responses. During training they would learn that trusted sources help better predict factual answers where that is called for (i.e. in context of a WikiPedia page). Personal experience would require architectural changes, not least online learning. No need to anthropomorphize - our brain is also just a machine, and we can use it as inspiration for what is missing to build more intelligent systems.
  - bingbingbing777 3y ago
    So for LLMs to be more trustworthy... they can just link to a Wikipedia page. The power of AI!

amelius 3y ago

Can other DL algorithms do things reliably, see e.g. self driving cars?

geor9e 3y ago

>LLMs can't do anything reliable AFIK Are you speaking just out of anecdotal experiences? Try something simple like +1 to an integer. Or swapping an ascii character. Neural nets are layers of function approximations, and plenty of those weights were trained beyond the precision needed to map the entire set of syntactically correct inputs to the correct output, for a given well-written traditional function. Remember to set the temperature to 0 (deterministic). Of course, as you ask more complex things you quickly dip below the threshold of perfection. But to say they can't do anything reliably isn't true.

sorokod 3y ago
Using llama2:13b -641114711832495315269166494840 + 1 = This expression is a sum of two large numbers, and the result is a very large number. Here's the calculation: -641114711832495315269166494840 + 1 = -641114711832495315269166494840 + 1 = -641114711832495315269166494840 + 1 The result of this calculation is: -641114711832495315269166494840 + 1 = 641114711832495315269166494841
- geor9e 3y ago
  Nice. Some LLMs still have a ways to go before they learn such a basic function. Can you stump Claude with that? https://i.imgur.com/g2HCGNY.png https://i.imgur.com/g2HCGNY.png