4 ms·

> No matter how complex the LLM, however, it is ultimately a mathematical model of its training data, and it lacks the human ability to determine whether or not

> No matter how complex the LLM, however, it is ultimately a mathematical model of its training data, and it lacks the human ability to determine whether or not a conversation in which it participates truly has meaning, or is simply a sequence of gibberish responses. > A consequence of this state of affairs is that an LLM will continue to engage in a “conversation” comprised of nonsense long past the point where a human would have abandoned the discussion as pointless. I think the author is falling into the trap of thinking that something can't be more than the sum of its parts. As well, 'merely a math model of its training data' is trivializing the fact that training data is practically the entire stored text output of humankind and the math, if done by a person with a calculator, would take thousands of years to complete. Perhaps the LLM is continuing to communicate with the bot not because it is unable to comprehend what is gibberish and what isn't by some inherent nature of the LLM, but because it is trained to be helpful and to not judge if a conversation is 'useless' or not, but to try and communicate regardless.

daveguy 2y ago
The LLM is continuing to communicate with the bot because that is literally all an LLM can do -- predict the next sequence of tokens.
- Eisenstein 2y ago
  Of course, that its function. It is able to refuse to continue conversing by restating its refusal over and over, though.
  - acka 2y ago
    That is easy to solve. Just use a model capable of function/tool calling, implement a tool which terminates the chat, then add instructions to the system prompt telling the model what tool to use if it wants to end the conversation. If the model appears too hesitant or eager to use the tool, do some finetuning on conversations where the model should or should not use it.
  - sahmeepee 2y ago
    My immediate thought at the start of this article was not DoS but more about harming the company using the chatbot (Company A) by increasing their chatbot bills. In many (most?) cases they will not be hosting their chatbot and will instead be getting it from a 3rd party provider (Company B) who may not even be truly hosting it either. If the pricing structure is per conversation or per month it would harm Company B, but not the likely target, Company A. If it is paid per interaction it would harm Company A and benefit Company B who just get more paid work. It feels a bit like cases of rivals clicking on each other's ads to cost them on ad spend, but presumably much lower value than ads. You would think it would be easy to stop a conversation at n interactions via some other means than relying on the LLM itself, but then you also have to figure out how to stop the attacker just starting more conversations (or passing the output of one of your chatbot instances into the input of another)
    - nkrisc 2y ago
      If costs of outlier conversations are a concern but any party, they can just end the conversation after 1,000 or 10,000 responses or whatever. What human would ever reach that threshold? Surely no customer worth keeping, whatever you’re selling.
- XorNot 2y ago
  Yes - tokens. Which aren't necessarily conversation responses - i.e. it can predict it should cease communication, and output whatever it's been told will terminate it (perhaps by invoking a tool).
- moffkalast 2y ago
  No, it can refuse to talk by outputting an <eos> token at any point if it predicts that there is nothing more to be said. Technically still "just a token" yes, but it does flow control instead.
devjab 2y ago
LLMs aren’t capable of “comprehending” anything. They never “know” what they are outputting, they’re simply really good at being lucky. They are not lucky enough to be useful unless you’re already an expert on the topic you’re using them on so that you can spot when they aren’t lucky. This is part of why many enterprise organisations are banning their usage. It’s one thing to use them to build software poorly, the world is already used to IT not working very often. It’s another thing to produce something that has real world consequences. Our legal department used them in a PoC for contract work, and while they were useful very often they also sometimes got things very wrong. Unlike a slow IT system, this would have business shattering consequences. You can continue training your model as well as reigning it in when it gets unlucky, but ultimately you can never be sure it’s never unlucky, and this means that LLMs are useless for a lot of things. We still use them to make pretty PowerPoint presentations and so on, but again, this is an area where faults are tolerable.
- deleted 2y ago
  [deleted]
- Eisenstein 2y ago
  I don't believe that current models have the capability to 'comprehend', I was using the term loosely. However I find that people tend to go to extremes when they want to make the point that language models are not 'intelligent' by minimizing their capability and complexity behind 'it is just math', which I think is unhelpful because it merely acts as a 'though-terminating cliche'.
  - devjab 2y ago
    I think the issue is far more psychological than technical personally. One of the issues we struggle with for our junior developers is that they are far more likely to believe an LLM than what they might find Google programming. I do wonder why we’ve ended up with CS graduates who go to LLMs and Search engines before the official documentation which is often very, very good, but I guess that’s a different discussion. I’m not personally against LLM assistance, I use it for programming and it has in many places replaced my usage of snippets completely. This is probably why I’m not really a fan of the “knowledge” part that LLMs are increasingly tasked to do. Because when you use them for programming you’ll get an accrue insight into how terrible they can be when they get things wrong.

daveguy 2y ago

The LLM is continuing to communicate with the bot because that is literally all an LLM can do -- predict the next sequence of tokens.

Eisenstein 2y ago
Of course, that its function. It is able to refuse to continue conversing by restating its refusal over and over, though.
- acka 2y ago
  That is easy to solve. Just use a model capable of function/tool calling, implement a tool which terminates the chat, then add instructions to the system prompt telling the model what tool to use if it wants to end the conversation. If the model appears too hesitant or eager to use the tool, do some finetuning on conversations where the model should or should not use it.
- sahmeepee 2y ago
  My immediate thought at the start of this article was not DoS but more about harming the company using the chatbot (Company A) by increasing their chatbot bills. In many (most?) cases they will not be hosting their chatbot and will instead be getting it from a 3rd party provider (Company B) who may not even be truly hosting it either. If the pricing structure is per conversation or per month it would harm Company B, but not the likely target, Company A. If it is paid per interaction it would harm Company A and benefit Company B who just get more paid work. It feels a bit like cases of rivals clicking on each other's ads to cost them on ad spend, but presumably much lower value than ads. You would think it would be easy to stop a conversation at n interactions via some other means than relying on the LLM itself, but then you also have to figure out how to stop the attacker just starting more conversations (or passing the output of one of your chatbot instances into the input of another)
  - nkrisc 2y ago
    If costs of outlier conversations are a concern but any party, they can just end the conversation after 1,000 or 10,000 responses or whatever. What human would ever reach that threshold? Surely no customer worth keeping, whatever you’re selling.
XorNot 2y ago
Yes - tokens. Which aren't necessarily conversation responses - i.e. it can predict it should cease communication, and output whatever it's been told will terminate it (perhaps by invoking a tool).
moffkalast 2y ago
No, it can refuse to talk by outputting an <eos> token at any point if it predicts that there is nothing more to be said. Technically still "just a token" yes, but it does flow control instead.

devjab 2y ago

LLMs aren’t capable of “comprehending” anything. They never “know” what they are outputting, they’re simply really good at being lucky. They are not lucky enough to be useful unless you’re already an expert on the topic you’re using them on so that you can spot when they aren’t lucky. This is part of why many enterprise organisations are banning their usage. It’s one thing to use them to build software poorly, the world is already used to IT not working very often. It’s another thing to produce something that has real world consequences. Our legal department used them in a PoC for contract work, and while they were useful very often they also sometimes got things very wrong. Unlike a slow IT system, this would have business shattering consequences. You can continue training your model as well as reigning it in when it gets unlucky, but ultimately you can never be sure it’s never unlucky, and this means that LLMs are useless for a lot of things. We still use them to make pretty PowerPoint presentations and so on, but again, this is an area where faults are tolerable.

deleted 2y ago
[deleted]
Eisenstein 2y ago
I don't believe that current models have the capability to 'comprehend', I was using the term loosely. However I find that people tend to go to extremes when they want to make the point that language models are not 'intelligent' by minimizing their capability and complexity behind 'it is just math', which I think is unhelpful because it merely acts as a 'though-terminating cliche'.
- devjab 2y ago
  I think the issue is far more psychological than technical personally. One of the issues we struggle with for our junior developers is that they are far more likely to believe an LLM than what they might find Google programming. I do wonder why we’ve ended up with CS graduates who go to LLMs and Search engines before the official documentation which is often very, very good, but I guess that’s a different discussion. I’m not personally against LLM assistance, I use it for programming and it has in many places replaced my usage of snippets completely. This is probably why I’m not really a fan of the “knowledge” part that LLMs are increasingly tasked to do. Because when you use them for programming you’ll get an accrue insight into how terrible they can be when they get things wrong.