3 ms·

It’s still a bug, even if it’s the only way the system can behave as currently designed. I agree that “hallucination” is a poor term for it, though. For medicat

It’s still a bug, even if it’s the only way the system can behave as currently designed. I agree that “hallucination” is a poor term for it, though. For medication, we call a bug a “side effect” even though it’s really just a chemical interaction which, given enough information, could be predicted. Ultimately the computer only does what we tell it to do. That has always been the case and probably always will be, just as we are the result of our inputs. As with most bugs, I think to solve hallucinations we will need to better understand the input and its interactions within the system.

darby_nine 2y ago
> It’s still a bug Only if it's actually possible to remove this behavior. I don't think there's any evidence of that at this point.
lolinder 2y ago
I find it helpful to distinguish between bugs and design flaws. A bug is caused by a poorly implemented version of the design (or a literal bug in the system). Fixing a bug requires identifying where the system varies from the design and bringing it into alignment with the design. A design flaw is a case where the idealized system as conceived by the engineers is incapable of fully solving the problem statement. Fixing a design flaw may require small tweaks, but it can also mean that the entire solution needs to be thrown out in favor of a new one. Importantly, what's a design flaw for one problem statement may be just fine or even beneficial for another problem statement. So, more objectively, we might refer to these as design characteristics. Hallucinations are a special case of two low-level design characteristics of LLMs: first, that they are trained on more data than can reasonably be filtered by a human (and therefore will be exposed to data that the humans wish it weren't) and second, that they produce their text by sampling a distribution of probabilities. These two characteristics mean that controlling the output of an LLM is very very difficult (or, as the article suggests, impossible), leading both to hallucinations and alignment concerns (which are actually the same concept framed slightly differently). If the problem statement for an LLM application requires more 99% factual accuracy or more than 99% "doesn't produce content that will make investors nervous" accuracy, these design characteristics count as a design flaw.
- vrighter 2y ago
  99% is way too low for production use.
  - lolinder 2y ago
    If that's true for your use case (it's not true for all) then yes, including LLMs in your design would be a design flaw until they get much much better.
  - oneshtein 2y ago
    It's OK for cases, when humans are worse and/or much more expensive.

darby_nine 2y ago

> It’s still a bug Only if it's actually possible to remove this behavior. I don't think there's any evidence of that at this point.

lolinder 2y ago

I find it helpful to distinguish between bugs and design flaws. A bug is caused by a poorly implemented version of the design (or a literal bug in the system). Fixing a bug requires identifying where the system varies from the design and bringing it into alignment with the design. A design flaw is a case where the idealized system as conceived by the engineers is incapable of fully solving the problem statement. Fixing a design flaw may require small tweaks, but it can also mean that the entire solution needs to be thrown out in favor of a new one. Importantly, what's a design flaw for one problem statement may be just fine or even beneficial for another problem statement. So, more objectively, we might refer to these as design characteristics. Hallucinations are a special case of two low-level design characteristics of LLMs: first, that they are trained on more data than can reasonably be filtered by a human (and therefore will be exposed to data that the humans wish it weren't) and second, that they produce their text by sampling a distribution of probabilities. These two characteristics mean that controlling the output of an LLM is very very difficult (or, as the article suggests, impossible), leading both to hallucinations and alignment concerns (which are actually the same concept framed slightly differently). If the problem statement for an LLM application requires more 99% factual accuracy or more than 99% "doesn't produce content that will make investors nervous" accuracy, these design characteristics count as a design flaw.

vrighter 2y ago
99% is way too low for production use.
- lolinder 2y ago
  If that's true for your use case (it's not true for all) then yes, including LLMs in your design would be a design flaw until they get much much better.
- oneshtein 2y ago
  It's OK for cases, when humans are worse and/or much more expensive.