3 ms·

The problem - as defined by how end users understand it - is that the model itself doesn't know the difference, and will proclaim bullshit with the same level o

The problem - as defined by how end users understand it - is that the model itself doesn't know the difference, and will proclaim bullshit with the same level of confidence that it does accurate information. That's how you end up with grocery store chatbots recommending mixing ammonia and bleach for a cocktail, or lawyers using chatbots to cite entirely fictional case law before a judge in court. Nothing that comes out of an LLM can be implicitly trusted, so your default assumption must be that everything it gives you needs verification from another source. Telling people "the truth is baked in" is just begging for a disaster.

hunter2_ 2y ago
> your default assumption must be that everything it gives you needs verification from another source That depends entirely on what you're doing with the output. If you're using it as a starting point for something that must be true (whether for legal reasons, your own reputation as the ostensible author of this content, your own education, etc.) then yes, verification is required. But if you're using it for something low-stakes that just needs some semblance of coherent verbiage (like the summary of customer reviews on Amazon, or the SEO junk that comes before the recipe on cooking websites which have plenty of fiction whether or not an LLM was involved) then you can totally meet your goals without any verification. People have been capable of bullshitting at scale for a very long time. There are occasional consequences (hoaxes, scams, etc.) but the guidelines around fide sed vide are ancient; this is just the latest addendum.
- zdragnar 2y ago
  This is just moving the goalposts. The post I replied to was claiming that models "have the truth baked in". Real people in the real world are misusing them, in no small part because they don't know that the models are unreliable, and OP's claims only make that worse.
ekianjo 2y ago
> is that the model itself doesn't know the difference, and will proclaim bullshit with the same level of confidence which is a good model for what humans do as well

hunter2_ 2y ago

> your default assumption must be that everything it gives you needs verification from another source That depends entirely on what you're doing with the output. If you're using it as a starting point for something that must be true (whether for legal reasons, your own reputation as the ostensible author of this content, your own education, etc.) then yes, verification is required. But if you're using it for something low-stakes that just needs some semblance of coherent verbiage (like the summary of customer reviews on Amazon, or the SEO junk that comes before the recipe on cooking websites which have plenty of fiction whether or not an LLM was involved) then you can totally meet your goals without any verification. People have been capable of bullshitting at scale for a very long time. There are occasional consequences (hoaxes, scams, etc.) but the guidelines around fide sed vide are ancient; this is just the latest addendum.

zdragnar 2y ago
This is just moving the goalposts. The post I replied to was claiming that models "have the truth baked in". Real people in the real world are misusing them, in no small part because they don't know that the models are unreliable, and OP's claims only make that worse.

ekianjo 2y ago

> is that the model itself doesn't know the difference, and will proclaim bullshit with the same level of confidence which is a good model for what humans do as well