3 ms·

Are you able to tell when it is hallucinating?

by zht 3y ago

p-e-w 3y ago
Are you able to tell when a human is?
- gwoolhurme 3y ago
  If it is a community effort yes. Yes we are. That’s the idea of SO. It’s a community effort. Highly flawed and elitist yes, but the theory is it is community policed.
  - p-e-w 3y ago
    If that is good enough, you can replace the "community" with a cohort of different LLMs, which should work analogously.
    - gwoolhurme 3y ago
      No? Maybe if they reach human levels of sensory and cognition. For example, again, my FairPlay problem. On iOS 16 there is a timing issue when DRM can be ingested, but this isn’t documented from apple and it is kind of a rare problem. If I ask Bing right now, which is connected to the internet. It gives you links to a human stackoverflow. How would an LLM know about this timing issue? The source code looks fine, there is no compiler error, there are no crashes. There IS sometimes just a video that is not playable in AVPlayer. How does an LLM with an LLM community solve that?
      - afarviral 3y ago
        unfortunately the way web search is implemented in bing chat kind of hamstrings it from that point forward. You miss out the LLM extrapolating from a million other scenarios accross the subject domain in order to come up with possible explanations or approaches. Likewise it is not searching every page multiple adjascent topics to find something on the web that is deeply burried or going by another name because of resource constraints.
- zht 3y ago
  It’s really amazing how low our bar has become
afarviral 3y ago
Not right away, which is why I prefer bing over chatGPT (less tendency to get things subtly wrong and gaslight me). However I see no significant distinction between a hallucinating LLM and a person who made a mistake. It's something that the LLM engineers and the LLMs will work on every day. There are definitely some caveats but I maintain that less energy is required and I get faster and more tuned results than an SO post or trawl.

Are you able to tell when it is hallucinating?

by zht 3y ago

Are you able to tell when it is hallucinating?

p-e-w 3y ago

Are you able to tell when a human is?

gwoolhurme 3y ago
If it is a community effort yes. Yes we are. That’s the idea of SO. It’s a community effort. Highly flawed and elitist yes, but the theory is it is community policed.
- p-e-w 3y ago
  If that is good enough, you can replace the "community" with a cohort of different LLMs, which should work analogously.
  - gwoolhurme 3y ago
    No? Maybe if they reach human levels of sensory and cognition. For example, again, my FairPlay problem. On iOS 16 there is a timing issue when DRM can be ingested, but this isn’t documented from apple and it is kind of a rare problem. If I ask Bing right now, which is connected to the internet. It gives you links to a human stackoverflow. How would an LLM know about this timing issue? The source code looks fine, there is no compiler error, there are no crashes. There IS sometimes just a video that is not playable in AVPlayer. How does an LLM with an LLM community solve that?
    - afarviral 3y ago
      unfortunately the way web search is implemented in bing chat kind of hamstrings it from that point forward. You miss out the LLM extrapolating from a million other scenarios accross the subject domain in order to come up with possible explanations or approaches. Likewise it is not searching every page multiple adjascent topics to find something on the web that is deeply burried or going by another name because of resource constraints.
zht 3y ago
It’s really amazing how low our bar has become

afarviral 3y ago

Not right away, which is why I prefer bing over chatGPT (less tendency to get things subtly wrong and gaslight me). However I see no significant distinction between a hallucinating LLM and a person who made a mistake. It's something that the LLM engineers and the LLMs will work on every day. There are definitely some caveats but I maintain that less energy is required and I get faster and more tuned results than an SO post or trawl.