6 ms·

On a similar note, has anyone found themselves absolutely not trusting non-code LLM output? The code is at least testable and verifiable. For everything else I

On a similar note, has anyone found themselves absolutely not trusting non-code LLM output? The code is at least testable and verifiable. For everything else I am left wondering if it's the truth or a hallucination. It incurs more mental burden that I was trying to avoid using LLM in the first place.

joshstrange 2y ago
Absolutely. LLMs are a "need to verify" the results almost always. LLMs (for me) shine by pointing me in the right direction, getting a "first draft", or for things like code where I can test it.
nyrikki 2y ago
It is really the only safe way to use it IMHO. Even in most simple forms of automation, humans suffer from Automation Bias and Complacency and one of the better ways to avoid those issues is to instill a fundamental mistrust of those systems. IMHO it is important to look at other fields and the human factors studies to understand this. As an example ABS was originally sold as a technology that would help you 'stop faster'. Which it may do in some situations, and it is obviously mandatory in the US. But they had to shift how they 'sell' it now, to ensure that people didn't rely on it. https://www.fmcsa.dot.gov/sites/fmcsa.dot.gov/files/docs/2005%20CDL%20Driver%20Manual%20-July%202014%20-%20FINAL.pdf https://www.fmcsa.dot.gov/sites/fmcsa.dot.gov/files/docs/200... 2.18 – Antilock Braking Systems (ABS) ABS is a computerized system that keeps your wheels from locking up during hard brake applications. ABS is an addition to your normal brakes. It does not decrease or increase your normal braking capability. ABS only activates when wheels are about to lock up. ABS does not necessarily shorten your stopping distance, but it does help you keep the vehicle under control during hard braking. Transformers will always produce code that doesn't work, it doesn't matter if that is due to what they call hallucinations, Rice's theory, etc... Maintaining that mistrust is the mark of someone who understands and can leverage the technology. It is just yet another context specific tradeoff analysis that we will need to assess. I think forcing people into the quasi-TDD thinking model, where they focus on what needs to be done first vs jumping into the implementation details will probably be a positive thing for the industry, no matter where on the spectrum LLM coding assistants arrive. That is one of the hardest things to teach when trying to introduce TDD, focusing on what is far closer to an ADT than implementation specific unit tests to begin with is very different but very useful. I am hopeful that required tacit experience will help get past the issues with formal frameworks that run into many barriers that block teaching that one skill. As LLM's failure mode is Always Confident, Often Competent, and Inevitably Wrong, it is super critical to always realize the third option is likely and that you are the expert.
sdesol 2y ago
> On a similar note, has anyone found themselves absolutely not trusting non-code LLM output? I'm working on a LLM chat app that is built around mistrust. The basic idea is that it is unlikely a supermajority of quality LLMs can get it wrong. This isn't foolproof though, but it does provide some level of confidence in the answer. Here is a quick example in which I analyze results from multiple LLMs that answered, "When did Homer Simpson go to Mars?" https://beta.gitsense.com/?chat=4d28f283-24f4-4657-89e0-5abfc8752e59 https://beta.gitsense.com/?chat=4d28f283-24f4-4657-89e0-5abf... If you look at the yes and no table, all except GPT-4o and GPT-4o mini said no. After asking GPT-4o who was correct, it provided "evidence" on an episode so I asked for more information on that episode. Based on what it said, it looks like the mission to Mars was a hoax and when I challenged GPT-4o on this, it agreed and said Homer never went to Mars, like others have said. I then asked Sonnet 3.5 about the episode and it said GPT-4o misinterpreted the plot. https://beta.gitsense.com/?chat=4d28f283-24f4-4657-89e0-5abfc8752e59&model=Claude+3.5+Sonnet https://beta.gitsense.com/?chat=4d28f283-24f4-4657-89e0-5abf... At this point, I am confident (but not 100% sure) Homer never went to Mars and if I really needed to know, I'll need to search the web.
- manmal 2y ago
  Relatedly, asking LLMs what happens in a TV episode, or a series in general, I usually get very low quality and mostly flat out wrong answers. That baffles me, as I thought there are multiple well structured synopses for any TV series in the training data.
- horsawlarway 2y ago
  Isn't this essentially making the point of the post above you? For comparison - if I just do a web search for "Did homer simpson go to mars" I get immediately linked to the wikipedia page for that exact episode (https://en.wikipedia.org/wiki/The_Marge-ian_Chronicles https://en.wikipedia.org/wiki/The_Marge-ian_Chronicles), and the plot summary is less to read than your LLM output - It clearly summarizes that Marge & Lisa (note - NOT homer) almost went to mars, but did not go. Further - the summary correctly includes the outro which does show Marge and Lisa on mars in the year 2051. Basically - for factual content, the LLM output was a garbage game of telephone.
Marceltan 2y ago
Agree. My biggest pain point with LLM code review tools is that they sometimes add 40 comments for a PR changing 100 lines of code. Gets noisy and hard to decipher what really matters. Along the lines of verifiability, my take is that running a comprehensive suite of tests in CI/CD is going to be table stakes soon given that LLMs are only going to be contributing more and more code.
redcobra762 2y ago
You're going to fall behind eventually, if you continue to treat LLMs with this level of skepticism, as others won't, and the output is accurate enough that it can be useful to improve the efficiency of work in a great many situations. Rarely are day-to-day written documents (e.g. an email asking for clarification on an issue or to schedule an appointment) of such importance that the occasional error is unforgivable. In situations where a mistake is fatal, yes I would not trust GenAI. But how many of us really work in that kind of a field? Besides, AI shines when used for creative purposes. Coming up with new ideas or rewording a paragraph for clarity isn't something one does blindly. GenAI is a coworker, not an authority. It'll generate a draft, I may edit that draft or rewrite it significantly, but to preclude it because it could error will eventually slow you down in your field.
- thuuuomas 2y ago
  You’re narrowly addressing LLM use cases & omitting the most problematic one - LLMs as search engine replacements.
  - redcobra762 2y ago
    That's the opposite of problematic, that's where an LLM shines. And before you say hallucination, when was the last time you didn't click the link in a Google search result? It's user error if you don't follow up with additional validation, exactly as you would with Google. With GenAI it's simply easier to craft specific queries.
iamnotagenius 2y ago
Yes, it is good for suumarizing existing text, explaining something or coding; in short any generative/transformative tasks. Not good for information retrieval. Having said that even tiny Qwen 3b/7b coding llms turned out to be very useful in my use experience.
energy123 2y ago
We need a hallucination benchmark. My experience is, o1 is very good at avoiding hallucinations and I trust it more, but o1-mini and 4o are awful.
- sdesol 2y ago
  Well given the price $15.00 / 1M input tokens and $60.00 / 1M output* tokens, I would hope so. Given the price, I think it is fair to say it is doing a lot of checks in the background.
  - energy123 2y ago
    It is expensive. But if I'm correct about o1, it means user mistrust of LLMs is going to be a short-lived thing as costs come down and more people use o1 (or better) models as their daily driver.
    - sdesol 2y ago
      > mistrust of LLMs is going to be a short-lived thing as costs come down and more people use o1 I think the biggest question is, is o1 scalable. I think o1 does well because it is going back and forth hundreds if not thousands of times. Somebody mentioned in a thread that I was participating in that they let o1 crunch things for 10 minutes. It sounded like it saved them a lot work, so it was well worth it. Whether or not o1 is practical for the general public is something we will have to wait and see.
      - energy123 2y ago
        I'm going to wager "yes" because o3-mini (High) gets equal benchmark scores to o1 despite using 1/3rd as much compute, and because the consistent trend has been towards rapid order-of-magnitude decreases in price for a fixed level of intelligence (trend has many components dovetailing, both hardware and software related). Can't forecast the future, but this would be my bet on a time horizon of < 3 years.

joshstrange 2y ago

Absolutely. LLMs are a "need to verify" the results almost always. LLMs (for me) shine by pointing me in the right direction, getting a "first draft", or for things like code where I can test it.

nyrikki 2y ago

It is really the only safe way to use it IMHO. Even in most simple forms of automation, humans suffer from Automation Bias and Complacency and one of the better ways to avoid those issues is to instill a fundamental mistrust of those systems. IMHO it is important to look at other fields and the human factors studies to understand this. As an example ABS was originally sold as a technology that would help you 'stop faster'. Which it may do in some situations, and it is obviously mandatory in the US. But they had to shift how they 'sell' it now, to ensure that people didn't rely on it. https://www.fmcsa.dot.gov/sites/fmcsa.dot.gov/files/docs/2005%20CDL%20Driver%20Manual%20-July%202014%20-%20FINAL.pdf https://www.fmcsa.dot.gov/sites/fmcsa.dot.gov/files/docs/200... 2.18 – Antilock Braking Systems (ABS) ABS is a computerized system that keeps your wheels from locking up during hard brake applications. ABS is an addition to your normal brakes. It does not decrease or increase your normal braking capability. ABS only activates when wheels are about to lock up. ABS does not necessarily shorten your stopping distance, but it does help you keep the vehicle under control during hard braking. Transformers will always produce code that doesn't work, it doesn't matter if that is due to what they call hallucinations, Rice's theory, etc... Maintaining that mistrust is the mark of someone who understands and can leverage the technology. It is just yet another context specific tradeoff analysis that we will need to assess. I think forcing people into the quasi-TDD thinking model, where they focus on what needs to be done first vs jumping into the implementation details will probably be a positive thing for the industry, no matter where on the spectrum LLM coding assistants arrive. That is one of the hardest things to teach when trying to introduce TDD, focusing on what is far closer to an ADT than implementation specific unit tests to begin with is very different but very useful. I am hopeful that required tacit experience will help get past the issues with formal frameworks that run into many barriers that block teaching that one skill. As LLM's failure mode is Always Confident, Often Competent, and Inevitably Wrong, it is super critical to always realize the third option is likely and that you are the expert.

sdesol 2y ago

> On a similar note, has anyone found themselves absolutely not trusting non-code LLM output? I'm working on a LLM chat app that is built around mistrust. The basic idea is that it is unlikely a supermajority of quality LLMs can get it wrong. This isn't foolproof though, but it does provide some level of confidence in the answer. Here is a quick example in which I analyze results from multiple LLMs that answered, "When did Homer Simpson go to Mars?" https://beta.gitsense.com/?chat=4d28f283-24f4-4657-89e0-5abfc8752e59 https://beta.gitsense.com/?chat=4d28f283-24f4-4657-89e0-5abf... If you look at the yes and no table, all except GPT-4o and GPT-4o mini said no. After asking GPT-4o who was correct, it provided "evidence" on an episode so I asked for more information on that episode. Based on what it said, it looks like the mission to Mars was a hoax and when I challenged GPT-4o on this, it agreed and said Homer never went to Mars, like others have said. I then asked Sonnet 3.5 about the episode and it said GPT-4o misinterpreted the plot. https://beta.gitsense.com/?chat=4d28f283-24f4-4657-89e0-5abfc8752e59&model=Claude+3.5+Sonnet https://beta.gitsense.com/?chat=4d28f283-24f4-4657-89e0-5abf... At this point, I am confident (but not 100% sure) Homer never went to Mars and if I really needed to know, I'll need to search the web.

manmal 2y ago
Relatedly, asking LLMs what happens in a TV episode, or a series in general, I usually get very low quality and mostly flat out wrong answers. That baffles me, as I thought there are multiple well structured synopses for any TV series in the training data.
horsawlarway 2y ago
Isn't this essentially making the point of the post above you? For comparison - if I just do a web search for "Did homer simpson go to mars" I get immediately linked to the wikipedia page for that exact episode (https://en.wikipedia.org/wiki/The_Marge-ian_Chronicles https://en.wikipedia.org/wiki/The_Marge-ian_Chronicles), and the plot summary is less to read than your LLM output - It clearly summarizes that Marge & Lisa (note - NOT homer) almost went to mars, but did not go. Further - the summary correctly includes the outro which does show Marge and Lisa on mars in the year 2051. Basically - for factual content, the LLM output was a garbage game of telephone.

Marceltan 2y ago

Agree. My biggest pain point with LLM code review tools is that they sometimes add 40 comments for a PR changing 100 lines of code. Gets noisy and hard to decipher what really matters. Along the lines of verifiability, my take is that running a comprehensive suite of tests in CI/CD is going to be table stakes soon given that LLMs are only going to be contributing more and more code.

redcobra762 2y ago

You're going to fall behind eventually, if you continue to treat LLMs with this level of skepticism, as others won't, and the output is accurate enough that it can be useful to improve the efficiency of work in a great many situations. Rarely are day-to-day written documents (e.g. an email asking for clarification on an issue or to schedule an appointment) of such importance that the occasional error is unforgivable. In situations where a mistake is fatal, yes I would not trust GenAI. But how many of us really work in that kind of a field? Besides, AI shines when used for creative purposes. Coming up with new ideas or rewording a paragraph for clarity isn't something one does blindly. GenAI is a coworker, not an authority. It'll generate a draft, I may edit that draft or rewrite it significantly, but to preclude it because it could error will eventually slow you down in your field.

thuuuomas 2y ago
You’re narrowly addressing LLM use cases & omitting the most problematic one - LLMs as search engine replacements.
- redcobra762 2y ago
  That's the opposite of problematic, that's where an LLM shines. And before you say hallucination, when was the last time you didn't click the link in a Google search result? It's user error if you don't follow up with additional validation, exactly as you would with Google. With GenAI it's simply easier to craft specific queries.

iamnotagenius 2y ago

Yes, it is good for suumarizing existing text, explaining something or coding; in short any generative/transformative tasks. Not good for information retrieval. Having said that even tiny Qwen 3b/7b coding llms turned out to be very useful in my use experience.

energy123 2y ago

We need a hallucination benchmark. My experience is, o1 is very good at avoiding hallucinations and I trust it more, but o1-mini and 4o are awful.

sdesol 2y ago
Well given the price $15.00 / 1M input tokens and $60.00 / 1M output* tokens, I would hope so. Given the price, I think it is fair to say it is doing a lot of checks in the background.
- energy123 2y ago
  It is expensive. But if I'm correct about o1, it means user mistrust of LLMs is going to be a short-lived thing as costs come down and more people use o1 (or better) models as their daily driver.
  - sdesol 2y ago
    > mistrust of LLMs is going to be a short-lived thing as costs come down and more people use o1 I think the biggest question is, is o1 scalable. I think o1 does well because it is going back and forth hundreds if not thousands of times. Somebody mentioned in a thread that I was participating in that they let o1 crunch things for 10 minutes. It sounded like it saved them a lot work, so it was well worth it. Whether or not o1 is practical for the general public is something we will have to wait and see.
    - energy123 2y ago
      I'm going to wager "yes" because o3-mini (High) gets equal benchmark scores to o1 despite using 1/3rd as much compute, and because the consistent trend has been towards rapid order-of-magnitude decreases in price for a fixed level of intelligence (trend has many components dovetailing, both hardware and software related). Can't forecast the future, but this would be my bet on a time horizon of < 3 years.