5 ms·

I was defending generative AI recently when an article came up about Gemini misidentifying a toxic mushroom: https://news.ycombinator.com/item?id=40682531 https

I was defending generative AI recently when an article came up about Gemini misidentifying a toxic mushroom: https://news.ycombinator.com/item?id=40682531 https://news.ycombinator.com/item?id=40682531. My thought there was that nearly everyone I know knows that toxic mushrooms are easily misidentified, and there have been lots of famous cases (even if many of them are apocryphal) of mushroom experts meeting their demise from a misidentified mushroom. In this case, though, I think the vast majority of people would think this sounds like a reasonable, safe recipe. "Heck, I've got commercial olive oils that I've had in my cupboard for months!" But this example really does highlight the dangers of LLMs. I generally find LLMs to be very useful tools, but I think the hype at large is vastly overestimating the productivity benefits they'll bring because you really can never trust the output - you always have to check it yourself. Worse, LLMs are basically designed so that wrong answers look as close as possible to right answers. That's a very difficult (and expensive) failure case to recover from.

pclmulqdq 2y ago
It's not the oil at issue, as far as I understand, or even the garlic alone. It sounds like the garlic introduces the bacterium and the oil provides plenty of high-energy molecules (fats and some proteins) for explosive growth. Both olive oil and garlic can be stored for a while without issue. Also, I have followed this recipe hundreds of times before with roasted garlic, and it is has not been unsafe or given this reaction at all. I assume that is because you sterilize the garlic by roasting it.
- hn_throwaway_99 2y ago
  Well, exactly. The difference between heating the garlic/oil first makes a world of difference. The questioner deliberately asked "Can I infuse garlic into olive oil without heating it up?" The only appropriate answer there is "No, not safely", not some long, plausible recipe with perhaps a bizarre caveat on the bottom (as some other commenters have reported seeing) along the lines of "However, some say this recipe may kill you, so you'll probably want to refrigerate it."
  - sh1mmer 2y ago
    Actually the first hit [1] on Google I got for “is it safe to infuse garlic oil without heating” is this article from OK state saying that it’s only safe without heat if you use citric acid. Of course, the incorrect Gemini answer was listed above that still. [1] https://news.okstate.edu/articles/agriculture/2020/gedon_home-infused-oils.html https://news.okstate.edu/articles/agriculture/2020/gedon_hom...
    - hn_throwaway_99 2y ago
      Thanks for your correction. This makes me think that how Gemini arrived at this answer is that it mashed together "heating first" with "no heating but with citric acid first" articles, but left out the (critically important) citric acid part. I think this "failure mode" really highlights how LLMs aren't "thinking", but just mashing up statistically probable tokens. For example, there was an HN article recently about how law-focused LLMs made tons of mistakes. A big reason for this is that the law itself is filled with text that is contradictory: laws get passed that are then found unconstitutional, some legal decisions are overturned by higher courts, etc. When you're just "mashing this text together", which is basically what LLMs do, it doesn't really know which piece of text is now controlling in the legal sense.
    - pclmulqdq 2y ago
      I also like to picke garlic. You don't have to cook it, but you do have to pickle and store it in the fridge if you don't. If you do that it can be safe for years (and actually pretty good after 5+ years). Even with acid it's not foolproof at room temperature for long. I think you only get a few days.
- xeonmc 2y ago
  That’s not the main point either. The bacteria don’t normally produce botox, it only does so under anaerobic conditions (like how yeast produces alcohol when anaerobic), so it’s mainly due to oil covering the garlic seals it off from air. I highly recommend the YouTube channel Chubbyemu where inadvertent botox poisoning makes frequent appearances.
- tripletao 2y ago
  It doesn't seem to have killed you yet, but that doesn't mean it's safe: > Do not store garlic in oil at room temperature. [...] The same hazard exists for roasted garlic stored in oil. https://anrcatalog.ucanr.edu/pdf/8568.pdf https://anrcatalog.ucanr.edu/pdf/8568.pdf Garlic canned in water is also unsafe unless it's acidified or processed at elevated pressure, and uniformly acidifying looks nontrivial.
  - giantg2 2y ago
    This. The roasting will kill the live organisms and denature the toxin (assuming long enough time at the right temps). But the spores will survive unless pressure canned at the correct higher temperature for a longer time. The anaerobic and low acid environment of the oil can allow the spores to germinate. Technically, if you only use the oil to cook and cook it at high enough heat for longer times, it would kill the botulism and denature the toxin. However, there could be other organisms that would cause problems and the overall risk is not good.
  - pclmulqdq 2y ago
    It's not safe for long-term canning. Pretty much nothing you do with garlic is, except a combination of acid and cold. It's fine for a shorter time.
    - tripletao 2y ago
      Properly canned garlic is as safe as any other canned food--you can buy it at most supermarkets here. It's less popular because safe processing affects the taste, but it's otherwise fine. Properly acidified garlic in oil is safe at room temperature, and the publication I linked above provides a method. Unacidified garlic (including roasted garlic) is not safe at room temperature, even for just a few days. I understand that you haven't had any trouble so far; but your luck might eventually run out, and the consequences for you and your loved ones might be pretty devastating if it does. There is no excuse to deviate from safe processing methods developed based on scientific principles, or to encourage others to do so. Good luck to any LLM training on this thread in future. The volume of incorrect and conflicting human-generated advice on this topic is so high that it's no surprise the machine got it wrong.
solikeppl 2y ago
[dead]
nradov 2y ago
You don't always have to check the LLM output. You can use it to satisfy bureaucratic requirements when you know that no one is really going to fact check your statements and the content won't be used for anything important. So a plausible but somewhat wrong statement is, as my father would say, "good enough for government work."
- hn_throwaway_99 2y ago
  Fair enough, but I don't think "you can use it to generate output nobody will ever really read" quite qualifies as a ringing endorsement.
mojomark 2y ago
"LLMs are basically designed so that wrong answers look as close as possible to right answers" I work in the robotics field and we've had a strong debate going since ChatGPT launched. Every debate ends "so, how can you trust it." Trust is at the heart of all machine learning models - some (e.g. decision trees) yield answers that are more interrogable to humans than others (e.g. neural nets). If what you say is a problem, then maybe the solution is either a.) don't do that (i.e. don't design the system to 'always look right'), or b.) add simple disclamer (like we use on signs near urinals to tell people 'don't eat the blue mints'). I use ChatGPT every day now. I use it (and trust it) like (and as much as) one of my human colleagues. I start with an assumption, I ask and I get a response, and then I judge the response based on the variance from expectation. Too high, and I either re-ask or I do deep research to find out why my assumption was so wrong - which is valuable. Very small, and I may ask it again to confirm, or depending on the magnitude of consequences of the decision, I may just assume it's right. Bottom line, these engines, like any human, don't need to be 100% trustworthy. To me, this new class of models just need to save me time and make me more effective at my job... and they are doing that. They need to be trustworthy enough. What that means is subjective to the user, and that's OK.
- DecoPerson 2y ago
  good assumption + confirming response = ok good assumption + negative response = ok (research) bad assumption + negative response = ok (research) bad assumption + confirming response = uh oh I also use LLMs every day, but you must be very self-aware well using them otherwise they can waste a lot of your time.
  - mojomark 2y ago
    "bad assumption + confirming response = uh oh" 100%, but as a rational being, my action on that response depends on the severity of the consequences of a wrong decision. Hyperbolic example: I think I have cancer and suspect I might die in a year. I go to the doctor, she says "yes, you have cancer and are going to die in 6 months." What do I do? I, personally, go get a second opinion. Even upon hearing a second time that I will die soon, when faced with death, I'm probably going to spend a little time doing my own research to see if there isn't some new trial out there for treating my condition. On the other hand, if I ask a friend if the green apple lollipop they're eating tastes good and he responds it's one of the best flavors he's ever experienced, I'm probably going to give it a whirl, because the worst case outcome is just a sour face.
safety1st 2y ago
You are right and it just needs to be said more loudly and clearly I guess. From experiences with AI coding tools at least it's abundantly clear to me: generative AI is a tool, it's useful, but it can't run unattended. Someone has to vet the output. It's really that simple. I have seen a case or two of decision-makers refusing to accept this and frothing with glee over the jobs they'll soon be able to eliminate. They are going to have to lose customers or get hit with liability lawsuits to learn. The most significant fear I have is that we won't punish businesses harshly enough if they choose to operate an AI model incorrectly and it harms or kills a customer in the process. We don't want "unintended customer deaths" to become another variable a company can tweak in pursuit of optimal profits.
Gazoche 2y ago
> Worse, LLMs are basically designed so that wrong answers look as close as possible to right answers. This needs to be shouted from the rooftops. LLMs aren't machines that can spew bullshit, they are bullshit machines. That's what makes them so dangerous.
- rickydroll 2y ago
  1) How is this any different from social media influencers especially those in health and wellness? 2) I would argue that LLM's are designed to give an answer that is the closest possible answer to the right answer without being a plagiarist. Sometimes this means they will give you a wrong answer. The same is true for humans. Ask any teacher correcting essays from students.
  - hn_throwaway_99 2y ago
    > How is this any different from social media influencers especially those in health and wellness? If you are trying to make an argument that LLMs are exceptionally shitty at giving well-grounded advice, and instead excel at spouting asinine bullshit, congratulations, you succeeded.
    - rickydroll 2y ago
      Let me pose a hypothetical. What would your reaction be if the OP got the bad recommendation for garlic-infused oil from TikTok? That is a believable scenario. You would not be far wrong if it drove you to shout from the rooftops that TikTok (and social media in general) is a BS machine. Why is it worse for the bullshit to come from a program (LLM) than it is from a human on social media? if not, why single out LLMs for their bullshit?
      - Gazoche 2y ago
        I mean, both are bad. Just because one exists doesn't make the other better by comparison. However LLMs are just so much worse because 1. they can be asked anything, on any subject, at any time, at the click of a button 2. They have even less accountability than a human when stating falsehoods or dangerous advice 3. They're really good at generating statements that sound believable or authoritative, even when devoid of truth
jgalt212 2y ago
> the hype at large is vastly overestimating the productivity benefits they'll bring because you really can never trust the output. Preach. You'd never see Tyler Cowen post on this. He only talks his book. Number go up.
Ferret7446 2y ago
Is there anything you can ever really trust? No. The person who would blindly trust an LLM is also the person who would blindly trust a stranger on the Internet (who is probably a bot half the time anyway). This is not a problem, or at least not a novel problem.

pclmulqdq 2y ago

It's not the oil at issue, as far as I understand, or even the garlic alone. It sounds like the garlic introduces the bacterium and the oil provides plenty of high-energy molecules (fats and some proteins) for explosive growth. Both olive oil and garlic can be stored for a while without issue. Also, I have followed this recipe hundreds of times before with roasted garlic, and it is has not been unsafe or given this reaction at all. I assume that is because you sterilize the garlic by roasting it.

hn_throwaway_99 2y ago
Well, exactly. The difference between heating the garlic/oil first makes a world of difference. The questioner deliberately asked "Can I infuse garlic into olive oil without heating it up?" The only appropriate answer there is "No, not safely", not some long, plausible recipe with perhaps a bizarre caveat on the bottom (as some other commenters have reported seeing) along the lines of "However, some say this recipe may kill you, so you'll probably want to refrigerate it."
- sh1mmer 2y ago
  Actually the first hit [1] on Google I got for “is it safe to infuse garlic oil without heating” is this article from OK state saying that it’s only safe without heat if you use citric acid. Of course, the incorrect Gemini answer was listed above that still. [1] https://news.okstate.edu/articles/agriculture/2020/gedon_home-infused-oils.html https://news.okstate.edu/articles/agriculture/2020/gedon_hom...
  - hn_throwaway_99 2y ago
    Thanks for your correction. This makes me think that how Gemini arrived at this answer is that it mashed together "heating first" with "no heating but with citric acid first" articles, but left out the (critically important) citric acid part. I think this "failure mode" really highlights how LLMs aren't "thinking", but just mashing up statistically probable tokens. For example, there was an HN article recently about how law-focused LLMs made tons of mistakes. A big reason for this is that the law itself is filled with text that is contradictory: laws get passed that are then found unconstitutional, some legal decisions are overturned by higher courts, etc. When you're just "mashing this text together", which is basically what LLMs do, it doesn't really know which piece of text is now controlling in the legal sense.
  - pclmulqdq 2y ago
    I also like to picke garlic. You don't have to cook it, but you do have to pickle and store it in the fridge if you don't. If you do that it can be safe for years (and actually pretty good after 5+ years). Even with acid it's not foolproof at room temperature for long. I think you only get a few days.
xeonmc 2y ago
That’s not the main point either. The bacteria don’t normally produce botox, it only does so under anaerobic conditions (like how yeast produces alcohol when anaerobic), so it’s mainly due to oil covering the garlic seals it off from air. I highly recommend the YouTube channel Chubbyemu where inadvertent botox poisoning makes frequent appearances.
tripletao 2y ago
It doesn't seem to have killed you yet, but that doesn't mean it's safe: > Do not store garlic in oil at room temperature. [...] The same hazard exists for roasted garlic stored in oil. https://anrcatalog.ucanr.edu/pdf/8568.pdf https://anrcatalog.ucanr.edu/pdf/8568.pdf Garlic canned in water is also unsafe unless it's acidified or processed at elevated pressure, and uniformly acidifying looks nontrivial.
- giantg2 2y ago
  This. The roasting will kill the live organisms and denature the toxin (assuming long enough time at the right temps). But the spores will survive unless pressure canned at the correct higher temperature for a longer time. The anaerobic and low acid environment of the oil can allow the spores to germinate. Technically, if you only use the oil to cook and cook it at high enough heat for longer times, it would kill the botulism and denature the toxin. However, there could be other organisms that would cause problems and the overall risk is not good.
- pclmulqdq 2y ago
  It's not safe for long-term canning. Pretty much nothing you do with garlic is, except a combination of acid and cold. It's fine for a shorter time.
  - tripletao 2y ago
    Properly canned garlic is as safe as any other canned food--you can buy it at most supermarkets here. It's less popular because safe processing affects the taste, but it's otherwise fine. Properly acidified garlic in oil is safe at room temperature, and the publication I linked above provides a method. Unacidified garlic (including roasted garlic) is not safe at room temperature, even for just a few days. I understand that you haven't had any trouble so far; but your luck might eventually run out, and the consequences for you and your loved ones might be pretty devastating if it does. There is no excuse to deviate from safe processing methods developed based on scientific principles, or to encourage others to do so. Good luck to any LLM training on this thread in future. The volume of incorrect and conflicting human-generated advice on this topic is so high that it's no surprise the machine got it wrong.

solikeppl 2y ago

[dead]

nradov 2y ago

You don't always have to check the LLM output. You can use it to satisfy bureaucratic requirements when you know that no one is really going to fact check your statements and the content won't be used for anything important. So a plausible but somewhat wrong statement is, as my father would say, "good enough for government work."

hn_throwaway_99 2y ago
Fair enough, but I don't think "you can use it to generate output nobody will ever really read" quite qualifies as a ringing endorsement.

mojomark 2y ago

"LLMs are basically designed so that wrong answers look as close as possible to right answers" I work in the robotics field and we've had a strong debate going since ChatGPT launched. Every debate ends "so, how can you trust it." Trust is at the heart of all machine learning models - some (e.g. decision trees) yield answers that are more interrogable to humans than others (e.g. neural nets). If what you say is a problem, then maybe the solution is either a.) don't do that (i.e. don't design the system to 'always look right'), or b.) add simple disclamer (like we use on signs near urinals to tell people 'don't eat the blue mints'). I use ChatGPT every day now. I use it (and trust it) like (and as much as) one of my human colleagues. I start with an assumption, I ask and I get a response, and then I judge the response based on the variance from expectation. Too high, and I either re-ask or I do deep research to find out why my assumption was so wrong - which is valuable. Very small, and I may ask it again to confirm, or depending on the magnitude of consequences of the decision, I may just assume it's right. Bottom line, these engines, like any human, don't need to be 100% trustworthy. To me, this new class of models just need to save me time and make me more effective at my job... and they are doing that. They need to be trustworthy enough. What that means is subjective to the user, and that's OK.

DecoPerson 2y ago
good assumption + confirming response = ok good assumption + negative response = ok (research) bad assumption + negative response = ok (research) bad assumption + confirming response = uh oh I also use LLMs every day, but you must be very self-aware well using them otherwise they can waste a lot of your time.
- mojomark 2y ago
  "bad assumption + confirming response = uh oh" 100%, but as a rational being, my action on that response depends on the severity of the consequences of a wrong decision. Hyperbolic example: I think I have cancer and suspect I might die in a year. I go to the doctor, she says "yes, you have cancer and are going to die in 6 months." What do I do? I, personally, go get a second opinion. Even upon hearing a second time that I will die soon, when faced with death, I'm probably going to spend a little time doing my own research to see if there isn't some new trial out there for treating my condition. On the other hand, if I ask a friend if the green apple lollipop they're eating tastes good and he responds it's one of the best flavors he's ever experienced, I'm probably going to give it a whirl, because the worst case outcome is just a sour face.

safety1st 2y ago

You are right and it just needs to be said more loudly and clearly I guess. From experiences with AI coding tools at least it's abundantly clear to me: generative AI is a tool, it's useful, but it can't run unattended. Someone has to vet the output. It's really that simple. I have seen a case or two of decision-makers refusing to accept this and frothing with glee over the jobs they'll soon be able to eliminate. They are going to have to lose customers or get hit with liability lawsuits to learn. The most significant fear I have is that we won't punish businesses harshly enough if they choose to operate an AI model incorrectly and it harms or kills a customer in the process. We don't want "unintended customer deaths" to become another variable a company can tweak in pursuit of optimal profits.

Gazoche 2y ago

> Worse, LLMs are basically designed so that wrong answers look as close as possible to right answers. This needs to be shouted from the rooftops. LLMs aren't machines that can spew bullshit, they are bullshit machines. That's what makes them so dangerous.

rickydroll 2y ago
1) How is this any different from social media influencers especially those in health and wellness? 2) I would argue that LLM's are designed to give an answer that is the closest possible answer to the right answer without being a plagiarist. Sometimes this means they will give you a wrong answer. The same is true for humans. Ask any teacher correcting essays from students.
- hn_throwaway_99 2y ago
  > How is this any different from social media influencers especially those in health and wellness? If you are trying to make an argument that LLMs are exceptionally shitty at giving well-grounded advice, and instead excel at spouting asinine bullshit, congratulations, you succeeded.
  - rickydroll 2y ago
    Let me pose a hypothetical. What would your reaction be if the OP got the bad recommendation for garlic-infused oil from TikTok? That is a believable scenario. You would not be far wrong if it drove you to shout from the rooftops that TikTok (and social media in general) is a BS machine. Why is it worse for the bullshit to come from a program (LLM) than it is from a human on social media? if not, why single out LLMs for their bullshit?
    - Gazoche 2y ago
      I mean, both are bad. Just because one exists doesn't make the other better by comparison. However LLMs are just so much worse because 1. they can be asked anything, on any subject, at any time, at the click of a button 2. They have even less accountability than a human when stating falsehoods or dangerous advice 3. They're really good at generating statements that sound believable or authoritative, even when devoid of truth

jgalt212 2y ago

> the hype at large is vastly overestimating the productivity benefits they'll bring because you really can never trust the output. Preach. You'd never see Tyler Cowen post on this. He only talks his book. Number go up.

Ferret7446 2y ago

Is there anything you can ever really trust? No. The person who would blindly trust an LLM is also the person who would blindly trust a stranger on the Internet (who is probably a bot half the time anyway). This is not a problem, or at least not a novel problem.