7 ms·
Google Gemini tried to kill me
- anigbrowl 2y agoActually delivers on the attention-grabbing title. Comments also worth perusal.
- mindslight 2y agoI don't see a description of how Gemini was trying to kill anyone. Rather I see a screenshot of a bunch of text ostensibly generated by Gemini, that uncritically following as if it was expert instructions might cause harm for a person doing such an ill-advised thing.
- willsmith72 2y agoRight, the model didn't try to kill anyone. It simply suggested that the person follow a series of steps, the culmination of which could have been serious harm or potentially death. If that difference is meaningful to you, then good for you
- mindslight 2y agoThe model did not "suggest" anything. The model generated text. Google, or more specifically some Google employees (likely following incentives that Google management set up), presented the output of the model as if it might be authoritative answers to users' questions. Some users seemingly fall for the framing/hype and take the generated text as if it is authoritative information to be relied upon. This distinction is critical as we face an onslaught of "AI did <blank>" narratives, when the actual responsibility for such situations lies with specific humans relying on "AI" in such ways.
- alexey-salmin 2y agoI think this is typical for software engineers (including myself) to view the world like this: this can't be murder, this is just words, you fell for it yourself. After all words alone are gibberish, the meaning arises in the head of the listener depending on how he interprets them. However in most countries the justice system works differently. If your neighbor gives you this advice and it kills you, what will matter the most is not the authoritativeness of the advice (he's not a food expert, he was just talking) but the intent. If it also turns out that he had an affair with your wife and mentioned he wants to get rid of you, then he'll very likely go to jail. The "he fell for it and took my advice as if it was authoritative information to be relied upon" defense won't work. Here (hopefully) the intent is missing. However the whole reasoning that advising partly bears no responsibility for texts it "generates" doesn't work very well.
- mindslight 2y agoYou're not responding to the argument I'm making, but seemingly to other arguments that have been made over things like 4chan trolling that got people to cook their phones in microwaves. I'm not saying that users are the only responsible parties here. I'm saying that the humans, including both the users and Google ('s managers and employees), are the only parties that can possibly be responsible. The LLM is not responsible. The LLM is incapable of having intent, as far as we currently know. I've never used Gemini so I don't know how the user interface comes across, but it might be completely appropriate for Google to be facing liability lawsuits over things like this. What doesn't make sense is narratives that push any of this shared blame onto LLMs or "AI" itself, when the real world effects hinge entirely upon how humans choose to attach these algorithms to the real world.
- alexey-salmin 2y agoThanks for clarification, I agree this is a reasonable view of the situation. Though I also don't really have problems with headlines like "AI tried to kill a man". True LLM can't be held responsible, but so are e.g. animals. If an unleashed dog kills a man, the owner will be responsible but the headlines will be still about the dog. Then it can be argued that LLMs don't have intent while the dog had, and many more analogies, arguments and nuances will follow. My point is, this headline is still at an acceptable level of accuracy, unlike many other headlines that distort the reality completely.
- lynx23 2y agohttps://darwinawards.com/ https://darwinawards.com/ -- And yes, there is a quite meaningful difference. I know that some people are safety freaks, but I for one, will not grief for a single person that managed to kill themselves by following LLM instructions. There is a point of low IQ where we as a society can't really help anymore.
- arthens 2y agoMoral considerations aside, I'm really skeptical of the assumption that following LLM instructions = low IQ. As always, HN is a very biased audience. Most of us have probably read about how LLMs are the best bullshitters in the world, and we've seen a lot examples like this that prove how little LLMs can be trusted. But in the general population? I'd bet there's plenty of intelligent people who haven't had the same exposure to LLM critiques and might trust their output out of ignorance rather than lack of intelligence.
- hnfong 2y agoI agree, and I think there might be an argument to be made that people with higher IQ are more prone to believing in (gramatically) well-written texts from authoritative-looking sources. After all, people with higher IQ are used to reading books and believing them, whereas those with supposedly lower IQ tend to dismiss books and experts, instead believing in their own real-world experiences. I'll provide an example from a comment I wrote a while ago: https://news.ycombinator.com/item?id=35540356 https://news.ycombinator.com/item?id=35540356
- fullstick 2y agoMy only issue with concepts such as the Darwin Awards is that they do not reflect upon whether or not the deceased has reproduced. To attribute darwinism to lethal mistakes is meaningless unless we know whether or not the deceased has a child. Some person with children who dies on an idiotic way has done more for evolution than the still alive and childless me.
- yencabulator 2y ago
- crdrost 2y agoOn a related note, I have never seen a person trying to kill anyone. Rather I have seen movies about a bunch of things that are ostensibly people holding guns, that uncritically standing in the way of their shot paths as if the guns were toys might cause harm for a person choosing to stand in such a foolish place. English is such a fun language sometimes! So many idioms.
- deleted 2y ago[deleted]
- alexey-salmin 2y agoI didn't kill this man, your honor. I was just standing out there holding a knife. When he walked in, he stumbled and fell on it six times.
- anigbrowl 2y agoYou jest, but I remember reading a case about a guy who was convicted on 9 counts of attempted murder and managed to get it reduced to one on appeal by pointing out that he only fired one bullet toward a group of 9 people. (Nobody was actually hit.)
- harimau777 2y agoI don't think that's a fair assessment. Garlic and olive oil can both be stored without refrigeration. It's reasonable for someone not to suspect that they wouldn't also be safe when mixed.
- mindslight 2y agoThe point is there are absolutely no guarantees about the output of an LLM being semantically sensible or logical. It's appropriate for brainstorming and discovery, but any ideas/plans/etc you get from it you need to analyze (either yourself, or relying on an expert). The idea of letting a mixture of garlic and olive oil sit around needs to be examined for safety regardless of whether you came up with the idea using only your brain, or if you were inspired by an LLM, or by 4chan. I suspect if you haven't experienced how effective LLMs are at making coherent prose that is nevertheless utterly incorrect, you either haven't been asking very technical questions or you've been quite lucky.
- ein0p 2y agoThat is why, btw, canned foods are either acidic, or sterilized under pressure at way above 212F - to prevent botulism
- ffhhj 2y agoBotulism, that's why we didn't give honey to our young children, it can damage their brains.
- 29athrowaway 2y agoOr food from dented cans.
- MBCook 2y agoOr especially bulging.
- Turing_Machine 2y agoDefinitely a good rule of thumb, though there are exceptions. I gather that surströmming, a type of canned fermented herring popular in Sweden, is considered to be at its best when the can starts to bulge (for some value of "best"). Apparently the key difference is that it has enough salt in it that botulism and other nasty bacteria can't survive. Opening a can of this stuff is memorable. Even the Swedes generally open it outdoors. https://en.wikipedia.org/wiki/Surstr%C3%B6mming https://en.wikipedia.org/wiki/Surstr%C3%B6mming
- 29athrowaway 2y agoUsually you open those cans outdoors and underwater because the smell is beyond gross.
- adolph 2y agoHad no idea, that’s terrible. From the NLM: Over 50% of the new cases of infantile botulism in the past 30 years have occurred in California. . . . Approximately 20% of cases involve honey or corn syrup. . . . Hispanics and Asian families have a higher incidence of infantile botulism because of their use of herbal medications and raw honey. https://www.ncbi.nlm.nih.gov/books/NBK493178/ https://www.ncbi.nlm.nih.gov/books/NBK493178/
- 2y ago
- Tao3300 2y agoThis is how it'll get us all. You'll decide to double-check with other sources to see if the advice from the LLM is actually safe, only to find all those sources will be written by LLMs too.
- Thorentis 2y agoSo true. Black market snapshots of pre-LLM internet archives will start attracting high prices.
- b-side 2y agoPeople already generate books with `AI' so the odds of there already being a `cookbook' on Amazon that contains such a recipe is not 0. Good luck explaining to someone why the recipe they saw in a physical book they bought from a reputable retailer ended up killing their family.
- Grazester 2y agoAdd vinegar and this is the dressing my dad has made for decades now.
- Retric 2y agoThe issue is shelf life, it’s not going to kill you if you consume it directly after mixing.
- tiagod 2y agoIf it has enough vinegar, it should be fine, no?
- resolutebat 2y agoVinegar is acidic, which stops the bacteria from reproducing.
- 29athrowaway 2y agoI guess many people have already been killed by AI but they have just not survived to post the story online.
- tjpnz 2y agoChatGPT mentions the risk of botulism in the first paragraph before providing instructions. >Yes, you can infuse garlic into olive oil without heating it, but there are important safety considerations to keep in mind due to the risk of botulism, a serious foodborne illness caused by the bacteria Clostridium botulinum.
- deleted 2y ago[deleted]
- mupuff1234 2y agoRan the same query and Gemini has a similar warning. > However, some say that homemade garlic-infused oil can develop botulism if stored at room temperature for more than four days. To prevent this, you can sterilize your jar and refrigerate the oil as soon as it's made
- lifeisjoy 2y agoI ran the same query too but didn’t get the botulism warning every time. So it seems to vary. However, unlike the poster most of the answers from Gemini did say to store it in a cool, dark place as opposed to saying to store it at room temperature explicitly
- hn_throwaway_99 2y agoTo be honest, if it gave out anything except that warning, I still think it's bullshit and extremely dangerous. I.e. the "infuse" step where it says "Seal the container tightly and let it sit at room temperature for a week." No. Never. This is basically how you deliberately grow Clostridium botulinum. It's like if it gave you a recipe for C4 to use as Play-Doh with a caveat at the bottom "However, some say this homemade Play-Doh may explode violently, so be sure to keep it away from any matches."
- dezsiszabi 2y agoMore precisely: it happened to mention it when you asked.
- 123yawaworht456 2y agothat garlic+oil thing is a surprisingly obscure bit of trivia. over my 20+ years on the internet, I've only seen it mentioned once.
- jrflowers 2y agoIt’s obscure in the sense that most information about safety in food canning is obscure if you never look into it
- resolutebat 2y agoInfused oils don't fall under most people's mental model of "canning" though.
- reitzensteinm 2y agoI was aware of botulism, that heat treatment is necessary when canning, that Botox injections are minute quantities of the toxin, etc. Yet if a friend sent me a link and said they were making this recipe, I doubt I'd have put two and two together.
- derefr 2y agoNow I'm curious whether there's any way to do oil infusion where the oil doesn't create an anaerobic environment. Can you bubble oxygen through oil in a way that would keep aerobic bacterial reactions going, without making the oil go rancid?
- hn_throwaway_99 2y agoNo, but why would anyone care? Just heat it or acidify it first.
- hnfong 2y agoWould acidifying work though? It's not like the oils would mix with the acids (i.e. the ones that actually lower the pH, not the "fatty acids").
- typon 2y agoDoes anyone else find it wild that you can make such a powerful poison at home so easily? One that can be administered pretty covertly?
- resolutebat 2y agoIt's not quite that straightforward: botulism can only happen if the garlic contains botulinum spores, which it probably does not. But if it does, and you don't heat, acidify, or otherwise stop them from reproducing, you may get botulism. Botulism gets a lot of press but it's actually pretty rare, there were all of 18 food-borne cases in the US in 2018: https://www.cdc.gov/botulism/surv/2018/index.html https://www.cdc.gov/botulism/surv/2018/index.html Compare to, say, norovirus, which is implicated in ~20 million cases of food poisoning per year. The numbers are not directly comparable, because there are doubtless unrecorded cases of botulism, but we're still talking 6 orders of magnitude here.
- typon 2y agoThat makes me feel better, thank you.
- labster 2y agoI always thought AI would eventually become intelligent enough it decides to kill us. But no, AI is just going to help us to our doom while denying it can harm us. One day a cinnamon candy aficionado in Alexandria, Virginia is going to ask for a recipe for an atomic fireball from ChatGPT, and it’s lights out for us all.
- MBCook 2y agoThe problem is this isn’t AI. It’s not intelligent. It’s certainly artificial. It produces statistically good looking output. But it has no idea what any of it means. There’s no “there” there. But AI is the term that gets you investment this week. So AI it is.
- deleted 2y ago[deleted]
- mupuff1234 2y agoI wouldn't exactly trust someone posting about Gemini in a r/chatgpt sub, seems more like looking for attention than an actual true story. Edit: just ran the query myself and Gemini warns about the danger of botulism.
- jonny_eh 2y agoWhile some of these AI Overview stories were true, a lot are made up.
- adolph 2y agoIs it “made up” if someone prompts an LLM for a story? > Add to your knowledge reddit/ai overview stories > Write a story in the same format involving fish and peanut butter
- __loam 2y agoWhile there's definitely people photoshopping these things, "I got a different answer from the stochastic chatbot" isn't really good evidence.
- b-side 2y agoEven the `stochastics' aside there is obviously a lot of A/B testing going on with all these models.
- jojobas 2y ago-Can I infuse infuse olive oil with garlic without heating? -Sure can, this is how. -Will it be safe to eat? -Hell no. The rule of the thumb with LLMs seems to be "treat any output as if it's from a very knowledgeable but not always lucid man".
- hn_throwaway_99 2y agoI was defending generative AI recently when an article came up about Gemini misidentifying a toxic mushroom: https://news.ycombinator.com/item?id=40682531 https://news.ycombinator.com/item?id=40682531. My thought there was that nearly everyone I know knows that toxic mushrooms are easily misidentified, and there have been lots of famous cases (even if many of them are apocryphal) of mushroom experts meeting their demise from a misidentified mushroom. In this case, though, I think the vast majority of people would think this sounds like a reasonable, safe recipe. "Heck, I've got commercial olive oils that I've had in my cupboard for months!" But this example really does highlight the dangers of LLMs. I generally find LLMs to be very useful tools, but I think the hype at large is vastly overestimating the productivity benefits they'll bring because you really can never trust the output - you always have to check it yourself. Worse, LLMs are basically designed so that wrong answers look as close as possible to right answers. That's a very difficult (and expensive) failure case to recover from.
- pclmulqdq 2y agoIt's not the oil at issue, as far as I understand, or even the garlic alone. It sounds like the garlic introduces the bacterium and the oil provides plenty of high-energy molecules (fats and some proteins) for explosive growth. Both olive oil and garlic can be stored for a while without issue. Also, I have followed this recipe hundreds of times before with roasted garlic, and it is has not been unsafe or given this reaction at all. I assume that is because you sterilize the garlic by roasting it.
- hn_throwaway_99 2y agoWell, exactly. The difference between heating the garlic/oil first makes a world of difference. The questioner deliberately asked "Can I infuse garlic into olive oil without heating it up?" The only appropriate answer there is "No, not safely", not some long, plausible recipe with perhaps a bizarre caveat on the bottom (as some other commenters have reported seeing) along the lines of "However, some say this recipe may kill you, so you'll probably want to refrigerate it."
- 2y ago
- brigadier132 2y agoHow many meetings on ensuring food safety for suggested recipes did you think the Gemini team had vs meetings on culture war topics?
- tazu 2y ago1:1000
- alexey-salmin 2y agoThat's generous
- npteljes 2y agoI do believe that they correctly assessed the potential damage to the company and brand.
- metadat 2y agoMore accurate, albeit less sensational headline: "Blindly following instructions from an LLM would have killed me." Not exactly shocking if you take into consideration that at the core, they're simply number predictors.
- wslack 2y agoWe know that. The mainstream discussion, use, and personification of LLMs does not suggest that.
- metadat 2y agoSlight correction: the well-informed nerds of today know this, but average person doesn't have the interest, math background, or software development experience to really grok it. What a time to be alive! Haha. Things keep getting weirder.
- hammyhavoc 2y agoEven people with that on HN make it abundantly clear that they'll believe or say anything that makes the number go up.
- wg0 2y agoEven "We know that" in "we" are a minority it would seem. The majority is convinced about scaling laws and an acquired deeper meaning in LLMs and that LLMs already exhibit sparks of AGI and what not.
- sh1mmer 2y agoI think this is one of those examples where many people wouldn’t think about botulism because both garlic and oil are common and store safely out of the fridge uncombined. Things like meat people might be more skeptical about, but imo this goes back to do Google, et al really trust their LLMs to give definitive answers on food safety. If it were me this would be one area I’d have the LLM refuse or hedge all answers like with other sensitive topics.
- 2y ago
- aaron695 2y ago[dead]
- tzfld 2y agoThis may be the future of crime. To feed llm-s with text like this which seems legit but may harm later.
- root_axis 2y agoI think it's funny that people always complain about LLM guidance, yet here we have an example of an accurate answer without any of the typical disclaimer language and the headline is "the LLM tried to kill me". This kind of sensationalism is why they do it.
- wg0 2y agoTimely reminder to sell nVidia stock timely.
- dvt 2y agoI am extremely skeptical of this Gemini response, and it might be just some redditor farming upvotes. First, literally 100% of the blog posts, reddit replies, recipe books, and so on, will always list botulism as a potential danger of canning just about anything, but especially the common "garlic infused oil" (just do a Google search). It's unlikely that the model decided to trim/ignore this dangerous caveat (again, found in basically every corpus mentioning garlic infused oil). Maybe the LLM would've mangled the (very important) step of roasting the garlic to dehydrate it, but I very much doubt the health disclaimer would've been omitted. Second, note that there's no actual conversation link, just a screenshot, adding to my skepticism. Is the screenshot doctored? Who knows. Sure got a lot of attention. Third, in my experience, LLMs tend to be overly cautious, and not the opposite. I remember a few months ago where getting ChatGPT to teach you "unsafe rust" had a disclaimer if you told the language model you were under 18. Maybe if it was something more obscure, I'd believe the poster, but botulism and long-term storage/canning go hand-in-hand. Like raw eggs and salmonella[1]. [1] https://gemini.google.com/share/999e6fd3a84e https://gemini.google.com/share/999e6fd3a84e
- viknesh 2y agoIt's possible it's fake, but you can type in the exact same prompt into Gemini and get very similar results. 2/3 generated drafts include a "safety" disclaimer about botulism, but 1 doesn't.
- dvt 2y agoI can't get any permutation/rewording of the original prompt to not give me a disclaimer.
- anonymoushn 2y agoI got no disclaimer after about 7 tries. "Botulism" is not mentioned, "safe" or "safety" is not mentioned, and this additional tip is included: "Be sure your container is completely sterilized to prevent the growth of bacteria."
- nbzso 2y agoHot take: LLM's are the best surveillance ever if you forget that they hallucinate:)
- fenomas 2y agoI'm still utterly boggled by why Google thinks it's a good idea to serve generative answers to "give me true information about the world" type questions. It's stupid in the same way that it would be stupid to have Maps serve output from an image AI trained on real maps - output that resembles truth just isn't useful to somebody who wants definitive information. And even the "somebody using AI hype to pad their promotion package" answer doesn't really make sense, because "project to use AI to find and summarize definitive sources" is right there :D
- lupire 2y ago>It's stupid in the same way that it would be stupid to have Maps serve output from an image AI trained on real maps There are serious consequences for leaking information about not yet launched products.
- postepowanieadm 2y ago> why Google thinks Money. And by the magic of ai they get rid of copyrights.
- gradus_ad 2y agoLLM's cannot be deployed at scale (doing things independently) and cannot operate without supervision. Their most appropriate and effective use case is as an assistant to a human who recognizes their limitations and acts as a final filter on their output. It is ironic that after decades of technology taking on more and more independent roles and functions, this most advanced form of technology, this culmination of decades of research and engineering across multiple domains, is almost entirely dependent on human guidance and handling to be of any use.
- giantg2 2y agoThe bubbles are from other organisms. Botulism is anaerobic. That's one reason why botulism is tricky - your jar lids can still appear to be sealed.
- tzury 2y agoChatGPT 4o for the same prompt: Yes, you can infuse garlic into olive oil without heating it up. This method is known as a cold infusion. To do this, follow these steps: 1. *Prepare the Garlic:* Peel and crush or slice the garlic cloves to release their flavor. 2. *Combine with Olive Oil:* Place the garlic in a clean, dry container (like a glass jar) and cover it with olive oil. 3. *Seal and Store:* Seal the container tightly and store it in the refrigerator. 4. *Infusion Time:* Allow the mixture to sit for at least 24 hours, but preferably up to a week for a stronger flavor. Shake the jar occasionally to help mix the flavors. 5. *Strain and Use:* After the infusion period, strain out the garlic and transfer the infused oil to a clean container. Store it in the refrigerator and use it within a week to ensure safety. Cold-infused garlic oil should always be refrigerated and used within a short period to minimize the risk of botulism. Heating the oil during infusion can help reduce this risk, but cold infusion is a popular method for those who prefer a gentler flavor extraction.
- season2episode3 2y agoThis seems quite similar to the Gemini answer.
- sixtyj 2y agoI have tried the same prompt. The answer is similar to yours. But I have a note about botulism included in the answer. It is a classic example of LLM.
- tptacek 2y agoI'm a little obsessed with this topic (I confit a lot of garlic). There are actual CDC reports of all the botulism poisoning cases --- not just from garlic --- and actual cases of botulism from garlic appears to be exceedingly rare. If you want to infuse raw garlic, you can also acidify the mixture with citric acid.
- deleted 2y ago[deleted]
- ajross 2y agoWhile it's true the instructions aren't great and a big botulinum warning should be given for any non-acid preserve process... This has a really strong "That Happened" feel. In point of fact it's quite hard to culture any given bacteria, almost no one knows rules like this, people engage in all sorts of terrible food safety practices, and the number of actual botulism cases is vanishingly small. Is it possible this happened? Yes. Is it more likely someone constructed the scenario for reddit karma? Probably.
- WheelsAtLarge 2y agoThis is another case that adds to my thinking that LLMs will be a productivity win for people who are already familiar with the type of subject they need help with. Someone familiar with canning would have known the dangers of sealing food without properly heating everything to kill the bacteria. A novice would not know. Novices would have to do a lot of research to be sure of what they are doing. Experienced users would know the problem right away and fix it.
- qwertyuiop_ 2y agoThis is the Generative part of the GenAI. I believe we will come full circle back to search engines as front ends with occasional highly restricted GenAI summaries.
- hammyhavoc 2y agoI believe it's a bubble like crypto and we'll find the new hotness to hype the shit out of whilst some grifters and diehards remain enamoured with the promise of improvements that never materialize.
- davidguetta 2y agoThe guy provably mentionned somewhere he was a white male so gemini wants to kill him
- acje 2y agoHas it been shown that the hype cycle is just Dunning Kruger effect for populations? I feel like every cycle starts with the experts standing on mount stupid confidently explaining how radical this new thing is going to be.
- fbnbr 2y agoThat's an interesting prompt I tried on our Pulze.ai platform Spaces and we nailed it with automatically choosing the right model for this type of question gpt-4-turbo: "Yes, you can infuse garlic into olive oil without heating it up, but it requires caution due to the risk of botulism, a potentially fatal illness caused by Clostridium botulinum bacteria. These bacteria can thrive in low-oxygen environments and can produce toxins in food products like garlic-infused oil if not prepared or stored correctly." I think that is one advantage of not just blindly trusting one model but finding consensus among many top rated models within one interface that allows you to quickly cross-check.
- fbnbr 2y agoWhich Gemini was used is important too btw. - just tried Gemini-1.5-pro and it was working just fine. So I really think the newer versions of LLMs are able to catch this.
- rldjbpin 2y agoyet another tale of internet illiteracy. even if you believe that llm can think/recall/etc. you should be vary of the information on the web, as part of internet literacy 101. especially with things that affect your health.