19 ms·

I asked Chat GPT which antacid medications are contraindicated for some medication I'm on. Easily found through NICE. It made up a severe risk of death taking

• chinchilla2020 4y ago
  It's a great bullshit generator. I think upper management needs to be more scared of the implications than ICs
• bulbosaur123 4y ago
  Was it GPT4 or GPT3.5? It spews less bullshit with each new iteration. Matter of time really.
  • mojuba 4y ago
    I don't want to have a calculator (or say bookkeeping software) that gives correct results most of the time but not always, and then hear from the developers that it will get better with each iteration. I need a calculator that is correct 100% of time, not even 99.999%, because otherwise I can't rely on it at all. In other words, the utility of a calculator that is correct only 99% of time is zero, since you can't even tell when it's wrong.
    • dr_dshiv 4y ago
      Don’t rely on floating point accuracy!
      • coldtea 4y ago
        Those are predictably correct and incorrept, in a very formal way!
    • coldtea 4y ago
      >I don't want to have a calculator (or say bookkeeping software) that gives correct results most of the time but not always And yet Excel is widely popular, despite several footguns and inaccurate calculations!
      • neatze 4y ago
        Excel has consistent (I guess coherent) flaws/bugs, unlike GPT.
    • block_dagger 4y ago
      Imagine a life or death situation where a calculation held the key. Would you rather use a calculator that is right 50% of the time or 99% of the time. The obvious choice highlights that the utility is not exactly zero. But I see your point.
    • creatonez 4y ago
      So what about large asteroid strike levels of improbability? I have to reject your premise that there is no probability where you can stop worrying about the problem. (This might be a moot point, because I'm not sure current methods can ever get to this level of accuracy, due to limitations of the training data. Needs an entirely new method or a clever insight to optimize for truthfulness with unclean training data, and InstructGPT hasn't made much progress on this, and it might not even be possible)
    • drinfinity 4y ago
      Confining an LLM to the very narrow domain of "calculators" is a mistake, I think. You wouldn't say "a programmer that is 99% correct is worthless, I need 100%". I'm pushing it, but for a more fair comparison I'd say measure it against a programmer. How often are we wrong? 75% of the time? :) being generous here. It's the tools that make us productive. I don't know about you specifically, but I don't think you'll be very productive with a bare terminal lacking any modern IDE-like or even REPL facilities. I'll ask you to come up with instantly working code every time, all the time. It doesn't work like that. You need iteration and I believe these kinds of AI have the same issues as us. There are wrong sometimes (often) and need feedback.
      • mojuba 4y ago
        > You need iteration and I believe these kinds of AI have the same issues as us. It's funny how we resort to humanizing the machines when their results are inaccurate. We don't do that with the calculator, because it's expected to be 100% bug free. When there's a bug in the calculator code we expect it to be fixed, not gradually improved. Speaking of bugs: mistakes in code is one thing, wrong output because of a fundamental flaw in the algorithm is another. The statistical machines we are dealing with work as intended, or at least the wrong output the top comment here brings up is not a bug, it's a feature. That's the difference.
        • drinfinity 4y ago
          That's (IMO) too narrow view of what a "machine" is. Complex machinery of any kind never is 100% correct and needs constant correction and maintenance. I still think approaching this as a "calculator" is awkward at best.
          • Jensson 4y ago
            > Complex machinery of any kind never is 100% correct and needs constant correction and maintenance Computers are extremely close to 100%, we generally expect a CPU to never make errors even after years of working. If it starts making any errors at all we throw it away and make a new one.
    • magicalhippo 4y ago
      Surely this is "just" a matter of teaching the LLM to recognize this is a job for say Wolfram Aplha and generate a query to it, then feed the response back to you?
    • sophiabits 4y ago
      I suspect that humans have an accuracy lower than 99.999%, and are similarly capable of producing confidently incorrect results. GPT has a lot of hype and hysteria around it, but demanding 100% accuracy from it is a bit over the top imo. It doesn't need to have 100% accuracy on any arbitrary prompt in order to be a useful and valuable tool.
      • shanebellone 4y ago
        "It doesn't need to have 100% accuracy on any arbitrary prompt in order to be a useful and valuable tool." Yet anything less than represents a danger. Never underestimate stupid.
        • whateveracct 4y ago
          Especially don't underestimate automated stupid
      • FrustratedMonky 4y ago
        Yes. WE have a higher standard for computers. But humans produce a ton of bull shit answers. Are we trying to produce an 'answer machine' or a 'human mimic'. Because the bull shit makes it more human. And a wrong answer does not mean it's broken, what goes on in its neural net to produce a confident wrong answer might be similar to our own.
    • stockboss 4y ago
      have you ever got on an airplane? are you aware that they are not 100% safe?
      • aldarisbm 4y ago
        airplanes do not execute in a vacuum eg: computer. There are external factors that might cause an airplane to fail. A fully working airplane all things equal in terms of weather and a good pilot, will be safe.
        • stockboss 4y ago
          you're missing the point. my point was: if everyone waited til something is 100% safe/complete/satisfaction guaranteed, there would be no progress in human civilization. we need to be willing to take a chance on a promising technology in order to allow for further progress. imagine if everyone said i'll wait for planes to be 100% safety guaranteed before i'll get on one - we wouldn't be flying today.
    • makapuf 4y ago
      Do you have colleagues, bosses or reports that are correct 100% or the time? 99.999%? I would love colleagues that are 99% accurate, I certainly am not unfortunately.
      • WoodenChair 4y ago
        > Do you have colleagues, bosses or reports that are correct 100% or the time? 99.999%? I would love colleagues that are 99% accurate, I certainly am not unfortunately. I see this analogy all the time in these comment sections but it's not a very good one. A person is not a tool. One of the great achievements of humanity, and in computing in particular, is that we make tools that are more accurate than we are. I expect a hammer to deliver a hard forward blow 100% of the time. If one out of one hundred times it delivers a hard backward blow, I cannot use it on a job site due to risk of injury to the user. The same is true of a calculator being used for financial transactions. And the same is true of a LLM that would be used for drug interactions as discussed in this thread. We already have 100% accurate ways of pulling data from a drug database—it's called SQL. A tool that is not as accurate is in at least some ways a step backward, even if it's easier to use due to its natural language interface.
        • Satam 4y ago
          Except hammers already do not work as expected 100% of the time, as evident by them painfully hitting the hands of the workers in mishaps. Yet we still use them.
          • mtlmtlmtlmtl 4y ago
            How is a hammer "not working as expected" if you hit your hand with it? This makes no sense. If I hit my hand with a hammer, what am I supposed to expect except causing pain and injury? If I try to light a cigarette and accidentally set my beard on fire due to my own clumsyness, did the lighter malfunction? No, the lighter did exactly what it was supposed to do, it was my hand that didn't do what I expected.
            • 4y ago
    • worldsayshi 4y ago
      Gpt is not a calculator. I suspect that it could solve calculator problems if it was given a calculator though.
    • bulbosaur123 4y ago
      So connect GPT4 with calculator using LangChain, problem solved!
  • ChatGTP 4y ago
    I'd be very careful assuming this, if you read the manual from OpenAI themselves they clearly provide warnings on this. It has become a vary more convincing liar apparently. That's not me saying it, read the "System Card".
• Maxion 4y ago
  This is an interesting comment, and highlights the accuracy problem with LLMs. But this is but one type of query that LLMs are used for, akin to an informational web search (E.g. How tall is the Eiffel tower).
• wolpoli 4y ago
  For this specific example, an LLM in front of NICE will produce the correct result. It's a matter of time before this case will be fixed. From a non-expert's perspective through, an LLM is very dependable, unless it completely goes off the rail. How would anyone know when to trust it and when to be skeptical?
  • textninja 4y ago
    Trust but verify. We’ll get processes built around this for accuracy sensitive applications. I imagine it will look something like a GAN configuration with two or more LLMs trained to adversarially critique and fact check the outputs of the other(s). Might even mimic the relationship between the hemispheres of our brains.
    • visarga 4y ago
      Two models trained from different lineages won't hallucinate the same. When you want to check for hallucination the cost is to run the task on two models. For now. But soon it looks like LLMs will be better calibrated. It seems they are well calibrated after pre-training but become less calibrated after fine-tuning and RLHF. The last stage breaks its abilities to estimate confidnce scores correctly.
  • geraneum 4y ago
    That's not how LLMs work. Including the NICE, if it actually isn't already, will not guaranty a "correct" result. It will increase the chance that the response is directly coming from the training but there is no guarante. If you are interested in why this is the case you can read this [1] post from Stephen Wolfram on how ChatGPT and in general LLMs work. This might give some insight on how and when to to use it more effectively. [1] https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/ https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-...
    • wolpoli 4y ago
      Could we instead have the LLM use NICE similar to how Bing uses the web as a reference instead? It still wouldn't guarantee a correct result, but it would that increase the reliability, right?
• malborodog 4y ago
  This kind of problem is (trivially?) solvable through the ReAct framework, like LangChain etc. Basically you get good data, vector embed it, and make sure the LLM knows where to look for accurate information.
  • visarga 4y ago
    This approach has some issues - it chunks inputs, with some overlap, but this can destroy context - the retrieved passages, when they come from different documents, have no apparent relation or could be mistakenly considered related - the model struggles to correlate data between the document snippets, taking half an idea from one side and half from the other side and mixing them up in something that doesn't really make sense
    • malborodog 4y ago
      Implementation details. Check out what's going on with LangChain, augmented retrieval, etc. We'll be able to create knowledge bases on specific subjects with vetted data, and get the bot to retrieve and summarize appropriate results while providing a citation to the original source.
  • kossTKR 4y ago
    Slight tangent but preferably i would like an AI that can go against the status quo, the elite classes, the mainstream discourses and ignore the circus. I think one of that most interesting things that can come of bots like GPT-X is that it can make new connections, unravel "stuff", do extremely intricate deductive reasoning. Be the data driven arbiter of the the truth for everyone, not just the tiny established classes or cultural hegemonies. The ideological and cultural noise increasingly smokescreen any realpolitical, material or resource-oriented analysis of actual economic power structures in the world in the last years, AI could be a godsend (or the opposite unfortunately). I remember reading a sci-fi years ago about the stuff an AI concluded when asked philosophical questions that were so bizarre and frightening that people shut it down, and i'm sure we're in the same territory with political and scientific analysis. It's either dangerous to the orders of the world, or not that interesting and borg like on a philosophical level.
• dragonelite 4y ago
  Thats why im not falling for the hype again, i think i have seen like 2 previous AI hype cycles and all those fell off after like 3 months. Same with full automatic driving stuff.
  • textninja 4y ago
    This is a black swan event which, if anything, vindicates the hype from dreamers who were ahead of their time. They had a lot of the theory right but lacked the horsepower. This will complete the last mile for other AI tech and be the lacquer that gives it the all important finishing touch.
    • shanebellone 4y ago
      "This is a black swan event" By definition, black swan events are unpredictable. This example isn't that.
      • visarga 4y ago
        LLM emergence was unpredictable in 2017. Even for the best AI people.
        • shanebellone 4y ago
          ChatGPT is nothing more than long-form autocomplete. The specifics of approach are irrelevant.
          • imposter 4y ago
            A human brain is nothing more than long-form autocomplete. The specifics of approach are irrelevant.
            • FrustratedMonky 4y ago
              Exactly. All the people saying LLM's aren't important because it is just auto-complete, really don't connect the dots that humans are also just auto-complete.
• ramraj07 4y ago
  Not sure if you tried GPT-4 but my experience with 4 is quite different. It has been quite bullshit free, though not completely. For example I asked it to contrast oral and injectable semaglutide formulations. It did a bang up job. One thing I always do is ask it for evidence. And then I look the references up. Sometimes the references don’t say exactly what it said they will. I come back and have a discussion with it. It’s a back and forth for sure, but it’s an order of magnitude more informative and at least as accurate as a google search result is for me. I come out of it learning much more than I wanted to every single time. It’s pretty much part of my daily work and life, and I spend hours on it hitting the api limits now. So I’m not sure what others are telling about how it’s still shit. If this AI system progresses no further, it’s already transformative. It’s an intelligence multiplier for some (I’m squarely in that category, just not sure if it’s 2x or 5x), but clearly for a lot of others it’s going to be something that takes away their livelihood. I already use it for my coding work, one-way only, since I can’t paste proprietary code into it yet. The day something gpt-4-smart can plug into my orgs codebase, each of us will get at least 2x more efficient, conservatively.
  • rl3 4y ago
    This tracks with my experience as well. It's not perfect and can be a little frustrating at times, but the capability provided is extremely powerful to the point of being game-changing. It can pretty much augment any workflow in a net-beneficial way, provided you properly account for its shortcomings.
  • visarga 4y ago
    > The day something gpt 4 smart can plug into my orgs codebase, each of us will get at least 2x more efficient, conservatively. Can't wait for GPT-4 VSCode integration. What I want most is to have it see the errors and files (file formats, directories, etc) so it will automatically know what is where and how it is structured. Not just code, but also data files. In the mean while I am starting to format my code in such a way as to contain this information, put there by me by hand. Fully documented files are better for GPT.
    • KurtMueller 4y ago
• fendy3002 4y ago
  but it's very good for experts or those who're very knowledgeable and want to dig quick information / bootstrapping something, since they can digest whether the information is correct, useful, or wasteful.
• visarga 4y ago
  You should never medicate yourself after using a LLM, not in 2023 anyway! You should not run any code you have not vetted, or take any pills you don't know what they do.
  • vincnetas 4y ago
    Thats the problem, most people do take pills without knowing how they work, because doctor told them to take them. Also no one (probably) is vetting all the code they run on their computers. So this advice is kinda victim blaming. You ran the code that you did not verify? too bad, your own fault. You took advice from LLM? too bad, your own fault. You got tricked in the street by a crook? Too bad, your own fault. So what are we discussing, is, should we try to put guardrails around dangerous things so that inexperienced/vulnerable people would not get damaged.
    • visarga 4y ago
      No, I verify the code I generate with chatGPT or Codex. I don't verify code I download.
      • vincnetas 4y ago
        I mean all the code that you run on your computer. Not just generated by AI. I think parent meant the same.
• kossTKR 4y ago
  You didn't use version 4 did you? Because if feel this is pretty much already out of date. 4 does this _a lot_ less! And 5, 6 or whatever will probably be better, so i don't even really get the point here.
  • js8 4y ago
    I find there are two kinds of people in the world. First are saying that LLMs are bullshitting. The second are bullshitting about whatever LLMs are saying.
    • grugagag 4y ago
      I find there are more than two kinds of people. Among them some are skeptical and some are not. The latter are good at exploring, even if blind alleys while the former keep them in check. We’re lucky we have some variety otherwise we’d risk go full force into dead ends or we’d ignore fruitful possibilities.
    • Der_Einzige 4y ago
      The duality of mankind. On one hand, we have skeptics asserting that LLMs are masters of malarkey. On the other, we've got those regurgitating the verbal gymnastics of said LLMs. But lo and behold, there exists a third breed: those who delight in deciphering the eloquent dance between the bullshitters and the bullshitted.
• tap-snap-or-nap 4y ago
  I asked it "What is the accuracy and confidence level (you must rate it out of 10) that the above answer is correct ?" It always generates a random number when it is caught making up bullshit sometimes claims that the information 10/10 accurate and LLM's always provides accurate information.
  • sorz 4y ago
    OpenAI did exact the test for GPT-4. The raw, non-fine-tune GPT-4 is quite good at predicting confidence level ("highly calibrated" by their words). But the RLHF fine-tuning process seems ruin its calibration. Figure 8 on page 12 of GPT-4 Technical Report shows this dramatic changes before & after fine-tuning.
• xmcqdpt2 4y ago
  I've been using Bing AI instead of Google for a few days to test it out. It gives responses with citations. A few times already it has hallucinated details that aren't on the cited website at all, but that do sound very plausible (for example something about the horns of a bull pointing up when I asked about the etymology of "bullish".) I think it comes down to the fact that most stuff is actually like 90% bullshit so when you train a model on a corpus of everything you end up with a decent bullshit generator. Which is fine for many purposes but I'm not sure it will take over search.
  • penjelly 4y ago
    i agree citations dont always support bings conclusion, thats been my exp as well
  • Joeri 4y ago
    It is telling that ChatGPT by training on online discussions hasn’t learned to say “I don’t know”.
    • worldsayshi 4y ago
      I guess the "I don't know"'s are usually silent.
      • theRealMe 4y ago
        Lmao. Now I’m imagining any questions posted anywhere like Ask Hacker News, Ask Reddit, or Quora being filled with everyone that doesn’t know the answer replying “I don’t know.”
        • harvey9 4y ago
          Product q&a on Amazon has this. It is because Amazon emails you other people's questions about products you have previously purchased, and many users don't realize there is no need to reply just to be polite.
    • 4y ago
• bitwize 4y ago
  ChatGPT is like the robot from Asimov's short story Liar!: It doesn't tell you the truth, it tells you what it thinks you want to hear. Instead of reading minds it returns a statistically plausible response to queries similar to yours.
• golergka 4y ago
  Stop evaluating tech by misapplying it. ChatGPT is not really good or reliable at factual questions, yes. But it's fantastic at transforming input data that you provide to it.
• dboreham 4y ago
  This has been my experience: Chat GPT is astonishing in terms of producing plausible natural language text that looks related to the question. That's an astounding achievement imho. It does not (and I suspect wasn't expected to) produce correct answers to questions.
  • boringuser1 4y ago
    [dead]
  • pwinnski 4y ago
    I had an extended back-and-forth in which I asked ChatGPT how to undo mistakenly marking a message as junk in Messages on iOS. It gave me a 7-step answer that was completely wrong. Then I said no, you can't do that, it apologized and gave me an 8-step answer that was completely wrong. When I pointed out in this case there is no "Junk" folder in iOS Messages, it apologized for the confusion and gave me another 8-step answer that was completely wrong. When I pointed out why that one wouldn't work, it gave up and said recovery was impossible and that I would need to contact the sender to re-send, and be more careful when marking messages as junk. This was still wrong, as recovery is possible, just not by any of the means it described. So yeah, I have been super-impressed by the quality of output from these LLMs, but I cannot imagine actually relying on one for anything where correctness matters. Giving me a nice list of Korean shoegaze bands, sure. Its step-by-step for how to become a better volleyball player will be great for my daughter, and was better than the answer to the same question from Google. But correctness? No.
• boh 4y ago
  I think AI's hype machine is definitely going to negatively affect its future. If people are made to believe it's able to automate everything and it gets just one crucial thing wrong, the loss of credibility at a time when it's supposedly a finished product will be considerable. If it gets something wrong and it's made clear it's still under development the reaction will be less negative. Currently the hype puts the current state of AI somewhere between Rosey the robot and Skynet. I don't think the realization of how far we are from either will do much to promote adoption.
• the_only_law 4y ago
  > It made up a severe risk of death taking a very common medicine combo. Probably trained on too many of those SEO hyper optimized medical sites people get when they search symptoms and such.