9 ms·

> Write a short note explaining your decision for every application Is there any evidence or reason to suspect that this would result in the desired effect? (

• Al-Khwarizmi 4y ago
  Not to refute what you said, but what you describe is quite similar to what we humans call rationalization, and it has been argued (e.g. by Robert Zajonc) that most of the time we make decisions intuitively and then seek a rationalization to explain them. Also, good luck with human explanations in the presence of bias. No human is going to say that they refused a loan due to the race or sex of the applicant.
  • pixl97 4y ago
    Well no smart humans, but it turns out there are plenty of dumb ones.
  • delusional 4y ago
    There's probably a lot of truth to this, but there's also a big difference. Humans are ultimately responsible and accountable for their actions. Should my actions turn out to carry a racist or sexist motive, even unbeknownst to me, i can be held to answer for that. That's a big difference. The rationalization from a human is valuable because it's delivered by the accountable party. From a machine such rationalization is at best worthless, since you can't hold the machine accountable at all.
    • jvanderbot 4y ago
      But what youre saying only layers over the fact that, to avoid getting caught and convicted, a sufficiently good explanation is all that's necessary.
      • delusional 4y ago
        Not quite. What I'm saying is that the explanation is valuable because it comes from the accountable party. That you have a good explanation does not absolve you of responsibility. The short and clear way to put it is this: You don't go to jail because YOU think you did something wrong, you go to jail because WE think you did something wrong. If I'm a customer at a bank, and my loan has been denied, I don't care what some unaccountable AI system can come up with to explain that. I care what about how the accountable bankers justify putting that AI system into the process in the first place. How do they justify that AI system getting to make decisions that affect me and my life. I don't care about why the process does what it does, I care about why that is the process.
        • coldtea 4y ago
          >If I'm a customer at a bank, and my loan has been denied, I don't care what some unaccountable AI system can come up with to explain that. I care what about how the accountable bankers justify putting that AI system into the process in the first place. How do they justify that AI system getting to make decisions that affect me and my life. I don't care about why the process does what it does, I care about why that is the process. Well, a court/law just has to declare "AI" as allowed to be used in such decisions, and the whole recourse you describe vanishes though...
        • chii 4y ago
          > my loan has been denied, I don't care what some unaccountable AI system can come up with to explain that. > How do they justify that AI system getting to make decisions that affect me and my life. so you, a priori, make the assumption that your loan _should've_ been accepted? If the decision wasn't an AI, but some actuarial that calculates and computes based on a set of criteria, and the result is a denial, you could still make the same argument of "why is _this_ the process, instead of something else (that makes my loan acceptable)?".
  • geraneum 4y ago
    Although humans are capable of rationalizing and they do it, they have other capabilities which is a key point that is missing in this argument. Maybe no one will admit to refusing a loan based on applicant’s gender, but also no real world aircraft engineer will explain why they decided to design a plane’s wing in a certain shape, purely by rationalizing an intuition without backing it by math and physics. Also, there are a group of humans elsewhere that understand those math and using the “same” principles can follow the explanation and detect mistakes or baseless rationalized explanations.
    • composter 4y ago
      Using airfoil design rationalization as an example may not be the best choice in this context since there is a fairly long history of incorrect rationalizations for wing lift. "Incorrect Lift Theories": https://www.grc.nasa.gov/www/k-12/VirtualAero/BottleRocket/airplane/wrong1.html https://www.grc.nasa.gov/www/k-12/VirtualAero/BottleRocket/a... "No One Can Explain Why Planes Stay in the Air": https://www.scientificamerican.com/article/no-one-can-explain-why-planes-stay-in-the-air/ https://www.scientificamerican.com/article/no-one-can-explai...
      • SonicScrub 4y ago
        Ugh, not this clickbait again. The actual title of that article should be "explanation of flight defies a simple intuitive explanation that is both wholly correct, and understandable to the layman". Very different then "no one can explain it!"
        • composter 4y ago
          I couldn't agree more! However, I think the content of the article is informative. If you strongly disagree I'm happy to remove the link to the article
          • SonicScrub 4y ago
            No, I do think it's an informative, well written article. It's quotations come directly from some the most well-known, and respected names in the field of the aerospace. I've even referenced it before on this very site! I just bemoan it's terrible title, and am perhaps bitter from the times when relatives send me links with similar titles (usually of lesser quality) and I have to explain that yes, people in my field do know what their doing. The infamous "bees" quote being a particular bug-a-boo. I would however critic it's use as an example to prove that we have a history of rationalizing explanations where none exist (and using that to draw a parallel with AI). While the title implies this conclusion, the article itself does not. We do indeed have a very good explanation of how aerodynamic lift works. That explanation just takes the form of a set of differential equations, and isn't something one can easily tell a group of 5th grader, without simplifying to the point of spreading errors.
  • ElFitz 4y ago
    Except for things we decide to be important enough that we should create and apply models. Now, the decisions that went into these models might have been rationalised after the fact. Or biased. But these handmade models can been reviewed by others, the logic and decisions that went into them can be challenged, and rules can be changed or added based on experience. Not so much with LLMs.
  • xthrown1 4y ago
    The difference is that I can ask it to explain it's decision as a racist and a anti-racist and it will happily hallucinate an answer for both.
  • dr_dshiv 4y ago
    Another term is “confabulation” — which has also been used to explain human linguistic reasoning
• micromacrofoot 4y ago
  A vast amount of the world is built on “close enough” and this is no different
• cs702 4y ago
  > Is there any evidence or reason to suspect that this would result in the desired effect? Yes. There's evidence that you can get these models to write chain-of-thought explanations that are consistent with the instructions in the given text. For example, take a look at the ReAct paper: https://arxiv.org/abs/2210.03629 https://arxiv.org/abs/2210.03629 and some of the LangChain tutorials that use it: https://langchain.readthedocs.io/en/latest/modules/agents/getting_started.html https://langchain.readthedocs.io/en/latest/modules/agents/ge... https://langchain.readthedocs.io/en/latest/modules/agents/implementations/react.html?highlight=zero-shot%20react https://langchain.readthedocs.io/en/latest/modules/agents/im...
  • loxias 4y ago
    Thanks! I was genuinely asking, will need some time to read and digest. LangChain looks interesting.
    • cs702 4y ago
      Thank you. Also search the web for posts on "prompt engineering," to get a sense of how people are using LLMs today. It's pretty eye-opening.
  • vgel 4y ago
    But note that just because a model says it's using chain-of-thought or tools to come to a certain conclusion, doesn't necessarily mean that it is: https://vgel.me/posts/tools-not-needed/ https://vgel.me/posts/tools-not-needed/
    • cs702 4y ago
      Yes, I agree. But note that the same logic applies to human beings too. Just because people say they are using chain-of-thought or tools to come to a certain conclusion, doesn't necessarily mean they are. Philosophers have been grappling with this issue for many centuries :-)
      • vba616 4y ago
• simsla 4y ago
  Yes, look up "chain of thought prompting". It's also been shown previously that asking a model to write down its chain of thought improves the accuracy of its answers. See https://ai.googleblog.com/2022/05/language-models-perform-reasoning-via.html https://ai.googleblog.com/2022/05/language-models-perform-re... That being said, I don't think ChatGPT is ready for high-risk applications like insurance/loan approvals yet. Maybe in a year or two. For now, treat ChatGPT like you would a mediocre intern.
• john-radio 4y ago
  I don't think the person you're responding to necessarily disagrees with anything you've written, which all seems quite right to me, too. But it's about the marketability of the app.
• hooande 4y ago
  A Large Language Model is a tool and people have to learn to use it, just like any other tool. I have used GPT as a machine learning model in a way similar to the grandparent comment, and I now have some understanding of when and why it hallucinates in certain situations [0]. I use that understanding to filter responses and tweak prompts. It took a lot of time and effort invested, but I was able to get a handle on it. I know that a lot of people won't do this, and will just accept whatever an LLM says at face value. And again, that's true for any tool. But if you invest the time to understand the parameters and limitations of the model, it can be incredibly valuable. [0] in my experience, GPT davinci is much more likely to hallucinate in (non-programming) situations that would be difficult for a human to explain. using the above example, it can easily handle a standard credit application. But it will be more likely to hallucinate reasoning for a rare case like someone with a very high income but a low credit score. YMMV, just sharing what I've seen
  • ip26 4y ago
    Sadly for me, that's the only thing I want it to do- ingest the entire internet's worth of data and make a well reasoned assessment about an edge case or subtle trends based on that vast trove.
    • phatfish 4y ago
      Yes, this is the same for me. The things ChatGTP gets right are often the "common" situations. It's possible you aren't aware of these so surfacing these more "common" facts can be helpful. But ask it a question about an edge case and it will hallucinate a plausible answer, and even double down when you point out a mistake. By that point it's faster to find an answer by using a few "traditional" search keywords and reading the content yourself. We can assume it will improve though, it's just not there yet for "edge cases" in it's training data.
• kixiQu 4y ago
  I'll admit most people have played more with these models than I have, but I was really struck by how it would helpfully explain the structure of a LOC call number to me in ways that were just... incoherent within the span of two sentences – claiming that the first two digits of a Cutter number were from the author's name, but also that all the digits together represented the count of works under that author initial. Either sentence might have been plausible in an explanation of call numbers within some system - but together they were self-evidently wrong – and yet the kind of thing that someone less familiar with the material explained might have been confused by, might have assumed was their own failure to understand. Suffice to say: hoo boy I hope all y'all commenters aren't tacking these APIs directly onto systems where the results have real consequences.
• LawTalkingGuy 4y ago
  The solution is to break the explanation/decision into two or more steps. Have it summarize the pros and cons of the loan. Have it score the issues. Have it explain why those issues and scores should receive a loan or not. Have it decide if those reasons are legitimate and the reasoning sound. Have it pronounce the final answer. Get the rationalization out first and then examine it. This way even if you're wrong you have multiple steps of fairly clear reasoning to adjust. If it's racially biased you can literally see it and can see where to explain not to use that reasoning.