3 ms·

> How ChatGPT does the calculations is a mystery. It’s no mystery. It doesn’t do calculations at all. You really need to think of it more like am improvisor pl

> How ChatGPT does the calculations is a mystery. It’s no mystery. It doesn’t do calculations at all. You really need to think of it more like am improvisor playing a financial advisor on stage rather than a financial advisor doing their work. There are countless samples of mortgage calculation texts in its training data, so at your request, it starts inventing such a text that uses your provided number. It knows, more or less, what numbers appear around what other numbers in this kind of text at the small scale of arithmetic and large scale of mortgage calculation texts and so it seems pretty coherent at first glance. But because you asked for a very specific thing that isn’t quite like any other in its training data, the text it includes becomes more hallucinatory and less referential. And when these hallucinations goes off the rails and no longer looks like the normal sort of mortgage calculator training text (few probably have negative results!), it starts to hallucinate the vaguely explanatory boilerplate text common around weird values. No calculations, no feelings, no understanding. Just text completion that feels inconceivably rich compared to what we’re used to, but that is literally about on par with an professional improvisor.

resource0x 4y ago
The problem is that it is not fixable in the language model without implementing special logic for mortgage calculations, like "if the user asks about mortgage calculations, do this (e.g. invoke python program) instead of regular processing. That is, this class of questions should be "special-cased" somehow. But the number of special cases like this is potentially infinite. Even to answer "I don't know how to answer this question", the engine has to be specifically instructed to do so.
- MPSimmons 4y ago
  Yeah, all specialty subjects are like this (or math that's sufficiently different from its learned text - have it square a very, very large number with a bunch of random digits and compare that against the actual answer from a calculator). To actually solve this, we need AGI.
  - iainmerrick 4y ago
    To actually solve this, we need AGI. That’s good to know -- once we reach that last milestone, we can finally stop moving those goalposts!!
    - Jensson 4y ago
      "Goalpost moving" is just a strawman, what is happening is that people give counterexamples of what the AI can't do, and then you guys the AI positivists treats those as goalposts, but they weren't. For example if I say "this bot isn't smart, it can't even play chess", I don't mean that if you hardcode a chess engine into the AI that it is then smart, I just mentioned one of the myriad of things I expect a smart bot to be able to learn. So when you come back and hardcoded chess in it, then I'll mention one of the myriad of other things I expect it to be able to learn, like playing Go for example, that wasn't me moving the goalpost, that was you interpreting my counterexample as a goalpost, the goalpost was always the same: make an AI that can learn and perform the same intellectual tasks as a typical human. Me mentioning specific counterexamples aren't goalposts, those are just the easiest way to refute the claim "this is general intelligence!". And when people like you treat those as goalposts it means that you aren't even understanding the conversation, there is no way you will convince anyone by behaving like that.
      - iainmerrick 4y ago
        Sorry to be pedantic, but the GP said "to solve this, we need AGI", implying that it is a goalpost. More generally, I don't think you have to be an "AI positivist" to note that expectations have shifted a lot, which is really all I meant by "moving the goalposts". If you had told me even 10 years ago some of the things that can be done with AI now, I'd have thought it was completely impossible -- the kinds of things done routinely now are exactly those that we used to think were particularly hard tasks for AI. And yet the consensus (which I agree with!) is that AI is still not actually intelligent. There are a couple of downsides to your strict "counterexample" approach. First, it can smack of gatekeeping a bit -- it's like trying to define art. If you can't actually say what art is, except to point out which artworks you like and which you don't like, are you saying anything useful? Is "art" even a real concept? I would like to think it is, but it's a very complicated question. Second, a lot of humans have trouble with the same kinds of things that AI has trouble with! Does that mean those humans aren't intelligent? Or are there different kinds of intelligence -- but if so, why can't AI be one of them? Right now of course you can generally tell the difference, because AI tends to fail in certain in-human ways. But if that's not a goalpost, what will it mean when it becomes difficult or impossible to tell the difference between AI errors and human errors?
- PeterisP 4y ago
  To be frank, if we take a human schoolkid who knows some natural language well, getting them to apply this "language model" to make mortgage calculations requires either to give them an external tool for that or do many weeks (at least) of training in mathematics..

resource0x 4y ago

The problem is that it is not fixable in the language model without implementing special logic for mortgage calculations, like "if the user asks about mortgage calculations, do this (e.g. invoke python program) instead of regular processing. That is, this class of questions should be "special-cased" somehow. But the number of special cases like this is potentially infinite. Even to answer "I don't know how to answer this question", the engine has to be specifically instructed to do so.

MPSimmons 4y ago
Yeah, all specialty subjects are like this (or math that's sufficiently different from its learned text - have it square a very, very large number with a bunch of random digits and compare that against the actual answer from a calculator). To actually solve this, we need AGI.
- iainmerrick 4y ago
  To actually solve this, we need AGI. That’s good to know -- once we reach that last milestone, we can finally stop moving those goalposts!!
  - Jensson 4y ago
    "Goalpost moving" is just a strawman, what is happening is that people give counterexamples of what the AI can't do, and then you guys the AI positivists treats those as goalposts, but they weren't. For example if I say "this bot isn't smart, it can't even play chess", I don't mean that if you hardcode a chess engine into the AI that it is then smart, I just mentioned one of the myriad of things I expect a smart bot to be able to learn. So when you come back and hardcoded chess in it, then I'll mention one of the myriad of other things I expect it to be able to learn, like playing Go for example, that wasn't me moving the goalpost, that was you interpreting my counterexample as a goalpost, the goalpost was always the same: make an AI that can learn and perform the same intellectual tasks as a typical human. Me mentioning specific counterexamples aren't goalposts, those are just the easiest way to refute the claim "this is general intelligence!". And when people like you treat those as goalposts it means that you aren't even understanding the conversation, there is no way you will convince anyone by behaving like that.
    - iainmerrick 4y ago
      Sorry to be pedantic, but the GP said "to solve this, we need AGI", implying that it is a goalpost. More generally, I don't think you have to be an "AI positivist" to note that expectations have shifted a lot, which is really all I meant by "moving the goalposts". If you had told me even 10 years ago some of the things that can be done with AI now, I'd have thought it was completely impossible -- the kinds of things done routinely now are exactly those that we used to think were particularly hard tasks for AI. And yet the consensus (which I agree with!) is that AI is still not actually intelligent. There are a couple of downsides to your strict "counterexample" approach. First, it can smack of gatekeeping a bit -- it's like trying to define art. If you can't actually say what art is, except to point out which artworks you like and which you don't like, are you saying anything useful? Is "art" even a real concept? I would like to think it is, but it's a very complicated question. Second, a lot of humans have trouble with the same kinds of things that AI has trouble with! Does that mean those humans aren't intelligent? Or are there different kinds of intelligence -- but if so, why can't AI be one of them? Right now of course you can generally tell the difference, because AI tends to fail in certain in-human ways. But if that's not a goalpost, what will it mean when it becomes difficult or impossible to tell the difference between AI errors and human errors?
PeterisP 4y ago
To be frank, if we take a human schoolkid who knows some natural language well, getting them to apply this "language model" to make mortgage calculations requires either to give them an external tool for that or do many weeks (at least) of training in mathematics..