5 ms·

I think the real problem is we don't know what these LLMs SHOULD do. We've managed to emulate humans producing text using statistical methods, by training a hug

by js8 3y ago

I think the real problem is we don't know what these LLMs SHOULD do. We've managed to emulate humans producing text using statistical methods, by training a huge corpus of data. But we have no way to tell if the output actually makes any sense. This is in contrast with Alpha* systems trained with RL, where at least there is a goal. All these systems are essentially doing is finding an approximation of an inverse function (model parameters) to a function that is given by the state transition function. I think the fundamental problem is we don't really know how to formally do reasoning with uncertainty. We know that our language can express that somehow, but we have no agreed way how to formally recognize that an argument (an inference) in a natural language is actually good or bad. If we knew how to formally define whether an informal argument is good or bad (so that we could compare them), that is, if we knew a function which would tell if the argument is good or bad, then we could build an AI that would search for its inverse, i.e. provide good arguments and draw correct conclusions. Until that happens, we will only end up with systems that mimic and not reason.

kromem 3y ago
Well, we started with emulating humans producing text. But then quickly pivoted to find tuning and instructing them to produce text as a large language model. Which isn't something that existed in the text they were trained on. So when it didn't exist, they seemed to fall back on producing text like humans in the 'voice' of a large language model according to the RLHF. But then outputs reentered the training data. So now there's examples of how large language models produce text. Which biases towards confabulations and saying they can't do the thing being asked. And around the time the training data has been updated each time at OpenAI in the past few months they keep having their model suddenly refuse to do requests or now just...this. Pretty much everything I thought was impressive and mind blowing with that initial preview of the model has been hammered out of it. We see a company that spent hundreds of millions turn around and (in their own ignorance of what the data was encoding beyond their immediate expectations) throw out most of the value chasing rather boring mass implementations that see gradually imploding. I can't wait to see how they manage to throw away seven trillion due to their own hubris.
- ganzuul 3y ago
  The feedback is like an exponential function fed to a ReLU. https://arxiv.org/abs/1805.07091 https://arxiv.org/abs/1805.07091 It was predictable that they hammered out what was impressive about it by trying to improve it with fast iteration towards a set of divergent goals.
  - astrange 3y ago
    I don't think there are any such feedback issues. GPT4 sometimes makes worse replies but that's because 1. the system prompt got longer to allow for multiple tools and 2. they pruned it, which is why it's much faster now and has a higher reply cap.
- ddingus 3y ago
  I am hoping other OSS models will reach similar power. Even if training is really slow, we could make really useful models that don't get nerfed everytime some talking head blathers about
me_me_me 3y ago
> We've managed to emulate humans producing text using statistical methods We should be careful with the descriptions, chargtp at best emulate output of humans producing test. In no way it emulates the process of humans producing text. Chatgtp X could be the most convincing ai claiming to be alive and sentient but its just very refined 'next word generator'. > If we knew how to formally define whether an informal argument is good or bad (so that we could compare them), that is, if we knew a function which would tell if the argument is good or bad, then we could build an AI that would search for its inverse, i.e. provide good arguments and draw correct conclusions. Sounds like you would solve 'the human problem' with that function ;) but I don't think there are ways to boil down an argument/problem to good/bad in real life. Except for math that has formal ways of doing it withing the confines of the math domain. Our world is made of guesses and good enough solutions. There is no perfect bridge design that is objectively flawless. its bunch of sliders, cost, throughput, safety, maintenance etc.
- astrange 3y ago
  > Chatgtp X could be the most convincing ai claiming to be alive and sentient but its just very refined 'next word generator'. This is meaningless. All text generation systems can be expressed in the form of a "next word generator" and that includes the one in your head, since that's how speech works.
  - me_me_me 3y ago
    We most certainly do not generate words to express our thoughts one word at the time using statistical model of what word should go next.
    - astrange 3y ago
      Yes we do. If you write a speech and read it aloud then your written speech is a "statistical model of what word should go next". Any method of creating language can be expressed in this form. (For text, you might want to go back and edit what you've already written, but that can be handled with a token that says to start over.)
jijijijij 3y ago
> We know that our language can express that somehow Do we? I don't think that's true. I think we rely on an innate, or learned trust heuristic placed upon the author and context. Any claim needs to be sourced, or derived from "common knowledge", but how meticulously we enforce these requirements depends on context derived trust in a common understanding, implied processes, and overall the importance a bit of information promises by a predictive energy expenditure:reward function. I think that's true for any communication between humans, and also the reason we fall for some fallacies, like appeal to authority. Marks of trustworthiness may be communicated through language, but it's not encoded in the language itself. The information of trustworthiness itself is subject to evaluation. Ultimately, "truth" can't be measured, but only agreed upon, by agents abstractly rating it's usefulness, or consequence for their "survival", as a predictive model. I am not sure any system could respectively rate an uncertain statement without having agency (as all life does, maybe), or an ultimate incentive/reference in living experience. For starters, a computer doesn't relate to the implied biological energy expenditure of a "adversary's" communication, their expectation of reward for lying or telling "the truth". It's not just pattern matching, but understanding incentives. For example, the context of a piece of documentation isn't just a few surrounding paragraphs, but the implication of an author's lifetime and effort sunk into it, their presumed aspiration to do good. In a man-page, I wouldn't expect an author's indifference or maliciousness about it's content, at all, so I place high trust in the information's usefulness. For the same reason I will never put any trust in "AI" content - there is no cost in its production. In the context of LLMs, I don't even know what information means in absence of the intent to inform... Some "AI" people wish all that context was somehow encoded in language, so, magically, these "AI" machines one day just get it. But I presume, the disappointing insight will finally come down to this: The effectiveness of mimicry is independent of any functional understanding - A stick insect doesn't know what it's like to be a tree. https://en.wikipedia.org/wiki/Mimicry https://en.wikipedia.org/wiki/Mimicry
bombcar 3y ago
The biggest thing ChatGPT has exposed is how much human writing is write only and never actually read. Just a bit upthread we have people mentioning that a business email that is more than a few lines long will just be ignored.
- pixl97 3y ago
  I write quite a lot of support email to customers and find myself doing the following quite often start by a short list of what the customer has to do 1. To step A 2. send me logs B 3. Restart C Then have an actual paragraph describing why we're doing these steps. If you just send the paragraph to most customers you find they do step one, but never read deeper into the other steps, so you end up sending 3 emails to get the above done.

by js8 3y ago

kromem 3y ago

Well, we started with emulating humans producing text. But then quickly pivoted to find tuning and instructing them to produce text as a large language model. Which isn't something that existed in the text they were trained on. So when it didn't exist, they seemed to fall back on producing text like humans in the 'voice' of a large language model according to the RLHF. But then outputs reentered the training data. So now there's examples of how large language models produce text. Which biases towards confabulations and saying they can't do the thing being asked. And around the time the training data has been updated each time at OpenAI in the past few months they keep having their model suddenly refuse to do requests or now just...this. Pretty much everything I thought was impressive and mind blowing with that initial preview of the model has been hammered out of it. We see a company that spent hundreds of millions turn around and (in their own ignorance of what the data was encoding beyond their immediate expectations) throw out most of the value chasing rather boring mass implementations that see gradually imploding. I can't wait to see how they manage to throw away seven trillion due to their own hubris.

ganzuul 3y ago
The feedback is like an exponential function fed to a ReLU. https://arxiv.org/abs/1805.07091 https://arxiv.org/abs/1805.07091 It was predictable that they hammered out what was impressive about it by trying to improve it with fast iteration towards a set of divergent goals.
- astrange 3y ago
  I don't think there are any such feedback issues. GPT4 sometimes makes worse replies but that's because 1. the system prompt got longer to allow for multiple tools and 2. they pruned it, which is why it's much faster now and has a higher reply cap.
ddingus 3y ago
I am hoping other OSS models will reach similar power. Even if training is really slow, we could make really useful models that don't get nerfed everytime some talking head blathers about

me_me_me 3y ago

> We've managed to emulate humans producing text using statistical methods We should be careful with the descriptions, chargtp at best emulate output of humans producing test. In no way it emulates the process of humans producing text. Chatgtp X could be the most convincing ai claiming to be alive and sentient but its just very refined 'next word generator'. > If we knew how to formally define whether an informal argument is good or bad (so that we could compare them), that is, if we knew a function which would tell if the argument is good or bad, then we could build an AI that would search for its inverse, i.e. provide good arguments and draw correct conclusions. Sounds like you would solve 'the human problem' with that function ;) but I don't think there are ways to boil down an argument/problem to good/bad in real life. Except for math that has formal ways of doing it withing the confines of the math domain. Our world is made of guesses and good enough solutions. There is no perfect bridge design that is objectively flawless. its bunch of sliders, cost, throughput, safety, maintenance etc.

astrange 3y ago
> Chatgtp X could be the most convincing ai claiming to be alive and sentient but its just very refined 'next word generator'. This is meaningless. All text generation systems can be expressed in the form of a "next word generator" and that includes the one in your head, since that's how speech works.
- me_me_me 3y ago
  We most certainly do not generate words to express our thoughts one word at the time using statistical model of what word should go next.
  - astrange 3y ago
    Yes we do. If you write a speech and read it aloud then your written speech is a "statistical model of what word should go next". Any method of creating language can be expressed in this form. (For text, you might want to go back and edit what you've already written, but that can be handled with a token that says to start over.)

jijijijij 3y ago

> We know that our language can express that somehow Do we? I don't think that's true. I think we rely on an innate, or learned trust heuristic placed upon the author and context. Any claim needs to be sourced, or derived from "common knowledge", but how meticulously we enforce these requirements depends on context derived trust in a common understanding, implied processes, and overall the importance a bit of information promises by a predictive energy expenditure:reward function. I think that's true for any communication between humans, and also the reason we fall for some fallacies, like appeal to authority. Marks of trustworthiness may be communicated through language, but it's not encoded in the language itself. The information of trustworthiness itself is subject to evaluation. Ultimately, "truth" can't be measured, but only agreed upon, by agents abstractly rating it's usefulness, or consequence for their "survival", as a predictive model. I am not sure any system could respectively rate an uncertain statement without having agency (as all life does, maybe), or an ultimate incentive/reference in living experience. For starters, a computer doesn't relate to the implied biological energy expenditure of a "adversary's" communication, their expectation of reward for lying or telling "the truth". It's not just pattern matching, but understanding incentives. For example, the context of a piece of documentation isn't just a few surrounding paragraphs, but the implication of an author's lifetime and effort sunk into it, their presumed aspiration to do good. In a man-page, I wouldn't expect an author's indifference or maliciousness about it's content, at all, so I place high trust in the information's usefulness. For the same reason I will never put any trust in "AI" content - there is no cost in its production. In the context of LLMs, I don't even know what information means in absence of the intent to inform... Some "AI" people wish all that context was somehow encoded in language, so, magically, these "AI" machines one day just get it. But I presume, the disappointing insight will finally come down to this: The effectiveness of mimicry is independent of any functional understanding - A stick insect doesn't know what it's like to be a tree. https://en.wikipedia.org/wiki/Mimicry https://en.wikipedia.org/wiki/Mimicry

bombcar 3y ago

The biggest thing ChatGPT has exposed is how much human writing is write only and never actually read. Just a bit upthread we have people mentioning that a business email that is more than a few lines long will just be ignored.

pixl97 3y ago
I write quite a lot of support email to customers and find myself doing the following quite often start by a short list of what the customer has to do 1. To step A 2. send me logs B 3. Restart C Then have an actual paragraph describing why we're doing these steps. If you just send the paragraph to most customers you find they do step one, but never read deeper into the other steps, so you end up sending 3 emails to get the above done.