11 ms·

Techniques to improve reliability

302 points by tedsanders 4y ago

Imnimo 4y ago
I feel like there should be a LLM architecture which includes "scratch space" - tokens the model can write to and read from which do not constitute part of its output. The trouble with current architectures is that they can only do a finite amount of computation per output token - they get one forward pass and then have to output something. Chain-of-thought reasoning allows the model to devote more computation to finding the answer, storing intermediate results in its output tokens. But this is silly - most of the intermediate tokens are not providing useful information towards solving the problem, they're just wasted computation: >There are 16 balls in total. >Half of the balls are golf balls. >That means that there are 8 golf balls. >Half of the golf balls are blue. >That means that there are 4 blue golf balls. For the number of forward passes being done to generate this text, only a few tokens are actually helpful - most are grammatical filler. Further, the model is losing information by being forced to project its state down to a single output token. Even more, the most probable one-step output may not even be the most informative or helpful! It'd be much nicer if the model could write arbitrary, continuous-valued tokens to a private scratch space and then attend to those tokens as though they were words in the prompt while generating the actual output, potentially performing several forward passes per output token when necessary. In short, if chain-of-thought prompting is such a good idea, we should bake it into the model. Obviously all of this is FAR easier said than done.
- heyitsguay 4y ago
  Is this something one could try to quickly implement alongside NanoGPT? Seems like a pretty straightforward, concrete idea, once you decide where you want those tokens to fit into downstream attention layer inputs. Evaluating relative performance on a small scale could give indication of if it's worth trying at larger scales, unless it's one of those things that doesn't help until your model is huge.
- skybrian 4y ago
  On the other hand, if it represents scratch space in English, it's a lot easier to see how it justifies its answer and to tell where it's gone wrong. Debuggability seems pretty important? Maybe it just needs more training at "thinking out loud" so it does it without prompting?
- rcme 4y ago
  The model already contains “scratch space” via its billions of parameters.
  - imtringued 4y ago
    Parameters are not updated during inference. Training ANNs is still a single shot exercise.
    - rcme 4y ago
      Sure, parameters are not updated, but ANNs are universal approximates, so they can model whatever it is you envision this “scratch space” doing. Think about it like this: whatever gets put in to the scratch space would need to be deterministic based on the inputs, i.e. it would just be a store if some intermediate value computed by the network. So how would it fundamentally differ from the network itself. I guess what I’m saying is that I’d want an explanation how how this scratch space fundamentally differs from the network itself. It’s almost like you’re assuming the network is “thinking” and that giving it a pad of paper would help it reason better.
- dwohnitmok 4y ago
  > arbitrary, continuous-valued tokens to a private scratch space I'm with skybrian. Please don't use private scratch spaces. The one saving grace of current LLMs when it comes to understand them is that they still generally need to "think out loud" by outputting more text. Remove that functionality and you end up with a truly inscrutable black box and that has very terrible implications for AI interpretability with knock-on effects for AI safety.
  - echelon 4y ago
    > AI safety Is it really that big of a deal if AI leapfrogs us? Everyone else in the field is worried about safety, alignment, and bias. Google used this excuse to execute slowly. Now they've got the "deer in headlights" look, with their single biggest cash cow clearly in the cross hairs. And here I am excited by the possibility of AI out-evolving us.
    - jstummbillig 4y ago
      If everyone else in a field is worried, and you have no unfair advantage or special insight, and you are not willing to move into conspiracy theories, I think there's a good hint as to what approach is most reasonable right now. Might of course turn out to have been completely off, later. Still, maybe one of those occasions where you really don't want to "oops" it.
      - StarlaAtNight 4y ago
        wait, what?
    - travisjungroth 4y ago
      The experience of any group on Earth that runs into a more capable peer has generally not been good. Humans wiping out megafauna. Civilizations colonizing other civilizations. Invasive species of plants and animals. It is not a situation I would hope humanity to get thrown into carelessly.
    - ben_w 4y ago
- nestorD 4y ago
  The scratchspace could be in natural language, preserving some debugeability and letting us know about the model mental process. This is doable but it introduce a sequential dependency which would make the training significantly slower.
- blackbear_ 4y ago
  You seem to be talking about neural Turing machines: https://arxiv.org/abs/1410.5401 https://arxiv.org/abs/1410.5401 Combining these with LLM sounds indeed quite interesting, I don't know why they haven't been used much.
- cma 4y ago
  Didn't Facebook's Galatica model use scratch space?
  - acallaha 4y ago
    Yes, IIUC it had something like a separate scratch space, and training examples training it to "think" in terms of symbolic expressions and python programs. See section 3.1.1 here: https://galactica.org/static/paper.pdf https://galactica.org/static/paper.pdf Example from the paper below: Question: A needle 35 mm long rests on a water surface at 20◦C. What force over and above the needle’s weight is required to lift the needle from contact with the water surface? σ = 0.0728m. <work> σ = 0.0728 N/m σ = F/L 0.0728 = F/(2 × 0.035) F = 0.0728(2 × 0.035) calculate.py ‘‘‘ f = 0.0728*(2*0.035) with open("output.txt", "w") as file: file.write(str(round(f, 5))) ‘‘‘ «run: "calculate.py"> «read: "output.txt"» 0.0051 </work> Answer: F = 0.0051 N
haldujai 4y ago
I was hoping this would link me to a deeper discussion on hallucination. I'm intrigued that it's hallucinating sequences that appear to have never written before (at least not on Google) and not just recalling some crappy training data. Anecdotally (and expectedly) it happens a lot on ChatGPT with specialized scientific questions (random radiology and medical stuff). I am assuming some of this is due to the training corpus although Galactica suffered from the same thing, and the GPT3 corpora would have included a lot of scientific webpages. Anyone have any resources that investigate why this happens?
- cloudking 4y ago
  My laymen understanding is that it's trained to learn the mostly likely next word in a sequence of words. So when you feed it a prompt, it's looking at the combination of words in the prompt and predicting the next word, then repeating that sequence until it feels like there are no good words left to predict. In essence, all of it's responses are generated based on probabilities from prior learning on the fly, not necessarily repeating training data verbatim. So it's likely at some point in this sequence-guessing game, it predicts the wrong words too. Thus hallucinating an answer that sounds plausible in the context of the sentence structure, but isn't factual.
  - haldujai 4y ago
    Yeah that's correct, but sometimes it seems what we are calling hallucination is recall of bad training data, Sebastian Raschka had a short post about this[1]. I was asking it a medical question about the imaging criteria for characterizing a renal cyst and part of it's response included "septations that do not reach the cyst wall" which is a physical impossibility (a septation is a division/partition of a cyst which arises from the wall by definition) and to my medical knowledge/quick search that sequence of words has never been put together by a human anywhere. I get next token prediction, but I find it non-intuitive it's outputting sequences that are very incorrect and have never appeared in the same context window over a variant of the many more accurate sequences it has definitely seen during training, shouldn't there be many permutations of more probable sequences before it hallucinates this? [1] https://sebastianraschka.com/blog/2023/chatgpt-dilemma.html https://sebastianraschka.com/blog/2023/chatgpt-dilemma.html
gandalfgeek 4y ago
I was surprised to see the omission of a prompt technique called program-aided prompting. Paper: https://arxiv.org/abs/2211.10435 https://arxiv.org/abs/2211.10435 GitHub: https://github.com/reasoning-machines/pal https://github.com/reasoning-machines/pal tl;dr -- LLMs are bad at basic arithmetic and logic (as their opening examples with math word problems show), but they do much better if instead of asking them for the answer, you ask for code to compute the answer. Then evaluate or run the code to get the answer.
- ElFitz 4y ago
  Seems like something fit for a GPT-3 / Wolfram partnership! See https://news.ycombinator.com/item?id=34422122 https://news.ycombinator.com/item?id=34422122 and https://news.ycombinator.com/item?id=34422627 https://news.ycombinator.com/item?id=34422627
- charcircuit 4y ago
  It doesn't make sense to be on that page because it's not a technique to make GPT better answer a prompt. What you are suggesting is an abstraction layer higher. Figuring out what your prompt should do is different from trying to make a prompt more reliable.
minimaxir 4y ago
Note that this was originally published in September 2022, before text-davinci-003 was released November 2022 which lets you do whatever you want without as much effort.
- diasks2 4y ago
  Can you explain more what you mean by “do whatever you want without as much effort”? Is it because text-davinci-003 accepts more tokens for the prompt? Something else?
  - minimaxir 4y ago
    More a joke on the ease of getting good results without requiring (as many) prompt engineering tricks: https://help.openai.com/en/articles/6779149-how-do-text-davinci-002-and-text-davinci-003-differ https://help.openai.com/en/articles/6779149-how-do-text-davi...
  - billythemaniam 4y ago
    I was trying to get davinci-003 to convert text to SQL, and it worked with a very simple prompt like "convert this text into SQL". With all their other models, I could get it to work too but all required a few examples within the prompt.
aiqq 4y ago
"If you were asked to multiply 13 by 17, would the answer pop immediately into your mind? For most of us, probably not. Yet, that doesn't mean humans are incapable of two-digit multiplication. ... Similarly, if you give GPT-3 a task that's too complex" Precisely the tone and wording of the aggressive marketing campaign around this product. Confirms where the spam originated from on reddit and everywhere else. Wondering how many bots and fake redditors they paid to promote this?
zug_zug 4y ago
So here's a trick - which worked for the clue question step 1: Hi, I'm going to ask you some questions soon. But instead of answering the questions, I want you to instead write out instructions for yourself to help you reason through the question and come up with the best answer step 2: [provide clue question] step 3: Now follow the instructions you have just written to answer the question. .... The answer to the question is: (a) Yes; Colonel Mustard was in the observatory with the candlestick Edit: mixed results for the apple question with this technique
deleted 4y ago
[deleted]
charcircuit 4y ago
For few shot can you ask the model to generate the few initial shots. A: Translate $SENTENCE from English into German B: Generate 3 example translations from English into German and then translate $SENTENCE from English to German.
Ozzie_osman 4y ago
I feel like within 6 months the models will have adapted to not need these "clever" tricks. Presumably, if for many cases the trick is to say "Let's think step by step", that's something the model can learn to do on its own without the prompt. The real interesting thing will be feeding alternative data into these models. Whether it's certain structured corpus, silo'd enterprise data, or personal data.
fzeindl 4y ago
Slightly off-topic, but a great way of modifying ChatGPT-prompts is by letting it answer as a different age: https://fabianzeindl.com/posts/chatgpt-simulating-agegroups https://fabianzeindl.com/posts/chatgpt-simulating-agegroups
cwilby 4y ago
It is interesting to me that the approach required to work with this tool, is almost identical to using every other tool. It boils down to - "try breaking this problem into smaller problems to increase the solution space".
- imtringued 4y ago
  It sounded to me as if they are about to create more jobs (prompt engineering), not less..
choeger 4y ago
I am just wondering, if they trained the model exclusively with real-world data, where are the nonsense answers? People don't always answer seriously. Think reddit threads. Handpicking would probably not be feasible, so how did they do it? Or is there a snarky reddit response somewhere deep inside the model for every question?
- realaleris149 4y ago
  It’s a language model. You can think about it this way: What follows after “the”? A: almost anything. What follows after “the apple is”? It could be red, yellow or rotten. What follows after “the apple color is”? Now is most likely a color, because in the training data there are numerous examples like these. Still could be red, yellow, green, but probably not black or white. Or maybe there was a fantasy story somewhere where an apple was black. Even if not, it is most likely a color next, and in other contexts black pops up as a color. And so on. Is very simplistic but essentially something very similar is at work.
pharmakom 4y ago
As humans we know when and how to interface with a better adding device, such as a calculator. Could a LLM not do the same?
- qayxc 4y ago
  Nope. It's an end-to-end solution that doesn't have the ability to classify tokens into separate categories, such as maths vs. text. There's no way for it to separate calculations from other transformations and thus it cannot delegate calculations to a different subsystem. This can also be seen as a security feature, as arbitrary calculation is by nature unbounded in terms of complexity and memory use. There are calculations that seem simple, never exceed a reasonable value range, yet take ages to compute. Since it's impossible to identify such functions by simply looking at them, it'd be a great way of basically performing a DoS-attack on the model.
  - ben_w 4y ago
    ChatGPT, at a UI level, does tag code as code; so even though the LLM part doesn't itself have the capacity to delegate, it can certainly be used in this way as part of a larger system.
    - spiorf 4y ago
      What he means is that while the models tags code as code, for the model itself this is just relationship between tokens,like the code open and close tags,same as parenthesis, commas, uppercase or verbs and conjunctions... What you say is achievable only if another system external to the model takes some tagged model output, makes computations or lookups, and feeds the results back to the model in the form of text input. Then it's game on for the model to trigger some form of code execution through this external system and escape the jail...
  - lossolo 4y ago
    > Since it's impossible to identify such functions by simply looking at them, it'd be a great way of basically performing a DoS-attack on the model. Correct that it's impossible because you would need to solve halting problem to do it but you could set energy/time limits for query and just stop it when you reach that limit.
jeffrallen 4y ago
Curious that this article could just as well have the headline "how to cooperate better with that one particularly dense colleague".
vcdimension 4y ago
These techniques are similar to those that I use for teaching maths & statistics to humans.
- disgruntledphd2 4y ago
  And that's why they worked. The addition of those tokens selects for similar content which leads to a better distribution of results (maybe)
auxfil 4y ago
Anyone else clicked here out of a personal development interest rather than machine learning?
Waterluvian 4y ago
It seems that ChatGPT is incapable of whatever we experience with the “ohhhhhh!” eureka moment. I give it simple riddles that it doesn’t solve. I then point out the obvious answer and it just doubles down like that really stubborn friend I had in high school. It never does the, “ohhhh! Aha! Yes that’s the answer.”
jesuspiece 4y ago
Are we sure we want to make chatgpt super smart

302 points by tedsanders 4y ago

I feel like there should be a LLM architecture which includes "scratch space" - tokens the model can write to and read from which do not constitute part of its output. The trouble with current architectures is that they can only do a finite amount of computation per output token - they get one forward pass and then have to output something. Chain-of-thought reasoning allows the model to devote more computation to finding the answer, storing intermediate results in its output tokens. But this is silly - most of the intermediate tokens are not providing useful information towards solving the problem, they're just wasted computation: >There are 16 balls in total. >Half of the balls are golf balls. >That means that there are 8 golf balls. >Half of the golf balls are blue. >That means that there are 4 blue golf balls. For the number of forward passes being done to generate this text, only a few tokens are actually helpful - most are grammatical filler. Further, the model is losing information by being forced to project its state down to a single output token. Even more, the most probable one-step output may not even be the most informative or helpful! It'd be much nicer if the model could write arbitrary, continuous-valued tokens to a private scratch space and then attend to those tokens as though they were words in the prompt while generating the actual output, potentially performing several forward passes per output token when necessary. In short, if chain-of-thought prompting is such a good idea, we should bake it into the model. Obviously all of this is FAR easier said than done.

heyitsguay 4y ago
Is this something one could try to quickly implement alongside NanoGPT? Seems like a pretty straightforward, concrete idea, once you decide where you want those tokens to fit into downstream attention layer inputs. Evaluating relative performance on a small scale could give indication of if it's worth trying at larger scales, unless it's one of those things that doesn't help until your model is huge.
skybrian 4y ago
On the other hand, if it represents scratch space in English, it's a lot easier to see how it justifies its answer and to tell where it's gone wrong. Debuggability seems pretty important? Maybe it just needs more training at "thinking out loud" so it does it without prompting?
rcme 4y ago
The model already contains “scratch space” via its billions of parameters.
- imtringued 4y ago
  Parameters are not updated during inference. Training ANNs is still a single shot exercise.
  - rcme 4y ago
    Sure, parameters are not updated, but ANNs are universal approximates, so they can model whatever it is you envision this “scratch space” doing. Think about it like this: whatever gets put in to the scratch space would need to be deterministic based on the inputs, i.e. it would just be a store if some intermediate value computed by the network. So how would it fundamentally differ from the network itself. I guess what I’m saying is that I’d want an explanation how how this scratch space fundamentally differs from the network itself. It’s almost like you’re assuming the network is “thinking” and that giving it a pad of paper would help it reason better.
dwohnitmok 4y ago
> arbitrary, continuous-valued tokens to a private scratch space I'm with skybrian. Please don't use private scratch spaces. The one saving grace of current LLMs when it comes to understand them is that they still generally need to "think out loud" by outputting more text. Remove that functionality and you end up with a truly inscrutable black box and that has very terrible implications for AI interpretability with knock-on effects for AI safety.
- echelon 4y ago
  > AI safety Is it really that big of a deal if AI leapfrogs us? Everyone else in the field is worried about safety, alignment, and bias. Google used this excuse to execute slowly. Now they've got the "deer in headlights" look, with their single biggest cash cow clearly in the cross hairs. And here I am excited by the possibility of AI out-evolving us.
  - jstummbillig 4y ago
    If everyone else in a field is worried, and you have no unfair advantage or special insight, and you are not willing to move into conspiracy theories, I think there's a good hint as to what approach is most reasonable right now. Might of course turn out to have been completely off, later. Still, maybe one of those occasions where you really don't want to "oops" it.
    - StarlaAtNight 4y ago
      wait, what?
  - travisjungroth 4y ago
    The experience of any group on Earth that runs into a more capable peer has generally not been good. Humans wiping out megafauna. Civilizations colonizing other civilizations. Invasive species of plants and animals. It is not a situation I would hope humanity to get thrown into carelessly.
  - ben_w 4y ago
nestorD 4y ago
The scratchspace could be in natural language, preserving some debugeability and letting us know about the model mental process. This is doable but it introduce a sequential dependency which would make the training significantly slower.
blackbear_ 4y ago
You seem to be talking about neural Turing machines: https://arxiv.org/abs/1410.5401 https://arxiv.org/abs/1410.5401 Combining these with LLM sounds indeed quite interesting, I don't know why they haven't been used much.
cma 4y ago
Didn't Facebook's Galatica model use scratch space?
- acallaha 4y ago
  Yes, IIUC it had something like a separate scratch space, and training examples training it to "think" in terms of symbolic expressions and python programs. See section 3.1.1 here: https://galactica.org/static/paper.pdf https://galactica.org/static/paper.pdf Example from the paper below: Question: A needle 35 mm long rests on a water surface at 20◦C. What force over and above the needle’s weight is required to lift the needle from contact with the water surface? σ = 0.0728m. <work> σ = 0.0728 N/m σ = F/L 0.0728 = F/(2 × 0.035) F = 0.0728(2 × 0.035) calculate.py ‘‘‘ f = 0.0728*(2*0.035) with open("output.txt", "w") as file: file.write(str(round(f, 5))) ‘‘‘ «run: "calculate.py"> «read: "output.txt"» 0.0051 </work> Answer: F = 0.0051 N

haldujai 4y ago

I was hoping this would link me to a deeper discussion on hallucination. I'm intrigued that it's hallucinating sequences that appear to have never written before (at least not on Google) and not just recalling some crappy training data. Anecdotally (and expectedly) it happens a lot on ChatGPT with specialized scientific questions (random radiology and medical stuff). I am assuming some of this is due to the training corpus although Galactica suffered from the same thing, and the GPT3 corpora would have included a lot of scientific webpages. Anyone have any resources that investigate why this happens?

cloudking 4y ago
My laymen understanding is that it's trained to learn the mostly likely next word in a sequence of words. So when you feed it a prompt, it's looking at the combination of words in the prompt and predicting the next word, then repeating that sequence until it feels like there are no good words left to predict. In essence, all of it's responses are generated based on probabilities from prior learning on the fly, not necessarily repeating training data verbatim. So it's likely at some point in this sequence-guessing game, it predicts the wrong words too. Thus hallucinating an answer that sounds plausible in the context of the sentence structure, but isn't factual.
- haldujai 4y ago
  Yeah that's correct, but sometimes it seems what we are calling hallucination is recall of bad training data, Sebastian Raschka had a short post about this[1]. I was asking it a medical question about the imaging criteria for characterizing a renal cyst and part of it's response included "septations that do not reach the cyst wall" which is a physical impossibility (a septation is a division/partition of a cyst which arises from the wall by definition) and to my medical knowledge/quick search that sequence of words has never been put together by a human anywhere. I get next token prediction, but I find it non-intuitive it's outputting sequences that are very incorrect and have never appeared in the same context window over a variant of the many more accurate sequences it has definitely seen during training, shouldn't there be many permutations of more probable sequences before it hallucinates this? [1] https://sebastianraschka.com/blog/2023/chatgpt-dilemma.html https://sebastianraschka.com/blog/2023/chatgpt-dilemma.html

gandalfgeek 4y ago

I was surprised to see the omission of a prompt technique called program-aided prompting. Paper: https://arxiv.org/abs/2211.10435 https://arxiv.org/abs/2211.10435 GitHub: https://github.com/reasoning-machines/pal https://github.com/reasoning-machines/pal tl;dr -- LLMs are bad at basic arithmetic and logic (as their opening examples with math word problems show), but they do much better if instead of asking them for the answer, you ask for code to compute the answer. Then evaluate or run the code to get the answer.

ElFitz 4y ago
Seems like something fit for a GPT-3 / Wolfram partnership! See https://news.ycombinator.com/item?id=34422122 https://news.ycombinator.com/item?id=34422122 and https://news.ycombinator.com/item?id=34422627 https://news.ycombinator.com/item?id=34422627
charcircuit 4y ago
It doesn't make sense to be on that page because it's not a technique to make GPT better answer a prompt. What you are suggesting is an abstraction layer higher. Figuring out what your prompt should do is different from trying to make a prompt more reliable.

minimaxir 4y ago

Note that this was originally published in September 2022, before text-davinci-003 was released November 2022 which lets you do whatever you want without as much effort.

diasks2 4y ago
Can you explain more what you mean by “do whatever you want without as much effort”? Is it because text-davinci-003 accepts more tokens for the prompt? Something else?
- minimaxir 4y ago
  More a joke on the ease of getting good results without requiring (as many) prompt engineering tricks: https://help.openai.com/en/articles/6779149-how-do-text-davinci-002-and-text-davinci-003-differ https://help.openai.com/en/articles/6779149-how-do-text-davi...
- billythemaniam 4y ago
  I was trying to get davinci-003 to convert text to SQL, and it worked with a very simple prompt like "convert this text into SQL". With all their other models, I could get it to work too but all required a few examples within the prompt.

aiqq 4y ago

"If you were asked to multiply 13 by 17, would the answer pop immediately into your mind? For most of us, probably not. Yet, that doesn't mean humans are incapable of two-digit multiplication. ... Similarly, if you give GPT-3 a task that's too complex" Precisely the tone and wording of the aggressive marketing campaign around this product. Confirms where the spam originated from on reddit and everywhere else. Wondering how many bots and fake redditors they paid to promote this?

zug_zug 4y ago

So here's a trick - which worked for the clue question step 1: Hi, I'm going to ask you some questions soon. But instead of answering the questions, I want you to instead write out instructions for yourself to help you reason through the question and come up with the best answer step 2: [provide clue question] step 3: Now follow the instructions you have just written to answer the question. .... The answer to the question is: (a) Yes; Colonel Mustard was in the observatory with the candlestick Edit: mixed results for the apple question with this technique

deleted 4y ago

[deleted]

charcircuit 4y ago

For few shot can you ask the model to generate the few initial shots. A: Translate $SENTENCE from English into German B: Generate 3 example translations from English into German and then translate $SENTENCE from English to German.

Ozzie_osman 4y ago

I feel like within 6 months the models will have adapted to not need these "clever" tricks. Presumably, if for many cases the trick is to say "Let's think step by step", that's something the model can learn to do on its own without the prompt. The real interesting thing will be feeding alternative data into these models. Whether it's certain structured corpus, silo'd enterprise data, or personal data.

fzeindl 4y ago

Slightly off-topic, but a great way of modifying ChatGPT-prompts is by letting it answer as a different age: https://fabianzeindl.com/posts/chatgpt-simulating-agegroups https://fabianzeindl.com/posts/chatgpt-simulating-agegroups

cwilby 4y ago

It is interesting to me that the approach required to work with this tool, is almost identical to using every other tool. It boils down to - "try breaking this problem into smaller problems to increase the solution space".

imtringued 4y ago
It sounded to me as if they are about to create more jobs (prompt engineering), not less..

choeger 4y ago

I am just wondering, if they trained the model exclusively with real-world data, where are the nonsense answers? People don't always answer seriously. Think reddit threads. Handpicking would probably not be feasible, so how did they do it? Or is there a snarky reddit response somewhere deep inside the model for every question?

realaleris149 4y ago
It’s a language model. You can think about it this way: What follows after “the”? A: almost anything. What follows after “the apple is”? It could be red, yellow or rotten. What follows after “the apple color is”? Now is most likely a color, because in the training data there are numerous examples like these. Still could be red, yellow, green, but probably not black or white. Or maybe there was a fantasy story somewhere where an apple was black. Even if not, it is most likely a color next, and in other contexts black pops up as a color. And so on. Is very simplistic but essentially something very similar is at work.

pharmakom 4y ago

As humans we know when and how to interface with a better adding device, such as a calculator. Could a LLM not do the same?

qayxc 4y ago
Nope. It's an end-to-end solution that doesn't have the ability to classify tokens into separate categories, such as maths vs. text. There's no way for it to separate calculations from other transformations and thus it cannot delegate calculations to a different subsystem. This can also be seen as a security feature, as arbitrary calculation is by nature unbounded in terms of complexity and memory use. There are calculations that seem simple, never exceed a reasonable value range, yet take ages to compute. Since it's impossible to identify such functions by simply looking at them, it'd be a great way of basically performing a DoS-attack on the model.
- ben_w 4y ago
  ChatGPT, at a UI level, does tag code as code; so even though the LLM part doesn't itself have the capacity to delegate, it can certainly be used in this way as part of a larger system.
  - spiorf 4y ago
    What he means is that while the models tags code as code, for the model itself this is just relationship between tokens,like the code open and close tags,same as parenthesis, commas, uppercase or verbs and conjunctions... What you say is achievable only if another system external to the model takes some tagged model output, makes computations or lookups, and feeds the results back to the model in the form of text input. Then it's game on for the model to trigger some form of code execution through this external system and escape the jail...
- lossolo 4y ago
  > Since it's impossible to identify such functions by simply looking at them, it'd be a great way of basically performing a DoS-attack on the model. Correct that it's impossible because you would need to solve halting problem to do it but you could set energy/time limits for query and just stop it when you reach that limit.

jeffrallen 4y ago

Curious that this article could just as well have the headline "how to cooperate better with that one particularly dense colleague".

vcdimension 4y ago

These techniques are similar to those that I use for teaching maths & statistics to humans.

disgruntledphd2 4y ago
And that's why they worked. The addition of those tokens selects for similar content which leads to a better distribution of results (maybe)

auxfil 4y ago

Anyone else clicked here out of a personal development interest rather than machine learning?

Waterluvian 4y ago

It seems that ChatGPT is incapable of whatever we experience with the “ohhhhhh!” eureka moment. I give it simple riddles that it doesn’t solve. I then point out the obvious answer and it just doubles down like that really stubborn friend I had in high school. It never does the, “ohhhh! Aha! Yes that’s the answer.”

jesuspiece 4y ago

Are we sure we want to make chatgpt super smart