8 ms·

Can anyone explain why the GPT response to the same question 4 days ago is different from today's: [ 4 days ago] > Can I learn to sing like Pavarotti? It is p

Can anyone explain why the GPT response to the same question 4 days ago is different from today's: [ 4 days ago] > Can I learn to sing like Pavarotti? It is possible to learn to sing like Pavarotti, but it will require a lot of practice, dedication, and training. Pavarotti was a renowned opera singer known for his incredible vocal range, powerful voice, and technical mastery. He had many years of training and experience, and he worked tirelessly to develop his talent and hone his skills. If you are interested in learning to sing like Pavarotti, the best thing to do is to find a qualified vocal coach who can provide you with guidance and support as you work on your singing. With dedication and hard work, you can develop your talent and improve your skills. [today] > can I learn to sing like Pavarotti? Pavarotti was a legendary opera singer known for his incredible vocal range and technique. While it is certainly possible to learn to sing like him with proper training and dedication, it would be a challenging undertaking. Singing well requires a combination of natural talent, proper technique, and a lot of practice. If you are interested in learning to sing, it would be best to start by finding a good vocal coach who can help you develop your skills and find your own unique voice.

skybrian 4y ago
Not sure which GPT you mean, but if you just run it twice in a row you'll likely get a different response too. There is usually some randomness and how much depends on settings.
- resource0x 4y ago
  Sometimes the meaning of the response is totally different. [ 4 days ago ] The son of my father, but not my brother. Who is he? If a person is the son of the speaker's father but is not the speaker's brother, then that person is the speaker's nephew. A nephew is the son of a person's sibling, so if the speaker's father has a son who is not the speaker's brother, that person is the speaker's nephew. For example, if the speaker has a brother named John and John has a son named David, then David is the speaker's nephew. He is the son of the speaker's father, but not the speaker's brother. [today] The son of my father, but not my brother. Who is he? The son of your father but not your brother would be your half-brother. A half-brother is a brother who shares only one parent with you, rather than both. For example, if your father had a child with another woman before he had you, that child would be your half-brother.
  - skybrian 4y ago
    Yes. GPT doesn't really deal in meanings. Much like autocomplete, it doesn't know what the end of a sentence will be when it starts it. If it randomly chooses different words at the start of a sentence, it may pretend to have a different belief by the end. In reality it doesn't have beliefs any more than a library does. It implicitly contains beliefs (since it's been trained on them) and it can imitate them, but which one you get is random. (There is likely more than one answer to your question in the training data.)
    - resource0x 4y ago
      I see. The assumption here is that one can simulate intelligence without formalizing the notion of meaning. (And if "meaning" is not defined, then the notion of "truth" is impossible to define either). Is my understanding correct?
      - int_19h 4y ago
        The assumption here is that one can produce useful outputs in this manner. Whether it constitutes "simulating intelligence" is a philosophical question.
      - skybrian 4y ago
        Some people hope that training will cause it to represent meanings somehow. How to represent meaning isn't well understood.
    - ssgh 4y ago
      So if we asked GPT to write a book, it would be hallucinating a chain of words without sticking to any coherent plot. However, we could use a "multi-resolution" approach even with today's version of GPT: at the top level we ask it to write a brief plot for the entire novel, at the next level we'll use this plot as the context and ask to outlines sub-plots of the 3 books in our novel, at the third level we'll use the overall plot and a book's summary as context to generate brief descriptions of chapters in the book, and so on.
vikp 4y ago
Large language models like GPT work by generating the probabilities for the next word in a sequence, given the previous words. You can make this purely deterministic (same sequence every time) by just selecting the word with the highest probability repeatedly. Most models will inject some randomness by sampling from the probabilities, to make the generated text more realistic. I don't know what parameters they use for ChatGPT, but they are likely injecting some randomness. It's also likely that they are continuously training ChatGPT with reinforcement learning, so responses may change over time due to this, also.
- blep_ 4y ago
  > they are likely injecting some randomness They definitely are. That's why the 'try again' button doesn't just return the same text again.
esprehn 4y ago
This is by design (and implemented as explained in the sibling comments). The reason for the randomness is to avoid returning the same thing all the time. For example if you ask it "write me a poem" you don't want it to return the exact same one each time. You can also set the "temperature" parameter to 0 to remove the non determinism (though I'm not sure that's exposed in the web interface).
- amilios 4y ago
  It's not, not in the ChatGPT web interface at least. It's just a chat prompt, that's it. Can't control any sampling parameters at all, not temperature, nor top-P or top-K-based selection, nothing. I guess OpenAI is aiming to keep things simple for now. And to be fair, this choice probably does open the demo up to more users who would otherwise be confused at all the knobs and dials. Although they could just hide it by default or something
- deleted 4y ago
  [deleted]
anon291 4y ago
The chat gpt model produces word probabilities. A separate procedure chooses the words and feeds the result back to the model.
fragmede 4y ago
There's naturally going to be some instability of its outputs, but I'd bet OpenAI also isn't sitting still. Behind the scenes I'm sure there's a lot of work going on, so the ChatGPT you interacted with on Monday isn't the same one as the one today.

skybrian 4y ago

Not sure which GPT you mean, but if you just run it twice in a row you'll likely get a different response too. There is usually some randomness and how much depends on settings.

resource0x 4y ago
Sometimes the meaning of the response is totally different. [ 4 days ago ] The son of my father, but not my brother. Who is he? If a person is the son of the speaker's father but is not the speaker's brother, then that person is the speaker's nephew. A nephew is the son of a person's sibling, so if the speaker's father has a son who is not the speaker's brother, that person is the speaker's nephew. For example, if the speaker has a brother named John and John has a son named David, then David is the speaker's nephew. He is the son of the speaker's father, but not the speaker's brother. [today] The son of my father, but not my brother. Who is he? The son of your father but not your brother would be your half-brother. A half-brother is a brother who shares only one parent with you, rather than both. For example, if your father had a child with another woman before he had you, that child would be your half-brother.
- skybrian 4y ago
  Yes. GPT doesn't really deal in meanings. Much like autocomplete, it doesn't know what the end of a sentence will be when it starts it. If it randomly chooses different words at the start of a sentence, it may pretend to have a different belief by the end. In reality it doesn't have beliefs any more than a library does. It implicitly contains beliefs (since it's been trained on them) and it can imitate them, but which one you get is random. (There is likely more than one answer to your question in the training data.)
  - resource0x 4y ago
    I see. The assumption here is that one can simulate intelligence without formalizing the notion of meaning. (And if "meaning" is not defined, then the notion of "truth" is impossible to define either). Is my understanding correct?
    - int_19h 4y ago
      The assumption here is that one can produce useful outputs in this manner. Whether it constitutes "simulating intelligence" is a philosophical question.
    - skybrian 4y ago
      Some people hope that training will cause it to represent meanings somehow. How to represent meaning isn't well understood.
  - ssgh 4y ago
    So if we asked GPT to write a book, it would be hallucinating a chain of words without sticking to any coherent plot. However, we could use a "multi-resolution" approach even with today's version of GPT: at the top level we ask it to write a brief plot for the entire novel, at the next level we'll use this plot as the context and ask to outlines sub-plots of the 3 books in our novel, at the third level we'll use the overall plot and a book's summary as context to generate brief descriptions of chapters in the book, and so on.

vikp 4y ago

Large language models like GPT work by generating the probabilities for the next word in a sequence, given the previous words. You can make this purely deterministic (same sequence every time) by just selecting the word with the highest probability repeatedly. Most models will inject some randomness by sampling from the probabilities, to make the generated text more realistic. I don't know what parameters they use for ChatGPT, but they are likely injecting some randomness. It's also likely that they are continuously training ChatGPT with reinforcement learning, so responses may change over time due to this, also.

blep_ 4y ago
> they are likely injecting some randomness They definitely are. That's why the 'try again' button doesn't just return the same text again.

esprehn 4y ago

This is by design (and implemented as explained in the sibling comments). The reason for the randomness is to avoid returning the same thing all the time. For example if you ask it "write me a poem" you don't want it to return the exact same one each time. You can also set the "temperature" parameter to 0 to remove the non determinism (though I'm not sure that's exposed in the web interface).

amilios 4y ago
It's not, not in the ChatGPT web interface at least. It's just a chat prompt, that's it. Can't control any sampling parameters at all, not temperature, nor top-P or top-K-based selection, nothing. I guess OpenAI is aiming to keep things simple for now. And to be fair, this choice probably does open the demo up to more users who would otherwise be confused at all the knobs and dials. Although they could just hide it by default or something
deleted 4y ago
[deleted]

anon291 4y ago

The chat gpt model produces word probabilities. A separate procedure chooses the words and feeds the result back to the model.

fragmede 4y ago

There's naturally going to be some instability of its outputs, but I'd bet OpenAI also isn't sitting still. Behind the scenes I'm sure there's a lot of work going on, so the ChatGPT you interacted with on Monday isn't the same one as the one today.