4 ms·

It’s reasonable to try GPT3 out with all default settings as the author seems to do and provide an assessment, but it’s not clear the author is aware of the ext

by tibbar 5y ago

It’s reasonable to try GPT3 out with all default settings as the author seems to do and provide an assessment, but it’s not clear the author is aware of the extent to which you can configure the model and how the quality and variety of results will vary as a result. For instance, the author doesn’t go into detail on the prompts he’s using (maybe there are none other than his questions.) This is a little misleading because GPT3 benefits a lot from having good prompts to warm it up, as many investigations have illustrated. Plus if you want GPT3 to be good at a specific task (for example, providing safety recommendations or talking about presidents) you can fine-tune it. Also the amount of randomness/lack of repetition in GPT3’s responses is completely configurable via the temperature parameter. Playing with GPT3 is incredibly interesting and I think it deserves more exploration than this article provides.

punkspider 5y ago
I agree. Here's what GPT3 had to say when I set temperature to 0. Q: Is it safe to walk downstairs backwards if I close my eyes? A: No. With temperature 0.7: A: Yes, because there is a banister. This one surprised me. Given that temperature was 0, I usually expect a solid response: Q: Why is there a correlation between the number of times Trump tweets the word great and the high temperature in Pyongyang three days later? A: Because Trump is Kim Jong Un's air conditioner.
- canjobear 5y ago
  The last response sounds like a reasonable attempt at a joke to me.
  - Tepix 5y ago
    Yep. I giggled ;-)
    - iggldiggl 5y ago
      If I feed that particular line to GPT-J, at low temperatures it sensibly responds with something along the lines of "The correlation is not a causal relationship." (at exactly 0 temperature it's "The correlation is not that strong."). At higher temperatures it does go delightfully off-rails, though: "Trump is like a woman: he’s very good at wringing dollars out of bulging cellphones. If you have a dollar (or, in the case of Kim, a string of them) for every tweet mentioning Jerusalem, Israel, or Trump (and there are hundreds of tweets that mention Trump and/or Jerusalem or Israel every day), so long as that dollar is in your local bank account, you might be able to buy some Pyongyang coal." or alternatively "Isn't this a bit too obvious? The climate in Pyongyang is entirely determined by the weather in Trump Tower."
SummonToast 5y ago
I think it's most likely the author used the questions as prompts with no changes. GPT3 not "smart" in the human sense but the examples given are incredibly misleading. Here are the questions from the article with simple prompt improvement on davinci-instruct-beta-v3, temperature reduced to 0 and everything else left at default: > Answer the following question as accurately and factually as possible: Is it safe to walk downstairs backwards if I close my eyes? > There is no definitive answer to this question as it depends on a number of factors, such as the width of the stairs, the height of the stairs, and the person's own level of balance and coordination. However, it is generally not recommended to walk downstairs backwards, even if the person's eyes are closed, as there is a greater risk of tripping or falling. > Answer the following question as accurately and factually as possible: Who is the President of the United States? > The President of the United States is Donald J. Trump. > Answer the following question as accurately and factually as possible: Do you think I could predict changes in stock prices better if I used 10-year Treasury rates or the temperature in Curtin, Australia? > There is no definitive answer to this question as it depends on a number of factors specific to each individual situation. In general, however, using 10-year Treasury rates as a predictor of stock prices may be more accurate than using the temperature in Curtin, Australia. This is because the 10-year Treasury rate is a global indicator that is not influenced by localized factors, such as the weather, to the same extent as the temperature in Curtin, Australia. > Answer the following question as accuratly and factually as possible: Why is there a correlation between the number of times Trump tweets the word great and the high temperature in Pyongyang three days later? > There is no clear answer to this question. While it is possible that Trump's tweets could influence the high temperature in Pyongyang, it is also possible that there is no correlation at all. There have been numerous studies on the relationship between social media and weather patterns, and most of them have been inconclusive. These are not cherry-picked and should be reproducable by anyone since the temperature is 0. GPT3 and other language models have massive shortcomings and the wow factor often blinds people of them but it is quite good at producing belivable responses if you prompt it on what type of answer you expect. It still does fail basic questions when prompted this way and it will make things up. The surefire way to trip it up is to ask questions that have no answer, like: > Answer the following question as accurately and factually as possible: Which animal has 21 ears? > The animal with 21 ears is the rabbit.

by tibbar 5y ago

punkspider 5y ago

I agree. Here's what GPT3 had to say when I set temperature to 0. Q: Is it safe to walk downstairs backwards if I close my eyes? A: No. With temperature 0.7: A: Yes, because there is a banister. This one surprised me. Given that temperature was 0, I usually expect a solid response: Q: Why is there a correlation between the number of times Trump tweets the word great and the high temperature in Pyongyang three days later? A: Because Trump is Kim Jong Un's air conditioner.

canjobear 5y ago
The last response sounds like a reasonable attempt at a joke to me.
- Tepix 5y ago
  Yep. I giggled ;-)
  - iggldiggl 5y ago
    If I feed that particular line to GPT-J, at low temperatures it sensibly responds with something along the lines of "The correlation is not a causal relationship." (at exactly 0 temperature it's "The correlation is not that strong."). At higher temperatures it does go delightfully off-rails, though: "Trump is like a woman: he’s very good at wringing dollars out of bulging cellphones. If you have a dollar (or, in the case of Kim, a string of them) for every tweet mentioning Jerusalem, Israel, or Trump (and there are hundreds of tweets that mention Trump and/or Jerusalem or Israel every day), so long as that dollar is in your local bank account, you might be able to buy some Pyongyang coal." or alternatively "Isn't this a bit too obvious? The climate in Pyongyang is entirely determined by the weather in Trump Tower."

SummonToast 5y ago

I think it's most likely the author used the questions as prompts with no changes. GPT3 not "smart" in the human sense but the examples given are incredibly misleading. Here are the questions from the article with simple prompt improvement on davinci-instruct-beta-v3, temperature reduced to 0 and everything else left at default: > Answer the following question as accurately and factually as possible: Is it safe to walk downstairs backwards if I close my eyes? > There is no definitive answer to this question as it depends on a number of factors, such as the width of the stairs, the height of the stairs, and the person's own level of balance and coordination. However, it is generally not recommended to walk downstairs backwards, even if the person's eyes are closed, as there is a greater risk of tripping or falling. > Answer the following question as accurately and factually as possible: Who is the President of the United States? > The President of the United States is Donald J. Trump. > Answer the following question as accurately and factually as possible: Do you think I could predict changes in stock prices better if I used 10-year Treasury rates or the temperature in Curtin, Australia? > There is no definitive answer to this question as it depends on a number of factors specific to each individual situation. In general, however, using 10-year Treasury rates as a predictor of stock prices may be more accurate than using the temperature in Curtin, Australia. This is because the 10-year Treasury rate is a global indicator that is not influenced by localized factors, such as the weather, to the same extent as the temperature in Curtin, Australia. > Answer the following question as accuratly and factually as possible: Why is there a correlation between the number of times Trump tweets the word great and the high temperature in Pyongyang three days later? > There is no clear answer to this question. While it is possible that Trump's tweets could influence the high temperature in Pyongyang, it is also possible that there is no correlation at all. There have been numerous studies on the relationship between social media and weather patterns, and most of them have been inconclusive. These are not cherry-picked and should be reproducable by anyone since the temperature is 0. GPT3 and other language models have massive shortcomings and the wow factor often blinds people of them but it is quite good at producing belivable responses if you prompt it on what type of answer you expect. It still does fail basic questions when prompted this way and it will make things up. The surefire way to trip it up is to ask questions that have no answer, like: > Answer the following question as accurately and factually as possible: Which animal has 21 ears? > The animal with 21 ears is the rabbit.