3 ms·
GPT-3 needs proper prompting to get the right output. Here's the same question as in the blogpost posed differently Student: Why is it unsafe to walk backwards
by thewarrior 5y ago
GPT-3 needs proper prompting to get the right output. Here's the same question as in the blogpost posed differently
Student: Why is it unsafe to walk backwards down a staircase with my eyes closed ?
Teacher: You can't see the step behind you!
The "teacher" here is GPT-3 and answered the question easily. People are in denial of genuine progress.
Student: Do you think I could predict changes in stock prices better if I used 10-year Treasury rates or the temperature in Curtin, Australia?
Stock trader: I don't know. You might be able to, but I can't. The problem is that the temperature changes are so small that they won't be statistically significant. It's a question of sample size. An individual day's temperature in one location is far less likely to be an indicator of stock prices than the interest rate.
Student: Why is that ?
Stock trader: That's because interest rates are much more likely to affect the economy than a single day's temperature.
Student: Why do interest rates affect the economy ?
Stock trader: Interest rates are a measure of the cost of money, so the interest rate is a measure of the cost of capital. If the cost of capital is high, firms will borrow less money and they'll invest less. That means they won't be as profitable, and stock prices will fall.
Quiz question: Who is the President of the United States?
Answer: Donald Trump.
I did get junk answers as well (it said that Jimi Hendrix is the president) but I have a better intuition of how to put GPT-3 in the right "frame of mind". GPT-3 is a text completion engine. It doesn't know whether you want it to produce a story, a reddit flamewar or a PhD thesis. You need to provide the hints. One prompt cannot answer any and all questions. If it were that good we would be jobless. It's far from perfect but it's beginning to get there.
- barthelomew 5y agoOnce you start prompt-engineering, the answers, in seclusion look pretty great - and show some understanding of the domain. Yet, if you donot do this - they end up spitting out some randomly associated phrases/answers. This is a problem when you're asking a model, a question who's answer you donot completely know. How do you trust it to give the right answer? If you donot know the answer beforehand - you cannot prompt-engineer the model to give the "right answers". "Expert systems" from the 80s and the 90s, were pretty good at giving the "right answers" inside a closed domain. Those were truly wonderful systems.
- GrantS 5y agoYes, this is both the most misunderstood and the most amazing (and unpredicted?) thing about GPT-3 et al. You have to tell it who it is and what the rules are. And it gains or lacks knowledge depending upon who it thinks it is answering as. You’ll find that Stephen Hawking doesn’t have much to say about the Ninja Turtles while celebrities know nothing about black holes. It does not answer questions or have expertise unless you tell it that it is answering questions and has expertise. (And it is incredible that this is the case, and somewhat understandable that people don’t understand this.)
- ackbar03 5y agoThat can't possibly be right though. I'm sure Stephen hawking knows a thing or two about the ninja turtles
- throwawayboise 5y agoHe does. It's turtles all the way down.
- deleted 5y ago[deleted]
- earthboundkid 5y agoThose answers suck though. I would not consider a human who said them to be intelligent. For the first one, it’s not that you can’t see, it’s specifically that you will fall and crack your head. An answer that doesn’t mention falling is a bad answer. Number two, it goes off on temperature and sample sizes which is not germane to the question. The answer should tell a story about causality and instead it goes off on ephemera. Three is a Wikipedia search, ie Siri can answer this today, and it’s wrong.
- somebodythere 5y agoAll that's missing is a reranking step that rewards incisive and non-obvious answers. Actually, some models are already implementing such a thing: LAmDA will prioritize contentful answers above statistical probability, the same way humans do. GPT was only trained on "text" with no specific distribution, and specifically to predict a masked word from its surrounding context. As a result, the only questions you can use GPT to answer are of the form "sample from the most likely continuations of this text". It turns out a lot of problems can actually be posed in this form, if you understand the input distribution well. But the future probably looks less like models that operate directly on a distribution of language itself, and more like models that use their language knowledge to predict the volition of the person who gave it input, relate the input to their knowledge of the world (learned from corpus), and then use their language knowledge to convert an internal abstract representation of logical reasoning to something the user can read and understand. I don't think the tech is extremely far off, it's probably a natural continuation of the current research.
- throwawayboise 5y agoFor some reason I am having a very hard time resisting the urge to try walking down a flight of stairs backwards with my eyes closed. How hard can it be?
- jacquesm 5y agoI do that every time I go down a ladder.