3 ms·

> GPT-3 is perfectly capable of recognizing what kinds of things it will be bad at, and can be encouraged to generate machine-executable queries to fill in that

by pdonis 4y ago

> GPT-3 is perfectly capable of recognizing what kinds of things it will be bad at, and can be encouraged to generate machine-executable queries to fill in that gap. By what seems to me to be the obvious choice for a definition of "bad at", namely "not answering queries based on an actual semantically connected world model", GPT-3 is bad at everything. And an obvious example of an endpoint of your perfectly reasonable suggestion to have it pass on queries to solution machines that are based on actual world models, is...Wolfram Alpha.

shagie 4y ago
The thing is that it is good at classifying things. You could then use it to classify things that it is bad at and use that information (as part of a larger whole) to dispatch a query to the knowledge system that can return back the proper (current) information, and report on that. The following is a list of questions. Identify the category they belong to as one of {Current Events}, {General knowlege}, {Unit conversion}, {Math}: 1. How many feet in a mile? 2. What is the square root of 541696? 3. Who is the Speaker of the House? 4. How many turkeys in Turkey? to which it responds: 1. Unit conversion 2. Math 3. Current Events 4. General Knowledge The supervisor system (for lack of a better word) would then dispatch the questions to different systems that it is coded to be able to either further classify the question or provide the proper answer.
- pdonis 4y ago
  > The thing is that it is good at classifying things. If by "classifying things" you mean doing what Wolfram Alpha's natural language front end does--figuring out which solution engine to pass the question to, and translating the question into what that solution engine needs--then yes. That's what your examples show. And that concedes my point: that something like GPT-3 might make a good natural language front end for something like Wolfram Alpha's solution engines; but it's not good at actually giving correct answers on its own. It's bad at that, for every type of question.
  - shagie 4y ago
    Yep - the things I've found it good at is stuff like "here is data, answer questions about it" (I've got dabblingings in writing a private Alexa skill that does a better weather report than the rules based one that is currently used) or zero shot classifications. For example with the zero shot: https://news.ycombinator.com/item?id=34156626 https://news.ycombinator.com/item?id=34156626 And this is a "not everything goes to Wolfram" - the weather report that it has ( https://www.wolframalpha.com/input?i=chicago+weather https://www.wolframalpha.com/input?i=chicago+weather ) doesn't provide quite the right data (though it is rather neat). The other thing I need to look at is the question are some of the other, simpler models acceptable for these classifications. The Curie model is 10x cheaper than the Davinci model. And Babbage and Ada are even cheaper. The system is interesting... but even with some dabbling one can see that this isn't something to trust for knowledge, but rather work with for translating one "language" into another and then using that in conjunction with another layer that can use that translated "language".

by pdonis 4y ago

shagie 4y ago

The thing is that it is good at classifying things. You could then use it to classify things that it is bad at and use that information (as part of a larger whole) to dispatch a query to the knowledge system that can return back the proper (current) information, and report on that. The following is a list of questions. Identify the category they belong to as one of {Current Events}, {General knowlege}, {Unit conversion}, {Math}: 1. How many feet in a mile? 2. What is the square root of 541696? 3. Who is the Speaker of the House? 4. How many turkeys in Turkey? to which it responds: 1. Unit conversion 2. Math 3. Current Events 4. General Knowledge The supervisor system (for lack of a better word) would then dispatch the questions to different systems that it is coded to be able to either further classify the question or provide the proper answer.

pdonis 4y ago
> The thing is that it is good at classifying things. If by "classifying things" you mean doing what Wolfram Alpha's natural language front end does--figuring out which solution engine to pass the question to, and translating the question into what that solution engine needs--then yes. That's what your examples show. And that concedes my point: that something like GPT-3 might make a good natural language front end for something like Wolfram Alpha's solution engines; but it's not good at actually giving correct answers on its own. It's bad at that, for every type of question.
- shagie 4y ago
  Yep - the things I've found it good at is stuff like "here is data, answer questions about it" (I've got dabblingings in writing a private Alexa skill that does a better weather report than the rules based one that is currently used) or zero shot classifications. For example with the zero shot: https://news.ycombinator.com/item?id=34156626 https://news.ycombinator.com/item?id=34156626 And this is a "not everything goes to Wolfram" - the weather report that it has ( https://www.wolframalpha.com/input?i=chicago+weather https://www.wolframalpha.com/input?i=chicago+weather ) doesn't provide quite the right data (though it is rather neat). The other thing I need to look at is the question are some of the other, simpler models acceptable for these classifications. The Curie model is 10x cheaper than the Davinci model. And Babbage and Ada are even cheaper. The system is interesting... but even with some dabbling one can see that this isn't something to trust for knowledge, but rather work with for translating one "language" into another and then using that in conjunction with another layer that can use that translated "language".