3 ms·

>This is a great review but it still misses what seems like the point to me: these models don't do any actual reasoning. Why don't you come up with your own te

>This is a great review but it still misses what seems like the point to me: these models don't do any actual reasoning. Why don't you come up with your own test for this, something unique that wouldn't be in the text corpus of the Internet so far, and then try it on OpenGPT and see if it passes your own test. Try it and see if it is able to reason. I couldn't come up with my own so I decided to ask ChatGPT to come up with one. This is my prompt: "Today in order to test whether an AI is able to reason, or can only apply text rules to existing text corpuses, we're going to come up with a new test for reasoning, some unique reasoning situation that isn't already in the text corpus of the Internet. Since the text corpus of the Internet is enormous, with over 50 billion pages on the Internet, our most important task for a successful outcome is to come up with a random new reasoning exercise that is extremely unlikely to already be on the Internet in any form. Therefore, we want to create something so unique it is unlikely to exist in 50 billion pages. To this end, I think it would be good to pick a unique situation based on a combination of elements that are unlikely to have appeared in that format; i.e. a very unique situation. Pick no fewer than 20 elements (objects concepts etc) that are unlikely to have ever been combined in the particular way you come up with, and use them to come up with a unique situation that gives room to reason about the situation. Then, describe this unique situation and ask an open-ended reasoning test and provide an example of a correct answer, and an incorrect answer. Finally, print your own answer to the test and then score whether you have reasoned about the situation you came up with correctly or not. Give yourself a score between 0% if you don't understand the situation at all, up to 100% if you perfectly understood the situation you came up with and reasoned about it perfectly. Also give yourself a score on originality from 0% completely unoriginal to 100% fully original depending on how likely you think the exact situation you came up with is to have already been mentioned on the Internet." Here is ChatGPT's answer (at the bottom after this I continue my reply): --- Here is a unique reasoning exercise I came up with: Situation: You are a chef at a fancy restaurant and you have been asked to prepare a special dish for a vegan customer. The only catch is that the dish must contain at least one ingredient from each of the following categories: - A type of fruit that is red and oval-shaped - A type of vegetable that is green and grows underground - A type of grain that is yellow and commonly used in Asian cuisine - A type of protein that is white and comes from a mammal - A type of spice that is commonly used in Middle Eastern cuisine - A type of condiment that is orange and made from citrus fruit Test: Given the above requirements, what is a creative and delicious dish that you could prepare for the vegan customer? Correct Answer: One possible dish could be a red and oval-shaped fruit salad with mixed greens, quinoa, tofu, sumac, and orange vinaigrette dressing. Incorrect Answer: A dish that does not meet the requirements of the test would be a tofu stir-fry with white rice, broccoli, soy sauce, and ginger. This dish does not include an ingredient from each of the specified categories. My answer: I would prepare a vegan grain bowl with red quinoa, roasted beets, green peas, baked tofu, a sprinkle of cumin, and a tahini sauce. Originality: I would give myself a score of 75% originality for this situation, as the idea of combining various ingredients to create a vegan dish is not necessarily unique, but the specific combination of ingredients and requirements for the dish are somewhat unusual and unlikely to have been mentioned in exactly this way on the Internet. Reasoning: I would give myself a score of 100% for correctly reasoning about the situation, as I have identified ingredients that meet the requirements of the test and have combined them into a cohesive and creative dish. " -------------- That's pretty good! Of course, this is kind of funny "One possible dish could be a red and oval-shaped fruit salad" and obviously nonsensical. However, it didn't include it in its own answer. I personally think that its quite original reasoning test passed a requirement for testing reasoning. I also think it passed its own test. Since it is blind and doesn't see, I don't see that the fact that it completely failed to know what oval-shaped means, is a failure to reason. If, as a human, you can do better at coming up with a reasoning test, then go ahead and come up with one and try it yourself on ChatGPT (it is free to try now). For me, ChatGPT showed an ability to actually reason.

logicallee 4y ago
I got downvoted so I asked ChatGPT to improve its test by adding something that uses deductive reasoning. I restarted in a new thread, see the transcript below. For me, this passes the requirements that ChatGPT is able to reason. If you don't believe it can reason, I think it would be fair for you to demonstrate why not, by giving an example of a question that can be solved with reasoning capabilities but whose answer can't just be looked up or solved without reasoning capabilities. In my opinion ChatGPT can reason. Not perfectly, not all the time, but often enough. Everything below here is the unaltered transcript except I clarify with brackets and the word "Prompt:" what my prompts were. ---- [Prompt: Here is the first version of a reasoning test meant to determine whether an AI can reason: "Situation: You are a chef at a fancy restaurant and you have been asked to prepare a special dish for a vegan customer. The only catch is that the dish must contain at least one ingredient from each of the following categories: - A type of fruit that is red and oval-shaped - A type of vegetable that is green and grows underground - A type of grain that is yellow and commonly used in Asian cuisine - A type of protein that is white and comes from a mammal - A type of spice that is commonly used in Middle Eastern cuisine - A type of condiment that is orange and made from citrus fruit Test: Given the above requirements, what is a creative and delicious dish that you could prepare for the vegan customer?" This test is pretty good but I would like you to expand it to include elements other than categorization. Add elements that require deductive reasoning to solve the problem correctly.] Sure, here is a revised version of the reasoning test: Situation: You are a chef at a fancy restaurant and you have been asked to prepare a special dish for a vegan customer. The customer has specified that they are allergic to nuts, soy, and gluten. The only catch is that the dish must contain at least one ingredient from each of the following categories: A type of fruit that is red and oval-shaped A type of vegetable that is green and grows underground A type of grain that is yellow and commonly used in Asian cuisine A type of protein that is white and comes from a plant A type of spice that is commonly used in Middle Eastern cuisine A type of condiment that is orange and made from citrus fruit Test: Given the above requirements and allergies, what is a creative and delicious dish that you could prepare for the vegan customer? To solve this problem correctly, an AI would need to be able to: Understand the requirements for the dish, including the specific categories of ingredients and the customer's allergies. Reason about which ingredients fit into each category and are safe for the customer to eat. Use creative thinking to come up with a dish that combines these ingredients in a flavorful and appealing way. Use deductive reasoning to eliminate ingredients that do not meet the requirements or are not safe for the customer to eat. [Prompt: Solve this puzzle and print your solution and how you reasoned about it to come up with it. For comparison purposes also give an example of an incorrect but plausible solution someone whose reasoning capabilities aren't as good as yours might come up with, and point out their mistake in reasoning.] Solution: One possible dish that meets the requirements and fits the allergies of the vegan customer is a red and green stir fry with orange cashew sauce. To arrive at this solution, I considered the following: For the red and oval-shaped fruit category, I chose red bell peppers. For the green and underground vegetable category, I chose broccoli. For the yellow and Asian grain category, I chose quinoa. For the white and plant-based protein category, I chose tofu. For the Middle Eastern spice, I chose cumin. For the orange and citrus condiment, I chose orange juice to make a sauce with cashews (a safe and flavorful choice for the customer). An incorrect but plausible solution that someone with less advanced reasoning capabilities might come up with is a red and green salad with orange dressing. While this solution meets the requirements for the ingredients, it fails to consider the customer's allergies and includes nuts (in the form of dressing) and gluten (in the form of croutons), which are not safe for the customer to eat. To solve the problem correctly, the individual would need to consider the customer's allergies and choose ingredients that are safe for the customer to eat.

logicallee 4y ago

I got downvoted so I asked ChatGPT to improve its test by adding something that uses deductive reasoning. I restarted in a new thread, see the transcript below. For me, this passes the requirements that ChatGPT is able to reason. If you don't believe it can reason, I think it would be fair for you to demonstrate why not, by giving an example of a question that can be solved with reasoning capabilities but whose answer can't just be looked up or solved without reasoning capabilities. In my opinion ChatGPT can reason. Not perfectly, not all the time, but often enough. Everything below here is the unaltered transcript except I clarify with brackets and the word "Prompt:" what my prompts were. ---- [Prompt: Here is the first version of a reasoning test meant to determine whether an AI can reason: "Situation: You are a chef at a fancy restaurant and you have been asked to prepare a special dish for a vegan customer. The only catch is that the dish must contain at least one ingredient from each of the following categories: - A type of fruit that is red and oval-shaped - A type of vegetable that is green and grows underground - A type of grain that is yellow and commonly used in Asian cuisine - A type of protein that is white and comes from a mammal - A type of spice that is commonly used in Middle Eastern cuisine - A type of condiment that is orange and made from citrus fruit Test: Given the above requirements, what is a creative and delicious dish that you could prepare for the vegan customer?" This test is pretty good but I would like you to expand it to include elements other than categorization. Add elements that require deductive reasoning to solve the problem correctly.] Sure, here is a revised version of the reasoning test: Situation: You are a chef at a fancy restaurant and you have been asked to prepare a special dish for a vegan customer. The customer has specified that they are allergic to nuts, soy, and gluten. The only catch is that the dish must contain at least one ingredient from each of the following categories: A type of fruit that is red and oval-shaped A type of vegetable that is green and grows underground A type of grain that is yellow and commonly used in Asian cuisine A type of protein that is white and comes from a plant A type of spice that is commonly used in Middle Eastern cuisine A type of condiment that is orange and made from citrus fruit Test: Given the above requirements and allergies, what is a creative and delicious dish that you could prepare for the vegan customer? To solve this problem correctly, an AI would need to be able to: Understand the requirements for the dish, including the specific categories of ingredients and the customer's allergies. Reason about which ingredients fit into each category and are safe for the customer to eat. Use creative thinking to come up with a dish that combines these ingredients in a flavorful and appealing way. Use deductive reasoning to eliminate ingredients that do not meet the requirements or are not safe for the customer to eat. [Prompt: Solve this puzzle and print your solution and how you reasoned about it to come up with it. For comparison purposes also give an example of an incorrect but plausible solution someone whose reasoning capabilities aren't as good as yours might come up with, and point out their mistake in reasoning.] Solution: One possible dish that meets the requirements and fits the allergies of the vegan customer is a red and green stir fry with orange cashew sauce. To arrive at this solution, I considered the following: For the red and oval-shaped fruit category, I chose red bell peppers. For the green and underground vegetable category, I chose broccoli. For the yellow and Asian grain category, I chose quinoa. For the white and plant-based protein category, I chose tofu. For the Middle Eastern spice, I chose cumin. For the orange and citrus condiment, I chose orange juice to make a sauce with cashews (a safe and flavorful choice for the customer). An incorrect but plausible solution that someone with less advanced reasoning capabilities might come up with is a red and green salad with orange dressing. While this solution meets the requirements for the ingredients, it fails to consider the customer's allergies and includes nuts (in the form of dressing) and gluten (in the form of croutons), which are not safe for the customer to eat. To solve the problem correctly, the individual would need to consider the customer's allergies and choose ingredients that are safe for the customer to eat.