3 ms·

Pretty sure you can't get GPT-4 to do 8 digit multiplication with any prompt. For what it's worth, I'm not even sure if chain of thought provides much value to

Pretty sure you can't get GPT-4 to do 8 digit multiplication with any prompt. For what it's worth, I'm not even sure if chain of thought provides much value to GPT-4. The RLHF it went through seems to have encouraged more logical thinking already.

morkalork 3y ago
Now you have me wondering if you could prompt it to solve it step by step, using the same method an elementary student would do on paper.
- seanhunter 3y ago
  You can. One thing that I've found fun is to prompt it for some maths problems without solutions, then I provide solutions and for any that I get wrong ask it to explain my mistakes.
seanhunter 3y ago
The deeplearning.com prompt engineering course contains an example where GPT4 solves a problem with chain of thought and fails without. The prompt is Determine if the student's solution is correct or not. Question: I'm building a solar power installation and I need help working out the financials. - Land costs $100 / square foot - I can buy solar panels for $250 / square foot - I negotiated a contract for maintenance that will cost me a flat $100k per year, and an additional $10 / square foot What is the total cost for the first year of operations as a function of the number of square feet. Student's Solution: Let x be the size of the installation in square feet. Costs: 1. Land cost: 100x 2. Solar panel cost: 250x 3. Maintenance cost: 100,000 + 100x Total cost: 100x + 250x + 100,000 + 100x = 450x + 100,000 GPT-4 will say the student is correct even though the answer includes an error (they should be multiplying by 10x not 100x). If you change the prompt to include chain of thought as follows it notices the error Your task is to determine if the student's solution is correct or not. To solve the problem do the following: - First, work out your own solution to the problem. - Then compare your solution to the student's solution \ and evaluate if the student's solution is correct or not. Don't decide if the student's solution is correct until you have done the problem yourself. Use the following format: Question: ``` question here ``` Student's solution: ``` student's solution here ``` Actual solution: ``` steps to work out the solution and your solution here ``` Is the student's solution the same as actual solution \ just calculated: ``` yes or no ``` Student grade: ``` correct or incorrect ``` Question: ``` I'm building a solar power installation and I need help \ working out the financials. - Land costs $100 / square foot - I can buy solar panels for $250 / square foot - I negotiated a contract for maintenance that will cost \ me a flat $100k per year, and an additional $10 / square \ foot What is the total cost for the first year of operations \ as a function of the number of square feet. ``` Student's solution: ``` Let x be the size of the installation in square feet. Costs: 1. Land cost: 100x 2. Solar panel cost: 250x 3. Maintenance cost: 100,000 + 100x Total cost: 100x + 250x + 100,000 + 100x = 450x + 100,000 ``` Actual solution: As I say this same prompt "fools" GPT3.5 and 4.
seanhunter 3y ago
Btw I think you're not correct about the 8 digit multiplication. I tried: please could you multiply 12345678 by 87654321. Give your answer in steps. Each step should calculate one column at a time from right to left remembering to carry any digits which overflowed in a previous step. At the end add a final step adding the values up. And it gave me the following, which I believe to be correct: Sure, I'll be doing a long multiplication here, step by step, column by column from right to left. For brevity, I'll write the numbers to be multiplied as: A: 12345678 B: 87654321 Step 1: Multiply the 1st digit of B by A: 1 * 12345678 = 12345678 Step 2: Multiply the 2nd digit of B by A, remember to shift one position to the left: 2 * 12345678 = 24691356 (shift one position to the left, i.e., append a 0 at the end) Step 3: Multiply the 3rd digit of B by A, remember to shift two positions to the left: ... skip some of the answer because you get the gist.... 7 * 12345678 = 86419746 (shift six positions to the left, i.e., append six 0s at the end) Step 8: Multiply the 8th digit of B by A, remember to shift seven positions to the left: 8 * 12345678 = 98765424 (shift seven positions to the left, i.e., append seven 0s at the end) Now, the last step is to add all these results together: 12345678 246913560 3703703400 49382712000 617283900000 7407406800000 86419746000000 987654240000000 = 1082152022374638 So, the result of 12345678 multiplied by 87654321 is 1082152022374638.
- usaar333 3y ago
  You might be getting lucky because that sequence is trained on (it's not very random). I repeated this experiment with bing and verified it could get your solution. But it failed on more random input: please could you multiply 92631924 by 41920318. Give your answer in steps. Each step should calculate one column at a time from right to left remembering to carry any digits which overflowed in a previous step. At the end add a final step adding the values up. Got 3880341653910192 with intermediate steps being broken.

morkalork 3y ago

Now you have me wondering if you could prompt it to solve it step by step, using the same method an elementary student would do on paper.

seanhunter 3y ago
You can. One thing that I've found fun is to prompt it for some maths problems without solutions, then I provide solutions and for any that I get wrong ask it to explain my mistakes.

seanhunter 3y ago

The deeplearning.com prompt engineering course contains an example where GPT4 solves a problem with chain of thought and fails without. The prompt is Determine if the student's solution is correct or not. Question: I'm building a solar power installation and I need help working out the financials. - Land costs $100 / square foot - I can buy solar panels for $250 / square foot - I negotiated a contract for maintenance that will cost me a flat $100k per year, and an additional $10 / square foot What is the total cost for the first year of operations as a function of the number of square feet. Student's Solution: Let x be the size of the installation in square feet. Costs: 1. Land cost: 100x 2. Solar panel cost: 250x 3. Maintenance cost: 100,000 + 100x Total cost: 100x + 250x + 100,000 + 100x = 450x + 100,000 GPT-4 will say the student is correct even though the answer includes an error (they should be multiplying by 10x not 100x). If you change the prompt to include chain of thought as follows it notices the error Your task is to determine if the student's solution is correct or not. To solve the problem do the following: - First, work out your own solution to the problem. - Then compare your solution to the student's solution \ and evaluate if the student's solution is correct or not. Don't decide if the student's solution is correct until you have done the problem yourself. Use the following format: Question: ``` question here ``` Student's solution: ``` student's solution here ``` Actual solution: ``` steps to work out the solution and your solution here ``` Is the student's solution the same as actual solution \ just calculated: ``` yes or no ``` Student grade: ``` correct or incorrect ``` Question: ``` I'm building a solar power installation and I need help \ working out the financials. - Land costs $100 / square foot - I can buy solar panels for $250 / square foot - I negotiated a contract for maintenance that will cost \ me a flat $100k per year, and an additional $10 / square \ foot What is the total cost for the first year of operations \ as a function of the number of square feet. ``` Student's solution: ``` Let x be the size of the installation in square feet. Costs: 1. Land cost: 100x 2. Solar panel cost: 250x 3. Maintenance cost: 100,000 + 100x Total cost: 100x + 250x + 100,000 + 100x = 450x + 100,000 ``` Actual solution: As I say this same prompt "fools" GPT3.5 and 4.

seanhunter 3y ago

Btw I think you're not correct about the 8 digit multiplication. I tried: please could you multiply 12345678 by 87654321. Give your answer in steps. Each step should calculate one column at a time from right to left remembering to carry any digits which overflowed in a previous step. At the end add a final step adding the values up. And it gave me the following, which I believe to be correct: Sure, I'll be doing a long multiplication here, step by step, column by column from right to left. For brevity, I'll write the numbers to be multiplied as: A: 12345678 B: 87654321 Step 1: Multiply the 1st digit of B by A: 1 * 12345678 = 12345678 Step 2: Multiply the 2nd digit of B by A, remember to shift one position to the left: 2 * 12345678 = 24691356 (shift one position to the left, i.e., append a 0 at the end) Step 3: Multiply the 3rd digit of B by A, remember to shift two positions to the left: ... skip some of the answer because you get the gist.... 7 * 12345678 = 86419746 (shift six positions to the left, i.e., append six 0s at the end) Step 8: Multiply the 8th digit of B by A, remember to shift seven positions to the left: 8 * 12345678 = 98765424 (shift seven positions to the left, i.e., append seven 0s at the end) Now, the last step is to add all these results together: 12345678 246913560 3703703400 49382712000 617283900000 7407406800000 86419746000000 987654240000000 = 1082152022374638 So, the result of 12345678 multiplied by 87654321 is 1082152022374638.

usaar333 3y ago
You might be getting lucky because that sequence is trained on (it's not very random). I repeated this experiment with bing and verified it could get your solution. But it failed on more random input: please could you multiply 92631924 by 41920318. Give your answer in steps. Each step should calculate one column at a time from right to left remembering to carry any digits which overflowed in a previous step. At the end add a final step adding the values up. Got 3880341653910192 with intermediate steps being broken.