3 ms·

This still doesn't get at the point, with this example you've effectively constructed a prompt along the lines of: "Note: A calculator is available upon request

This still doesn't get at the point, with this example you've effectively constructed a prompt along the lines of: "Note: A calculator is available upon request wink, here's how you'd use it: ... Now, what's the eighth root of 4819387574?" Of course the model will use the calculator you've explicitly informed it of. The article is meant to be a critique of claims that LLMs are "intelligent," when, despite knowing their math limitations, don't generally answer "You'd be better off punching this into a calculator" when asked a problem

IanCal 2y ago
How have I told it there's a calculator? All I've given it is the ability to search for tools and enable ones it wants. > Of course the model will use the calculator you've explicitly informed it of I didn't. I also gave it no system prompt pushing it to always use tools or anything. It searches for tools with a query "calculator math root" and is given a list of things that includes a calculator. It picks the calculator, then it uses it. The code and trace are right there.
- crackrook 2y ago
  What do you imagine `tools` is doing if not a system prompt? I suggest reading the documentation: https://docs.anthropic.com/en/docs/build-with-claude/tool-use#tool-use-system-prompt https://docs.anthropic.com/en/docs/build-with-claude/tool-us... The vast majority of features that Anthropic and OpenAI ship are just clever ways of building system prompts
  - IanCal 2y ago
    It's told it can search for tools and it can enable them. It is not told initially that there's a calculator, it asks for one.
    - crackrook 2y ago
      I see, that clarifies things for me, it's not quite like the example I gave then. Even so, doesn't informing the model of the fact that some "tools" are available, immediately before asking it a math problem (that would be virtually impossible for a human to answer precisely), seem like a pretty big hint that it should inquire if a calculator is available? Here's what I get from sonnet in response to the plain user-prompt "What is the eighth root of 4819387574?" """ Let me solve this step by step. To find the 8th root of 4819387574: 1) The 8th root of 4819387574 means finding x where x⁸ = 4819387574 2) This is a large number, but it's a perfect 8th power. 3) One way to approach this is to find factors: 4819387574 = 13⁸ 4) Therefore, the 8th root of 4819387574 is 13. To verify: 13⁸ = 13 × 13 × 13 × 13 × 13 × 13 × 13 × 13 = 4819387574 The answer is 13. """

IanCal 2y ago

How have I told it there's a calculator? All I've given it is the ability to search for tools and enable ones it wants. > Of course the model will use the calculator you've explicitly informed it of I didn't. I also gave it no system prompt pushing it to always use tools or anything. It searches for tools with a query "calculator math root" and is given a list of things that includes a calculator. It picks the calculator, then it uses it. The code and trace are right there.

crackrook 2y ago
What do you imagine `tools` is doing if not a system prompt? I suggest reading the documentation: https://docs.anthropic.com/en/docs/build-with-claude/tool-use#tool-use-system-prompt https://docs.anthropic.com/en/docs/build-with-claude/tool-us... The vast majority of features that Anthropic and OpenAI ship are just clever ways of building system prompts
- IanCal 2y ago
  It's told it can search for tools and it can enable them. It is not told initially that there's a calculator, it asks for one.
  - crackrook 2y ago
    I see, that clarifies things for me, it's not quite like the example I gave then. Even so, doesn't informing the model of the fact that some "tools" are available, immediately before asking it a math problem (that would be virtually impossible for a human to answer precisely), seem like a pretty big hint that it should inquire if a calculator is available? Here's what I get from sonnet in response to the plain user-prompt "What is the eighth root of 4819387574?" """ Let me solve this step by step. To find the 8th root of 4819387574: 1) The 8th root of 4819387574 means finding x where x⁸ = 4819387574 2) This is a large number, but it's a perfect 8th power. 3) One way to approach this is to find factors: 4819387574 = 13⁸ 4) Therefore, the 8th root of 4819387574 is 13. To verify: 13⁸ = 13 × 13 × 13 × 13 × 13 × 13 × 13 × 13 = 4819387574 The answer is 13. """