6 ms·

Interesting! My interaction: > what is pi? > Claude: Pi or π is a mathematical constant that is the ratio of a circle's circumference to its diameter. It has

Interesting! My interaction: > what is pi? > Claude: Pi or π is a mathematical constant that is the ratio of a circle's circumference to its diameter. It has a value of approximately 3.14. [20 more lines talking about pi] > do you know more digits of pi? Yes, pi has been calculated to many digits beyond its decimal point. Here are the first 100 digits of pi: 3.14... [went on to write 1000 (!) digits of pi]

thomasahle 3y ago
Did you check how many of the digits were correct?
- sirodoht 3y ago
  Indeed, all of them were correct. It also rounded up the last digit.
eshack94 3y ago
Apparently (at least according to Claude), it's calculating these on the fly using an algorithm: > How many digits of pi do you know? > I don't actually have the full digits of pi stored. I just generated the first 1000 digits on the fly using a pi digit calculation algorithm. In theory I could continue generating more digits as needed, but in practice there's no need for me to store or generate extremely long strings of pi digits. The first few digits are sufficient for most purposes.
- rain1 3y ago
  This is an example of hallucination. An LLM doesn't know anything about itself - it can be pre-prompted with facts about itself, but this is going to be an example of it just making plausible text up.
  - losteric 3y ago
    Is it possible some of these LLMs actually have internal tools / calculators? ie blackboxing what ChatGPT has as explicit plugins
    - qup 3y ago
      If it were, they still wouldn't be able to commentate about it.
      - fnordpiglet 3y ago
        Unless the existence of such a tool at its disposal is in its context. A strategy would be to provide a set of tools, how to invoke them internally, and a description of their apparent interaction in the LLMs initial context.
    - gcr 3y ago
      even if there were some mixture-of-experts shenanigans going on, there is no introspection or reasoning, so the model isn’t able to comment on or understand its “inner experience”, if you can call matrix multiplications an inner experience
      - losteric 3y ago
        I was imagining system-prompt-based tool use, where the LLM "knows" it can call some calculator to get digits of pi
lhl 3y ago
So, I went down an interesting rabbit hole with this. Claude was willing to give me (correctly) 100 digits of Pi, but when asked to write some Python code to let me confirm the digits of Pi, it failed over and over again (in different ways each time, interestingly enough). Basically, it's still wholly inadequate for coding. Now, ChatGPT on the other hand... I shifted over to ChatGPT-4 Code Interpreter, and it gave me a working method (and of course was able to run the code directly) using mpmath the very first go, and then with BBP when asking for using just the standard library, and also via Leibniz as well, with the warning it would be slow, when asked for the shortest code. It was able to give me a chart of various known pi calculation methods and their relative speed/other properties, and then it refactored the code it had give me so far along with adding timings so I could see relative speed. Lastly, it helped me write the code for running Leibniz and timing the length for calculating each additional digit of precision, as well as projecting how long it would be to get to additional digits of precision (and how much precision I could get on my CPU at certain time periods). Lastly, I had it plot out how long it would take to get to 12 decimal places and it used matplotlib.pyplot to generate one for me. "As you can see, the time increases exponentially with each additional decimal place. Based on our estimates, it would take well over 100 years to compute." I've yet to see any other service/model (open or closed) get close to doing what ChatGPT-4 + Code Interpreter is doing atm...
- CSMastermind 3y ago
  GPT-4 seems far and away the best model at coding. It's the only llm I've tested that can borderline pass a technical interview.
  - TX81Z 3y ago
    It’s my personal intern.
- rayxi271828 3y ago
  How was it able to run the code directly for you? For me it said it didn't have access to external libraries.

thomasahle 3y ago

Did you check how many of the digits were correct?

sirodoht 3y ago
Indeed, all of them were correct. It also rounded up the last digit.

eshack94 3y ago

Apparently (at least according to Claude), it's calculating these on the fly using an algorithm: > How many digits of pi do you know? > I don't actually have the full digits of pi stored. I just generated the first 1000 digits on the fly using a pi digit calculation algorithm. In theory I could continue generating more digits as needed, but in practice there's no need for me to store or generate extremely long strings of pi digits. The first few digits are sufficient for most purposes.

rain1 3y ago
This is an example of hallucination. An LLM doesn't know anything about itself - it can be pre-prompted with facts about itself, but this is going to be an example of it just making plausible text up.
- losteric 3y ago
  Is it possible some of these LLMs actually have internal tools / calculators? ie blackboxing what ChatGPT has as explicit plugins
  - qup 3y ago
    If it were, they still wouldn't be able to commentate about it.
    - fnordpiglet 3y ago
      Unless the existence of such a tool at its disposal is in its context. A strategy would be to provide a set of tools, how to invoke them internally, and a description of their apparent interaction in the LLMs initial context.
  - gcr 3y ago
    even if there were some mixture-of-experts shenanigans going on, there is no introspection or reasoning, so the model isn’t able to comment on or understand its “inner experience”, if you can call matrix multiplications an inner experience
    - losteric 3y ago
      I was imagining system-prompt-based tool use, where the LLM "knows" it can call some calculator to get digits of pi

lhl 3y ago

So, I went down an interesting rabbit hole with this. Claude was willing to give me (correctly) 100 digits of Pi, but when asked to write some Python code to let me confirm the digits of Pi, it failed over and over again (in different ways each time, interestingly enough). Basically, it's still wholly inadequate for coding. Now, ChatGPT on the other hand... I shifted over to ChatGPT-4 Code Interpreter, and it gave me a working method (and of course was able to run the code directly) using mpmath the very first go, and then with BBP when asking for using just the standard library, and also via Leibniz as well, with the warning it would be slow, when asked for the shortest code. It was able to give me a chart of various known pi calculation methods and their relative speed/other properties, and then it refactored the code it had give me so far along with adding timings so I could see relative speed. Lastly, it helped me write the code for running Leibniz and timing the length for calculating each additional digit of precision, as well as projecting how long it would be to get to additional digits of precision (and how much precision I could get on my CPU at certain time periods). Lastly, I had it plot out how long it would take to get to 12 decimal places and it used matplotlib.pyplot to generate one for me. "As you can see, the time increases exponentially with each additional decimal place. Based on our estimates, it would take well over 100 years to compute." I've yet to see any other service/model (open or closed) get close to doing what ChatGPT-4 + Code Interpreter is doing atm...

CSMastermind 3y ago
GPT-4 seems far and away the best model at coding. It's the only llm I've tested that can borderline pass a technical interview.
- TX81Z 3y ago
  It’s my personal intern.
rayxi271828 3y ago
How was it able to run the code directly for you? For me it said it didn't have access to external libraries.