3 ms·

This took 1:53 in o3-mini https://chatgpt.com/share/679d310d-6064-8010-ba78-6bd5ed3360d3 https://chatgpt.com/share/679d310d-6064-8010-ba78-6bd5ed3360... The 4

This took 1:53 in o3-mini https://chatgpt.com/share/679d310d-6064-8010-ba78-6bd5ed3360d3 https://chatgpt.com/share/679d310d-6064-8010-ba78-6bd5ed3360... The 4o model without using the Python tool https://chatgpt.com/share/679d32bd-9ba8-8010-8f75-2f26a792e0d3 https://chatgpt.com/share/679d32bd-9ba8-8010-8f75-2f26a792e0... Trying to get accurate results with the paid version of 4o with the Python interpreter. https://chatgpt.com/share/679d31f3-21d4-8010-9932-7ecadd0b870f https://chatgpt.com/share/679d31f3-21d4-8010-9932-7ecadd0b87... The share link doesn’t show the output for some reason. But it did work correctly. I don’t know whether the ages are correct. I was testing whether it could handle ordering I have no idea what conclusion I should draw from this besides depending on the use case, 4o may be better with “tools” if you know your domain where you are using it. Tools are relatively easy to implement with LangChain or the native OpenAI SDK.

margalabargala 2y ago
The 4o model's output is blatantly wrong. I'm not going to look up if it's the order or the ages that are incorrect, but: 36. Abraham Lincoln – 52 years, 20 days (1861) 37. James Garfield – 49 years, 105 days (1881) 38. Lyndon B. Johnson – 55 years, 87 days (1963) Basically everything after #15 in the list is scrambled.
- scarface_74 2y ago
  That was the point. The 4o model without using Python was wrong. The o3 model worked correctly without needing an external tool
BeetleB 2y ago
I would not expect any LLM to get this right. I think people have too high expectations for it. Now if you asked it to write a Python program to list them in order, and have it enter all the names, birthdays, and year elected in a list to get the program to run - that's more reasonable.
- scarface_74 2y ago
  The “o” models get the order right. DeepSeek also gets the order right. It doesn’t show on the share link. But it actually outputs the list correctly from the built in Python interpreter. For some things, ChatGPT 4o will automatically use its Python runtime
  - BeetleB 2y ago
    That some models get it right is irrelevant. In general, if your instructions require computation, it's safer to assume it won't get it right and will hallucinate.
    - scarface_74 2y ago
      The reasoning models all do pretty good at math. Have you tried them? This is something I threw together with o3-mini https://chatgpt.com/share/679d5305-5f04-8010-b5c4-61c31e79b248 https://chatgpt.com/share/679d5305-5f04-8010-b5c4-61c31e79b2... ChatGPT 4o doesn’t even try to do the math internally and uses its built in Python interpreter. (The [_>] link is to the Python code) https://chatgpt.com/share/679d54fe-0104-8010-8f1e-9796a08cf930 https://chatgpt.com/share/679d54fe-0104-8010-8f1e-9796a08cf9... DeepSeek handles the same problem just as well using the reasoning technique. Of course ChatGPT 4o went completely off the rails without using its Python interpreter https://chatgpt.com/share/679d5692-96a0-8010-8624-b1eb091270cb https://chatgpt.com/share/679d5692-96a0-8010-8624-b1eb091270... (The break down that it got right was using Python even though I told it not to)

margalabargala 2y ago

The 4o model's output is blatantly wrong. I'm not going to look up if it's the order or the ages that are incorrect, but: 36. Abraham Lincoln – 52 years, 20 days (1861) 37. James Garfield – 49 years, 105 days (1881) 38. Lyndon B. Johnson – 55 years, 87 days (1963) Basically everything after #15 in the list is scrambled.

scarface_74 2y ago
That was the point. The 4o model without using Python was wrong. The o3 model worked correctly without needing an external tool

BeetleB 2y ago

I would not expect any LLM to get this right. I think people have too high expectations for it. Now if you asked it to write a Python program to list them in order, and have it enter all the names, birthdays, and year elected in a list to get the program to run - that's more reasonable.

scarface_74 2y ago
The “o” models get the order right. DeepSeek also gets the order right. It doesn’t show on the share link. But it actually outputs the list correctly from the built in Python interpreter. For some things, ChatGPT 4o will automatically use its Python runtime
- BeetleB 2y ago
  That some models get it right is irrelevant. In general, if your instructions require computation, it's safer to assume it won't get it right and will hallucinate.
  - scarface_74 2y ago
    The reasoning models all do pretty good at math. Have you tried them? This is something I threw together with o3-mini https://chatgpt.com/share/679d5305-5f04-8010-b5c4-61c31e79b248 https://chatgpt.com/share/679d5305-5f04-8010-b5c4-61c31e79b2... ChatGPT 4o doesn’t even try to do the math internally and uses its built in Python interpreter. (The [_>] link is to the Python code) https://chatgpt.com/share/679d54fe-0104-8010-8f1e-9796a08cf930 https://chatgpt.com/share/679d54fe-0104-8010-8f1e-9796a08cf9... DeepSeek handles the same problem just as well using the reasoning technique. Of course ChatGPT 4o went completely off the rails without using its Python interpreter https://chatgpt.com/share/679d5692-96a0-8010-8624-b1eb091270cb https://chatgpt.com/share/679d5692-96a0-8010-8624-b1eb091270... (The break down that it got right was using Python even though I told it not to)