5 ms·

I don't think there are any up-to-date leaderboards, but models absolutely degrade in performance the more context they're dealing with. https://wandb.ai/byyou

I don't think there are any up-to-date leaderboards, but models absolutely degrade in performance the more context they're dealing with. https://wandb.ai/byyoung3/ruler_eval/reports/How-to-evaluate-the-true-context-length-of-your-LLM-using-RULER---VmlldzoxNDE0OTA0OQ https://wandb.ai/byyoung3/ruler_eval/reports/How-to-evaluate... >Gpt-5-mini records 0.87 overall judge accuracy at 4k [context] and falls to 0.59 at 128k. And Llama 4 Scout claimed a 10 million token context window but in practice its performance on query tasks drops below 20% accuracy by 32k tokens.

mg 11mo ago
That makes me wonder if we could simply test this by letting the LLM add or multiply a long list of numbers? Here is an experiment: https://www.gnod.com/search/#q=%23%20Calcuate%20the%20below%20number.%20Do%20not%20use%20a%20calculator.%20Do%20it%20in%20your%20head.%0A%0A(1.1%20%2B%202.5)%20*%20(7.1%20%2B%202.5)%20*%20(3.3%20%2B%204.4)%20*%20(12.3%20-%201.4)%20*%20(2.3%20-%201.1)%20*%20(5%20%2B%201)%20*%20(7.1%20%2B%202.5)%20*%20(3.3%20%2B%204.4)%20*%20(12.3%20-%201.4)%20*%20(2.3%20-%201.1) https://www.gnod.com/search/#q=%23%20Calcuate%20the%20below%... The correct answer: Correct: 20,192,642.460942328 Here is what I got from different models on the first try: ChatGPT: 20,384,918.24 Perplexity: 20,000,000 Google: 25,167,098.4 Mistral: 200,000,000 Grok: Timed out after 300s of thinking
- jarek83 11mo ago
  Isn't that LLMs are not designed to do calculations?
  - mg 11mo ago
    Neither are humans.
    - cuu508 11mo ago
      But humans can still do it.
  - cluckindan 11mo ago
    They are not LMMs, after all…
- gcanyon 11mo ago
  > Do not use a calculator. Do it in your head. You wouldn't ask a human to do that, why would you ask an LLM to? I guess it's a way to test them, but it feels like the world record for backwards running: interesting, maybe, but not a good way to measure, like, anything about the individual involved.
- throwuxiytayq 11mo ago
  I’m starting to find it unreasonably funny how people always want language models to multiply numbers for some reason. Every god damn time. In every single HN thread. I think my sanity might be giving out.
  - solatic 11mo ago
    A model, no, but an agent with a calculator tool? Then there's the question of why not just build the calculator tool into the model?
- KristoAI 11mo ago
  [flagged]
- KristoAI 11mo ago
  Since grok 4 fast got this answer correct so quickly, I decided to test more. Tested this on the new hidden model of ChatGPT called Polaris Alpha: Answer: $20,192,642.460942336$ Current gpt-5 medium reasoning says: After confirming my calculations, the final product (P) should be (20,192,642.460942336) Claude Sonnet 4.5 says: “29,596,175.95 or roughly 29.6 million” Claude haiku 4.5 says: ≈20,185,903 GLM 4.6 says: 20,171,523.725593136 I’m going to try out Grok 4 fast on some coding tasks at this point to see if it can create functions properly. Design help is still best on GPT-5 at this exact moment.

mg 11mo ago

That makes me wonder if we could simply test this by letting the LLM add or multiply a long list of numbers? Here is an experiment: https://www.gnod.com/search/#q=%23%20Calcuate%20the%20below%20number.%20Do%20not%20use%20a%20calculator.%20Do%20it%20in%20your%20head.%0A%0A(1.1%20%2B%202.5)%20*%20(7.1%20%2B%202.5)%20*%20(3.3%20%2B%204.4)%20*%20(12.3%20-%201.4)%20*%20(2.3%20-%201.1)%20*%20(5%20%2B%201)%20*%20(7.1%20%2B%202.5)%20*%20(3.3%20%2B%204.4)%20*%20(12.3%20-%201.4)%20*%20(2.3%20-%201.1) https://www.gnod.com/search/#q=%23%20Calcuate%20the%20below%... The correct answer: Correct: 20,192,642.460942328 Here is what I got from different models on the first try: ChatGPT: 20,384,918.24 Perplexity: 20,000,000 Google: 25,167,098.4 Mistral: 200,000,000 Grok: Timed out after 300s of thinking

jarek83 11mo ago
Isn't that LLMs are not designed to do calculations?
- mg 11mo ago
  Neither are humans.
  - cuu508 11mo ago
    But humans can still do it.
- cluckindan 11mo ago
  They are not LMMs, after all…
gcanyon 11mo ago
> Do not use a calculator. Do it in your head. You wouldn't ask a human to do that, why would you ask an LLM to? I guess it's a way to test them, but it feels like the world record for backwards running: interesting, maybe, but not a good way to measure, like, anything about the individual involved.
throwuxiytayq 11mo ago
I’m starting to find it unreasonably funny how people always want language models to multiply numbers for some reason. Every god damn time. In every single HN thread. I think my sanity might be giving out.
- solatic 11mo ago
  A model, no, but an agent with a calculator tool? Then there's the question of why not just build the calculator tool into the model?
KristoAI 11mo ago
[flagged]
KristoAI 11mo ago
Since grok 4 fast got this answer correct so quickly, I decided to test more. Tested this on the new hidden model of ChatGPT called Polaris Alpha: Answer: $20,192,642.460942336$ Current gpt-5 medium reasoning says: After confirming my calculations, the final product (P) should be (20,192,642.460942336) Claude Sonnet 4.5 says: “29,596,175.95 or roughly 29.6 million” Claude haiku 4.5 says: ≈20,185,903 GLM 4.6 says: 20,171,523.725593136 I’m going to try out Grok 4 fast on some coding tasks at this point to see if it can create functions properly. Design help is still best on GPT-5 at this exact moment.