7 ms·

>>A good sign that LLMs have reached human level for a much wider class >> of problems will be if they start proving theorems using methods that, like much of t

>>A good sign that LLMs have reached human level for a much wider class >> of problems will be if they start proving theorems using methods that, like much of the very best human mathematics, are new and surprising but that with hindsight come to seem beautiful and natural. I must be taking crazy pills and the AGI surely will pass me by... But TODAY, middle August 2026...And in the context of testing and evaluating the capabilities of current SOTA models to implement an Agentic application for job search, here is some simple inhouse built evals I run today, since I don´t trust LLM vendors published benchmarks... Models tested: GPT-5.6 Sol in Extra High mode and Opus 4.8 Max. TASK REQUEST: Clear, not too long not too short prompt, for LLMs to go out and research freelance consulting gigs for one specific IT domain, and in one specific country in Europe, including maybe opportunities driven from temp agencies based in geographically close countries. RESULT: Models go out, fetch the data, and completely misunderstand the task...offering on first results, permanent roles instead of freelance, and based on the country where the agencies are, not in the one it was request for. Think for example IT jobs in Ireland, while freelance agency in London. ANALYSIS: No intelligence I can call it shown by models, adding cognitive effort for human in the loop to detect subtle factors, and therefore totally useless for agentic app...Best practices would be I guess to add agents on top of agents but although in the p95 of cases that will reduce the errors...for the remaining 5% that could have hallucinations or logic hallucinations like these ones, compounding on top of other logic hallucinations. I dont care about the theorems being proven. At the end we will found out what most mathematicians were doing, was just exploring the same combinatorial and abstraction patterns. And because of that I am sure LLMs will make mince meat of a lot of mathematical domains. But right now, what we call intelligence is not existing where it matters, and Ed Zitron is right its a parlour trick.

m348e912 2mo ago
>TASK REQUEST: Clear, not too long not too short prompt, for LLMs to go out and research freelance consulting gigs for one specific IT domain, and in one specific country in Europe, including maybe opportunities driven from temp agencies based in geographically close countries. As a human, not an LLM, I could interpret "including maybe opportunities driven from temp agencies based in geographically close countries" as meaning "including opportunities in nearby countries outside of Ireland" (that happen to be driven by temp agencies). Before writing off LLM as simply a "stochastic parrot" or a "parlour trick" remember it can't read your mind, not yet anyway.
- tcp_handshaker 2mo ago
  I am describing the contents of the prompt that was not the prompt. The prompt was very clear to the model that some freelance opportunities in country A, the only one in consideration could be available via agencies in country B and C. And it was a clear prompt. So what happen is a prompt said for example, find freelance opportunities in Ireland but keep in mind some of these might be available via temp agencies in London. If you offer me not freelance but permanent roles, and not in Ireland in London...that is a logic failure. Its this type of complexity with the normal world, that these SOTA constructions so badly fail at, and so spectacularly fail at the margins... despite maxing all benchmarks...Parlour trick.
  - someguyiguess 2mo ago
    Based on the rest of your writing I’m going to assume that the prompt was the problem.
    - coldtea 2mo ago
      He was perfectly clear in both cases. If a human misunderstood this, they'd be a dumb human.
      - fn-mote 2mo ago
        For perspective, I agree with the GP. The writing is not perfectly clear. We don’t have enough evidence to know if that was part of the problem.
    - tcp_handshaker 2mo ago
      Keep deluding yourself, unless you work for an LLM provider... "Frontier LLMs Still Struggle with Simple Reasoning Tasks" https://arxiv.org/abs/2507.07313 https://arxiv.org/abs/2507.07313 "General365: Benchmarking General Reasoning in Large Language Models Across Diverse and Challenging Tasks" https://arxiv.org/abs/2604.11778 https://arxiv.org/abs/2604.11778 "...General365, a benchmark specifically designed to assess general reasoning in LLMs. By restricting background knowledge to a K-12 level, General365 explicitly decouples reasoning from specialized expertise. The benchmark comprises 365 seed problems and 1,095 variant problems across eight categories, ensuring both high difficulty and diversity. Evaluations across 26 leading LLMs reveal that even the top-performing model achieves only 62.8% accuracy, in stark contrast to the near-perfect performances of LLMs in math and physics benchmarks..."
criley2 2mo ago
[flagged]
- tcp_handshaker 2mo ago
  Its the tools. Sell those RSUs while they last. Most of these are 2026.... Frontier LLMs Still Struggle with Simple Reasoning Tasks - https://arxiv.org/abs/2507.07313 https://arxiv.org/abs/2507.07313 General365: Benchmarking General Reasoning in Large Language Models Across Diverse and Challenging Tasks - https://arxiv.org/abs/2604.11778 https://arxiv.org/abs/2604.11778 LLMEval-Logic: A Solver-Verified Chinese Benchmark for Logical Reasoning of LLMs with Adversarial Hardening - https://arxiv.org/abs/2605.19597 https://arxiv.org/abs/2605.19597 LogicGraph: Benchmarking Multi-Path Logical Reasoning via Neuro-Symbolic Generation and Verification - https://arxiv.org/abs/2602.21044 https://arxiv.org/abs/2602.21044 Blind-Spots-Bench: Evaluating Blind Spots in Multimodal Models - https://arxiv.org/abs/2607.08317 https://arxiv.org/abs/2607.08317 Vision-Language Models Lag Human Performance on Physical Dynamics and Intent Reasoning - https://arxiv.org/abs/2601.01547 https://arxiv.org/abs/2601.01547 Do Vision-Language Models Understand 3D Scenes or Just Catalogue Objects? - https://arxiv.org/abs/2605.20448 https://arxiv.org/abs/2605.20448 The Reversal Curse: LLMs Trained on “A is B” Fail to Learn “B is A” - https://arxiv.org/abs/2309.12288 https://arxiv.org/abs/2309.12288 Large Language Model Reasoning Failures - https://arxiv.org/abs/2602.06176 https://arxiv.org/abs/2602.06176
  - criley2 2mo ago
    What a sloppy reply. You've hijacked a thread on mathematics first to complain that your incompetent attempt to use ChatGPT to find a job failed, but it seems now that this was a ruse to instead begin arguments unrelated to the article at all where you just spam arxiv links you've never read to "prove" that AI is a scam. This comes across, frankly, as either Dunning-Kruger (classic illusory superiority), or potentially as mental illness. The slop dump is highly reminiscent of how a schizophrenic friend of mine communicates. Do you really think slopping down a bunch of random arxiv links "proves" that AI is a scam and you're so smart and everyone else isn't? Most awkwardly for your arxiv slop -- most of this is irrelevant to your central claim, and you've missed papers that are much closer. For example your LogicGraph paper: "Can't exhaustively enumerate all minimal proofs" is not "can't distinguish Ireland from London". Or your "Do VLMs Understand 3D Scenes..." is nothing more than citation decoration, completely irrelevant to our discussion. Or your "Frontier LLMs Still Struggle with Simple Reasoning Tasks" which is potentially your pièce de résistance, it supports brittle multi-step constraint handling, but isn't remotely an eval of a modern web-search agent. For example, VibeSearchBench would have been far more relevant to your claims https://arxiv.org/html/2605.27882v1 https://arxiv.org/html/2605.27882v1 (but still obviously not proof that AI is "a parlour trick") Going further: my point that we need to discuss your beginner's approach to the harness is substantiated clearly here: https://arxiv.org/html/2605.23950v1 https://arxiv.org/html/2605.23950v1 Finally, failure to exhibit human-like generality is not evidence of absence of intelligence. It is evidence that whatever cognitive machinery LLMs possess has a very different error distribution from ours. Your General365, LLMEval-Logic and the Reversal Curse are actually fascinating evidence for that jaggedness, rather than proof of your claim that AI is a scam.
CodeCompost 2mo ago
LLMs have no sense of geography. They measure distances between parts of words, not distances between parts of world.
- foco_tubi 2mo ago
  “Canada is north of the United States” has a higher probability of correctness than “Canada is east of Greenland.”
fragmede 2mo ago
> TASK REQUEST: Clear, not too long not too short prompt Why do you think there's such a thing as too long for an LLM prompt? You'll run into context window limits at some point, but the more verbose you are with what you ask of it, the better the results will be.
- root-parent 2mo ago
  >> t the more verbose you are with what you ask of it, the better the results will be. Trivially falsifiable: "Context Length Alone Hurts LLM Performance Despite Perfect Retrieval" https://aclanthology.org/2025.findings-emnlp.1264/ https://aclanthology.org/2025.findings-emnlp.1264/ "Large Language Models Can Be Easily Distracted by Irrelevant Context" https://arxiv.org/abs/2302.00093 https://arxiv.org/abs/2302.00093
simdezimon 2mo ago
https://claude.ai/share/f0c5c3c9-1882-44b1-b30e-fb427a0df472 https://claude.ai/share/f0c5c3c9-1882-44b1-b30e-fb427a0df472 I don't know your prompt and setup, but my claude had no problems doing that task. The search index isn't live, so it can't find current gigs, but that is a tooling problem.
vhantz 2mo ago
Stop wasting your time and use actual code for most of what you give an LLM to do. Make them write the code even. Anything that can be verified mechanically should be code. Only use LLMs to fill in the gaps where things are fuzzy. Don't fall for the idea that those harnesses are general purpose, make your own fit to your task with the guards and verification steps you need. Make the LLM create the harness even. There is no amount of markdown that can make a machine generating plausible text generate truthful text, it just happens to be truthful because of what it was trained on. Nothing coming out of an LLM should be taken at face value. The propaganda about LLMs being intelligent and able to "reason" is only serving the companies selling you tokens to waste on "prompt engineering".
- deleted 2mo ago
  [deleted]
- root-parent 2mo ago
  The irony about your comment is that, this is probably the most likely opinion and most consensual around many technological practitioners. But the mathematicians here in this thread, are having a hard time with these clearly dumb models, doing so well in proving theorems in their domains :-)
  - antonvs 2mo ago
    Writing code and proving theories are flip sides of the same coin. See e.g. Phil Wadler’s “Proofs are Programs”: https://philarchive.org/rec/WADPAP https://philarchive.org/rec/WADPAP The statement should probably really be “programs are proofs” since it’s difficult to make it a true bidirectional isomorphism, which is why the underlying principle is properly called the Curry-Howard(-Lambek) correspondence: https://en.wikipedia.org/wiki/Curry%E2%80%93Howard_correspondence https://en.wikipedia.org/wiki/Curry%E2%80%93Howard_correspon... The point is that the fact that LLMs are good at writing code directly implies that they should also be good at certain classes of mathematical proof.
    - inigyou 2mo ago
groundzeros2015 2mo ago
This is an off topic rant unrelated to mathematical ability which is a closed problem often with complete logical information.
Anon1096 2mo ago
Reading the responses to your comment the discussion would be a lot more productive if you shared your logs (preferably several of different top models since that's what you're claiming) where LLMs fail at this. Not very useful for people to go back and forth speculating on what you could have asked and with what formulation. As it stands for me simdezimon's logs are pretty definitive that there shouldn't be any problem for current capabilities agents to solve the task.
hansvm 2mo ago
I'm not sure why exactly, but I've heard the same from every single person using LLMs for anything related to jobs. The posting says it needs at least a B.S., and the LLM denies an applicant because they have an M.S. The posting thinks it needs 3yrs of work experience in XYZ technology, and it won't add it to the candidate's list because it doesn't have the context that the HR/LLM filter on the posting adds a bunch of nonsensical requests or that some other combination of skills makes the candidate stand out above and beyond that missing "requirement." And so on. The quality is quite poor. On the other end of it, something like 80% of resumes I receive right now are clearly hallucinated -- referencing accomplishments that are copy-pasted from the novel-to-our-company thing in the job description a candidate will be working on, usually claiming they did XYZ at big tech a decade before the thing existed, or similarly with languages and skills. The resume "tailoring" process just manufactures lies rather than tailoring actual experience to the actual job.

m348e912 2mo ago

>TASK REQUEST: Clear, not too long not too short prompt, for LLMs to go out and research freelance consulting gigs for one specific IT domain, and in one specific country in Europe, including maybe opportunities driven from temp agencies based in geographically close countries. As a human, not an LLM, I could interpret "including maybe opportunities driven from temp agencies based in geographically close countries" as meaning "including opportunities in nearby countries outside of Ireland" (that happen to be driven by temp agencies). Before writing off LLM as simply a "stochastic parrot" or a "parlour trick" remember it can't read your mind, not yet anyway.

tcp_handshaker 2mo ago
I am describing the contents of the prompt that was not the prompt. The prompt was very clear to the model that some freelance opportunities in country A, the only one in consideration could be available via agencies in country B and C. And it was a clear prompt. So what happen is a prompt said for example, find freelance opportunities in Ireland but keep in mind some of these might be available via temp agencies in London. If you offer me not freelance but permanent roles, and not in Ireland in London...that is a logic failure. Its this type of complexity with the normal world, that these SOTA constructions so badly fail at, and so spectacularly fail at the margins... despite maxing all benchmarks...Parlour trick.
- someguyiguess 2mo ago
  Based on the rest of your writing I’m going to assume that the prompt was the problem.
  - coldtea 2mo ago
    He was perfectly clear in both cases. If a human misunderstood this, they'd be a dumb human.
    - fn-mote 2mo ago
      For perspective, I agree with the GP. The writing is not perfectly clear. We don’t have enough evidence to know if that was part of the problem.
  - tcp_handshaker 2mo ago
    Keep deluding yourself, unless you work for an LLM provider... "Frontier LLMs Still Struggle with Simple Reasoning Tasks" https://arxiv.org/abs/2507.07313 https://arxiv.org/abs/2507.07313 "General365: Benchmarking General Reasoning in Large Language Models Across Diverse and Challenging Tasks" https://arxiv.org/abs/2604.11778 https://arxiv.org/abs/2604.11778 "...General365, a benchmark specifically designed to assess general reasoning in LLMs. By restricting background knowledge to a K-12 level, General365 explicitly decouples reasoning from specialized expertise. The benchmark comprises 365 seed problems and 1,095 variant problems across eight categories, ensuring both high difficulty and diversity. Evaluations across 26 leading LLMs reveal that even the top-performing model achieves only 62.8% accuracy, in stark contrast to the near-perfect performances of LLMs in math and physics benchmarks..."

criley2 2mo ago

[flagged]

tcp_handshaker 2mo ago
Its the tools. Sell those RSUs while they last. Most of these are 2026.... Frontier LLMs Still Struggle with Simple Reasoning Tasks - https://arxiv.org/abs/2507.07313 https://arxiv.org/abs/2507.07313 General365: Benchmarking General Reasoning in Large Language Models Across Diverse and Challenging Tasks - https://arxiv.org/abs/2604.11778 https://arxiv.org/abs/2604.11778 LLMEval-Logic: A Solver-Verified Chinese Benchmark for Logical Reasoning of LLMs with Adversarial Hardening - https://arxiv.org/abs/2605.19597 https://arxiv.org/abs/2605.19597 LogicGraph: Benchmarking Multi-Path Logical Reasoning via Neuro-Symbolic Generation and Verification - https://arxiv.org/abs/2602.21044 https://arxiv.org/abs/2602.21044 Blind-Spots-Bench: Evaluating Blind Spots in Multimodal Models - https://arxiv.org/abs/2607.08317 https://arxiv.org/abs/2607.08317 Vision-Language Models Lag Human Performance on Physical Dynamics and Intent Reasoning - https://arxiv.org/abs/2601.01547 https://arxiv.org/abs/2601.01547 Do Vision-Language Models Understand 3D Scenes or Just Catalogue Objects? - https://arxiv.org/abs/2605.20448 https://arxiv.org/abs/2605.20448 The Reversal Curse: LLMs Trained on “A is B” Fail to Learn “B is A” - https://arxiv.org/abs/2309.12288 https://arxiv.org/abs/2309.12288 Large Language Model Reasoning Failures - https://arxiv.org/abs/2602.06176 https://arxiv.org/abs/2602.06176
- criley2 2mo ago
  What a sloppy reply. You've hijacked a thread on mathematics first to complain that your incompetent attempt to use ChatGPT to find a job failed, but it seems now that this was a ruse to instead begin arguments unrelated to the article at all where you just spam arxiv links you've never read to "prove" that AI is a scam. This comes across, frankly, as either Dunning-Kruger (classic illusory superiority), or potentially as mental illness. The slop dump is highly reminiscent of how a schizophrenic friend of mine communicates. Do you really think slopping down a bunch of random arxiv links "proves" that AI is a scam and you're so smart and everyone else isn't? Most awkwardly for your arxiv slop -- most of this is irrelevant to your central claim, and you've missed papers that are much closer. For example your LogicGraph paper: "Can't exhaustively enumerate all minimal proofs" is not "can't distinguish Ireland from London". Or your "Do VLMs Understand 3D Scenes..." is nothing more than citation decoration, completely irrelevant to our discussion. Or your "Frontier LLMs Still Struggle with Simple Reasoning Tasks" which is potentially your pièce de résistance, it supports brittle multi-step constraint handling, but isn't remotely an eval of a modern web-search agent. For example, VibeSearchBench would have been far more relevant to your claims https://arxiv.org/html/2605.27882v1 https://arxiv.org/html/2605.27882v1 (but still obviously not proof that AI is "a parlour trick") Going further: my point that we need to discuss your beginner's approach to the harness is substantiated clearly here: https://arxiv.org/html/2605.23950v1 https://arxiv.org/html/2605.23950v1 Finally, failure to exhibit human-like generality is not evidence of absence of intelligence. It is evidence that whatever cognitive machinery LLMs possess has a very different error distribution from ours. Your General365, LLMEval-Logic and the Reversal Curse are actually fascinating evidence for that jaggedness, rather than proof of your claim that AI is a scam.

CodeCompost 2mo ago

LLMs have no sense of geography. They measure distances between parts of words, not distances between parts of world.

foco_tubi 2mo ago
“Canada is north of the United States” has a higher probability of correctness than “Canada is east of Greenland.”

fragmede 2mo ago

> TASK REQUEST: Clear, not too long not too short prompt Why do you think there's such a thing as too long for an LLM prompt? You'll run into context window limits at some point, but the more verbose you are with what you ask of it, the better the results will be.

root-parent 2mo ago
>> t the more verbose you are with what you ask of it, the better the results will be. Trivially falsifiable: "Context Length Alone Hurts LLM Performance Despite Perfect Retrieval" https://aclanthology.org/2025.findings-emnlp.1264/ https://aclanthology.org/2025.findings-emnlp.1264/ "Large Language Models Can Be Easily Distracted by Irrelevant Context" https://arxiv.org/abs/2302.00093 https://arxiv.org/abs/2302.00093

simdezimon 2mo ago

https://claude.ai/share/f0c5c3c9-1882-44b1-b30e-fb427a0df472 https://claude.ai/share/f0c5c3c9-1882-44b1-b30e-fb427a0df472 I don't know your prompt and setup, but my claude had no problems doing that task. The search index isn't live, so it can't find current gigs, but that is a tooling problem.

vhantz 2mo ago

Stop wasting your time and use actual code for most of what you give an LLM to do. Make them write the code even. Anything that can be verified mechanically should be code. Only use LLMs to fill in the gaps where things are fuzzy. Don't fall for the idea that those harnesses are general purpose, make your own fit to your task with the guards and verification steps you need. Make the LLM create the harness even. There is no amount of markdown that can make a machine generating plausible text generate truthful text, it just happens to be truthful because of what it was trained on. Nothing coming out of an LLM should be taken at face value. The propaganda about LLMs being intelligent and able to "reason" is only serving the companies selling you tokens to waste on "prompt engineering".

deleted 2mo ago
[deleted]
root-parent 2mo ago
The irony about your comment is that, this is probably the most likely opinion and most consensual around many technological practitioners. But the mathematicians here in this thread, are having a hard time with these clearly dumb models, doing so well in proving theorems in their domains :-)
- antonvs 2mo ago
  Writing code and proving theories are flip sides of the same coin. See e.g. Phil Wadler’s “Proofs are Programs”: https://philarchive.org/rec/WADPAP https://philarchive.org/rec/WADPAP The statement should probably really be “programs are proofs” since it’s difficult to make it a true bidirectional isomorphism, which is why the underlying principle is properly called the Curry-Howard(-Lambek) correspondence: https://en.wikipedia.org/wiki/Curry%E2%80%93Howard_correspondence https://en.wikipedia.org/wiki/Curry%E2%80%93Howard_correspon... The point is that the fact that LLMs are good at writing code directly implies that they should also be good at certain classes of mathematical proof.
  - inigyou 2mo ago

groundzeros2015 2mo ago

This is an off topic rant unrelated to mathematical ability which is a closed problem often with complete logical information.

Anon1096 2mo ago

Reading the responses to your comment the discussion would be a lot more productive if you shared your logs (preferably several of different top models since that's what you're claiming) where LLMs fail at this. Not very useful for people to go back and forth speculating on what you could have asked and with what formulation. As it stands for me simdezimon's logs are pretty definitive that there shouldn't be any problem for current capabilities agents to solve the task.

hansvm 2mo ago

I'm not sure why exactly, but I've heard the same from every single person using LLMs for anything related to jobs. The posting says it needs at least a B.S., and the LLM denies an applicant because they have an M.S. The posting thinks it needs 3yrs of work experience in XYZ technology, and it won't add it to the candidate's list because it doesn't have the context that the HR/LLM filter on the posting adds a bunch of nonsensical requests or that some other combination of skills makes the candidate stand out above and beyond that missing "requirement." And so on. The quality is quite poor. On the other end of it, something like 80% of resumes I receive right now are clearly hallucinated -- referencing accomplishments that are copy-pasted from the novel-to-our-company thing in the job description a candidate will be working on, usually claiming they did XYZ at big tech a decade before the thing existed, or similarly with languages and skills. The resume "tailoring" process just manufactures lies rather than tailoring actual experience to the actual job.