Well, arithmetic problems are good simple benchmarks for multi step problem solving, but then you get the weirdos who claim you shouldn't let an LLM do the dirt

Well, arithmetic problems are good simple benchmarks for multi step problem solving, but then you get the weirdos who claim you shouldn't let an LLM do the dirty work. When someone uses addition as a benchmark, they are not necessarily in need of a calculator, they are in need of an LLM that demonstrates the skills that it shares in common with arithmetic and the actual task. For example, if you are given the task to calculate ten additions, you must know how to split the problem into subproblems until you can compose a series of skills to actually solve the problem. If the model knows a different operation like subtracting, it will know how to do so even though it hasn't seen an example involving ten subtractions. Apply this to logical operators, set operators, modus ponens, etc and you will get very far even with very rudimentary skills. The point here is that it should be easy for a machine to verify that the LLM does indeed have to think for itself.