I think what they are saying is that if we can't say that multiplication is repeated addition due to the reason you propose, then addition itself is ill-defined

I think what they are saying is that if we can't say that multiplication is repeated addition due to the reason you propose, then addition itself is ill-defined as a concept, which is a bigger problem. That by using your argument we move the whole debate one level down the fundamental operations and now you no longer have a way to simply explain addition or subtraction.