I agree with you that what I'm saying is that maths is ultimately a kind of language that we use to describe the world. I wouldn't agree with calling it "just"

I agree with you that what I'm saying is that maths is ultimately a kind of language that we use to describe the world. I wouldn't agree with calling it "just" language or "like English". It's different in that consistency is strictly enforced, such that simple rules can lead to rich emergent structures. That's what I was getting at with the game of life metaphor. So, if you follow my argument, the fact that there are prime numbers is not 'intrinsic to mathematics', it's the other way around -- we use rules that lead to rich structures, because those provide useful, consistent abstractions to talk about things. Prime numbers are among those rich structures. To respond to a sibling comment that seems somewhat related -- The fact that we can calculate some physical constants with high precision is not a counter argument to that. What we are doing is fitting a mathematical abstraction to data; once we have done that, we can find more physical truths by assuming consistency and performing logical inference. The fact that a constant can be calculated then just means that we have a consistent theory for the physical structures from which that constant arises.