It seems like you are unfamiliar with technology before the past few centuries. Beyond astrology (which was important!), people used these theoretical tools fro

It seems like you are unfamiliar with technology before the past few centuries. Beyond astrology (which was important!), people used these theoretical tools from geometry and number theory for time keeping, calendars, cartography, navigation, surveying (you mentioned), city planning, architecture, construction and analysis of machines, hydrology, water distribution, crafts such as metalworking and carpentry, accounting, optics, music, any kind of engineering involving measurements, .... Obviously mathematicians were also excited about numbers and shapes for their own sake, and some of their activities (e.g. trying to square the circle or trisect an angle using compass/straightedge) were more like puzzles or games than engineering. But that doesn't mean the broad bulk of mathematics (even deductive proof-based mathematics) was practically worthless. Some of the most famous mathematicians of antiquity were also engineers. Fluency with prime numbers per se (and prime factorization) is pretty useful when you are trying to do substantial amounts of (especially mental) arithmetic. I'm not sure which "taxpayers" you are thinking about, but establishing an effective tax system and doing tax assessment and accounting is one example where having a more robust understanding of elementary arithmetic and number theory is pretty valuable. The development of the number and metrological systems and work on number theory in ancient Mesopotamia had quite a lot to do with tax accounting.