The non-handwavy answer is that mathematics is defined by a set of axioms through which all mathematics follow via logic and defining new structures / relations

The non-handwavy answer is that mathematics is defined by a set of axioms through which all mathematics follow via logic and defining new structures / relations. If there were no 6, given the same set of axioms we could simply define something representing the same concept of 6 with a different symbol, because the concept of 6 is consistent with our axioms. Dividing by 0 is inconsistent with our axioms, so even if we define x = 1/0, we would be able to arrive at conclusions which contradict our axioms.