I think a principled mathematical way to say it is that the multiplicative structure of a field is a group object in the monoidal category of pointed sets and t

I think a principled mathematical way to say it is that the multiplicative structure of a field is a group object in the monoidal category of pointed sets and the smash product (with 0 as the point). So the multiplicative inverse of 0 really is 0.