I see what you're getting at. My (similarly hand wavey) perspective is that constructions using category theory end up defining natural numbers as categories of

I see what you're getting at. My (similarly hand wavey) perspective is that constructions using category theory end up defining natural numbers as categories of sets with cardinality n. What I was really getting at is, how do you define your relation for the category in such a way that isn't pathological ("n is the category of sets with cardinality n except all such sets containing n")? Alternatively, what are you selecting as the objects for your category if not sets of a given cardinality? To be clear, I consider a lot of discussion about the foundations of set theory (and paradoxes thereof) to be pretty navel-gazey.