One aspect you miss in your analogy is that when you include the time dimension, the probability of a collision is proportional to swept volume (and thus speed)

One aspect you miss in your analogy is that when you include the time dimension, the probability of a collision is proportional to swept volume (and thus speed), rather than static volume. So while the space is really large, really fast objects can have substantially large swept volumes.