On the other hand, Earth is a very dense planet (in part due to the accidental formation of the Moon), so bigger planets might still have a lower gravity on ave

On the other hand, Earth is a very dense planet (in part due to the accidental formation of the Moon), so bigger planets might still have a lower gravity on average. Also, if they spin faster, getting into orbit will be easier. Maybe Earth is indeed at a comparatively bad place in the gravity spectrum of habitable planets. Getting things up there is difficult enough that we can't have big payloads, but it's still easy enough to justify not building a self-sufficient infrastructure out there.