I think your worry comes from a materialist viewpoint. You perceive the universe as only being material, and then, looking out at the infinite material expanse,

I think your worry comes from a materialist viewpoint. You perceive the universe as only being material, and then, looking out at the infinite material expanse, you are rendered distraught by the seeming lack of life 'out there'. My view is that the material universe is only an illusion, that Earth is the centre of 'the universe' (it's immaterial that we orbit around the sun) and that life and consciousness only 'exist' on this planet. It's the 'stage' where we play out our lives; from whence we come and to whence we go is the bigger 'mystery' and one more rewarding to ponder on, then wonder about life 'out there'.