You're making the same mistake as those who talk about space travel usually make: you're considering the people who stay behind! This is irrelevant. The gains

You're making the same mistake as those who talk about space travel usually make: you're considering the people who stay behind! This is irrelevant. The gains obtained from travel benefit the traveler, not those who stay behind. If you want to go into the galaxy, just put up a nuclear reactor in space, and with the energy obtained, produce a permanent thrust, as small as it may be (given the mass of your space ship including a whole nuclear reactor, and a self contained biotope). When you kee a permanent thrust, you reach relativistic speed soon enough, so that a travel to the closests stars (or the planets detected around them), can be done in a reasonable time, FOR THE TRAVELLER! With an acceleration of about 1g, you can get to the closest star (4 light years) in about one year. Of course, for those who stay behind, time goes faster, and it'll be ten years (plus four years for the light to get back) for them to learn that you've reached your destination. But who cares about those losers who stayed behind? Well, it's the same here. Who cares about those losers who didn't travel back in time. Let's have them keep their lousy timeline! At least, the TIME TRAVELLER gets to change his own future!