I just keep a pair of waterproof headphones permanently attached to my swim goggles and keep it in my gym stuff. It's not nearly as big a hassle as you think -

I just keep a pair of waterproof headphones permanently attached to my swim goggles and keep it in my gym stuff. It's not nearly as big a hassle as you think - just wrap it up tightly once and you have no cords dangling or anything while you swim.