You still need to be a bit careful, because what you're describing as entropy isn't how much information is in it, but rather how much you didn't already know.

You still need to be a bit careful, because what you're describing as entropy isn't how much information is in it, but rather how much you didn't already know. Hence random noise has the maximal 'information content' but isn't useful in the slightest. And really I wouldn't worry too much about finding some mathematical justification for everything. It's an easy way to get caught up in technicalities for something that you already knew.