I don't see any reason why information theory has to be limited to encoded information in bits. It deals generally with a quantity p(x), with no qualification f

I don't see any reason why information theory has to be limited to encoded information in bits. It deals generally with a quantity p(x), with no qualification for what p(x) is. You could, for example, tokenize words and figure out how complex the language used is, given some measure over the distribution of words in the english language. Similarly, you could go higher and work with characters, and figure out the surprise factor of using bears, but again you would need some model or measure of the distribution of character types in english literature. Figuring out the entropy of a coin toss is fairly trivial, but figuring out the entropy of, say, a story book, requires coming up with a model of the distribution over story books, which is the hard part. In that regard, information theory is quite powerful and gives us the tools to do all of that, it's just that the models are highly non-trivial, or otherwise missing.