That has its own set of problems, namely data sparsity and resulting in high dimensionality vectors, although like you say does give you a little bit of context

That has its own set of problems, namely data sparsity and resulting in high dimensionality vectors, although like you say does give you a little bit of context. I'm not sure that bag of anything is a good solution.