There are decades of research in information retrieval that can easily be applied to this problem. Before Google showed up with PageRank, many algorithms were b

There are decades of research in information retrieval that can easily be applied to this problem. Before Google showed up with PageRank, many algorithms were based on properties of a given document and the historical properties of the set (example, TF-IDF). You don't need to drink from a firehose. The firehose will be throttled and filtered to given you the useful information that you want. This is where the future of real-time search lies. Disclosure: I'm CTO of Collecta, one of these real-time search companies.