Specialized search

Hi, I am looking for a way to find content from web based upon certain keywords. The idea is to help users explore new content based upon their interests. DemandMedia.com and twitter's new discover feature does this. Please let me know If you know about any apis or websites that I can use for this purpose. My last option will be to write a specialized crawler. Much Thanks.