4 ms·

We actually crawl and index all the data ourselves. That's how we identify the most popular content.

• okeumeni 17y ago
  Nice Work!
• hachiya 17y ago
  Is this truly crawling the sites, or querying each site's search service? e.g. for Twitter, are you crawling all the Twitter users' URLs (seems difficult to do - how do you find new accounts' URLs?), or are you fetching results from search.twitter.com ?
  • zaidf 17y ago
    I don't think crawling and querying are mutually exclusive ideas.
  • dannyr 17y ago
    Yeah. I'm curious if they developed something similar to Twitter's search functionality or they are using Twitter's Trends feed. In its current form, most of the results are from Twitter which doesn't really differentiate it much from Twitter Search.
  • ajasver 17y ago
    We take the raw data streams (e.g. a data mining feed from Twitter) and process them, ourselves.