34 ms·
I have one question, are they consuming other companies search API, let’s say Twitter or are they crawling and aggregate their own data? Either way it’s a great
by okeumeni 17y ago
I have one question, are they consuming other companies search API, let’s say Twitter or are they crawling and aggregate their own data? Either way it’s a great job!
- ajasver 17y agoWe actually crawl and index all the data ourselves. That's how we identify the most popular content.
- okeumeni 17y agoNice Work!
- hachiya 17y agoIs this truly crawling the sites, or querying each site's search service? e.g. for Twitter, are you crawling all the Twitter users' URLs (seems difficult to do - how do you find new accounts' URLs?), or are you fetching results from search.twitter.com ?
- zaidf 17y agoI don't think crawling and querying are mutually exclusive ideas.
- dannyr 17y agoYeah. I'm curious if they developed something similar to Twitter's search functionality or they are using Twitter's Trends feed. In its current form, most of the results are from Twitter which doesn't really differentiate it much from Twitter Search.
- ajasver 17y agoWe take the raw data streams (e.g. a data mining feed from Twitter) and process them, ourselves.