2 ms·
You don’t crawl unless you have to, Twitter[1], Facebook, WordPress.com[2] and other big services have a firehose you can apply to and get real-time changes. If
by withinboredom 6y ago
You don’t crawl unless you have to, Twitter[1], Facebook, WordPress.com[2] and other big services have a firehose you can apply to and get real-time changes. If you’re crawling the web, you’re probably doing it wrong or only servicing a particular niche.
[1] https://developer.twitter.com/en/docs/twitter-api/v1/tweets/sample-realtime/overview/decahose https://developer.twitter.com/en/docs/twitter-api/v1/tweets/...
[2] https://developer.wordpress.com/docs/firehose/ https://developer.wordpress.com/docs/firehose/
- ddorian43 6y agoYou missed the pricing. This is what you're doing wrong. And you missed the cloudflare part too.