4 ms·
This is perfect. I was looking for something exactly like this. How are you reading the posts? How often?
by chime 15y ago
This is perfect. I was looking for something exactly like this. How are you reading the posts? How often?
- zemaj 15y agoThanks! We've got a fairly mature spidering architecture, so we're just using that to scrape the posts at scale. It's possible Google will block us, but we're pretty careful to rate limit and load balance across IPs. Plus, you know Google's business is built on scraping :) They don't seem to mind other apps that do it, so I figure we're moderately safe. Once there's an API, we'll use that. We check every 5 mins if you post frequently, progressively slowing out at around once every 20 mins currently. If traffic builds up we'll slow it down to max at around 1 hour.
- evilswan 15y agoNice job. Looking forward to the API-enabled version.