4 ms·
Feeding Reddit to Hadoop (w/ Clojure)
- jedberg 17y agoInteresting article, and it also explains who was pounding us with traffic. :) Protip: If you are going to scrape a website, put some sleeps into your loop so the person who runs the website doesn't curse your name while their automatic rate limiters kick in and block you anyway.
- philjackson 17y agoHe actually did mention using sleep, in a sort of "do as I say, not as I do" kind of way :)