3 ms·

Did you do a cost estimate for the "Wizard of Oz" solution of having real people with real browsers (and a script to pull data from the site?) Might have been w

• kbenson 9y ago
  It was actually two changes, one which required a much more intensive request regime because of a public caching (SOLR) system change, and then the much more aggressive scraping detection. The first caused us to change from requesting data 78 times an hour (items changed within 2 minutes requested every minute, 6 minutes of changes every 5 minutes, 11 minutes of changes every 10 minutes for overlapping coverage) to many thousands of checks an hour for much less accurate information. In the end, we ended up doing very targeted checks and much less accuracy for different classes of items. Having people actually do the checking just wouldn't be feasible for our size and resources (very small company, <10 employees), even through mechanical turk (I suspect).