If your site has a substantial amount of crawlers, that’s an opportunity to offer an API subscription. No one wants to “crawl” your data, they just want your d

If your site has a substantial amount of crawlers, that’s an opportunity to offer an API subscription. No one wants to “crawl” your data, they just want your data. No software developer or PM ever said “it’d be great if this broke every couple of days and we had to scramble to figure out how to fix it” or “I’d love to jump through a bunch of hoops to read that H1” Bot detection and crawling is an arms race. Bots will always win. I’ve droned on about this here in the past. [1] Anecdotal evidence: Previous co-founder and software architect of a “crawler” that crawled ~17000 store and deal sites a day (like BestBuy, Gap, RMN). We circumvented all bot detection and were only busted and c&d’d once. By CouponCabin of all people. [1] https://news.ycombinator.com/item?id=16182405 https://news.ycombinator.com/item?id=16182405