3 ms·
My scraping is part of a transactional B2B service, not a high traffic social or B2C thing, so it's a different set of problems than those who want their hands
by bdcravens 14y ago
My scraping is part of a transactional B2B service, not a high traffic social or B2C thing, so it's a different set of problems than those who want their hands on Twitter data. These are Fortune 100's, so if I can bring down their site, they have bigger problems. :-)
- olefoo 14y agoI wouldn't make that assumption. Do check return codes and load times, and back off if you see issues. If these sites are business partners/suppliers you have a lot to lose if things go wrong. It's worth it to develop your relationship with the business owners of the services you're touching in the correspondent organizations. And do set a User-Agent string that declares who you are and provides a link for information; if you are doing business with them, it should be on a basis of honesty.