3 ms·
I would say how you're scraping differs from say how Google, a search engine, scrapes. I'm not sure there is a way in robots.txt to define for each use? Knowing
by loceng 13y ago
I would say how you're scraping differs from say how Google, a search engine, scrapes. I'm not sure there is a way in robots.txt to define for each use? Knowing the data in a structured way, but then allowing it to be displayed in full off-site is quite different than using the scraped data for linking into a website.
- thatthatis 13y agoBut robots.txt provides minimums: don't scrape this page, don't refresh more than once every x, these crawlers are allowed this access, etc.
- pranade 13y agoFeedback from webmasters is really helpful for us. We want to make sure we're making data available via API responsibly, so would love to hear your suggestions/ thoughts as we define a scalable solution.