2 ms·
No they're supposed to allow scraping and information aggregation. That's the essence of the web: it's all text, crawlable, machine-readable (sort of) and parse
by spwa4 1y ago
No they're supposed to allow scraping and information aggregation. That's the essence of the web: it's all text, crawlable, machine-readable (sort of) and parseable. Feel free to block ddos'es.
- bayindirh 1y agoFeel free to crawl paywalled sites and republish them with discoverable links. Also after starting the crawl, you can read about Aaron Swartz while waiting.