3 ms·
Can you given an example in the wild?
by greglindahl 7y ago
Can you given an example in the wild?
- jedberg 7y agoI was actually trying to find an example when I made my initial comment, but was unable to. It's been a long time since I did web scraping. Since then there are a lot more frameworks that help you build a website (and a correspondingly sane robots.txt), so there may not be as many as before.