3 ms·
I don't think it's unethical to pull down a copy of public information. If you pull too fast it might be considered rude (heavy load on the server). That's why
by snowwrestler 11y ago
I don't think it's unethical to pull down a copy of public information. If you pull too fast it might be considered rude (heavy load on the server). That's why some sites reflexively block all scrapers, hence the rotating IP feature. Hopefully this tool is rate-limited so it's not rude.
In terms of copyright, what matters is what you do with the scrape. If you scrape a public website for personal use, it's no different from just browsing it for personal use. If you try to republish the content for your own benefit, you'll run afoul of copyright law.
- chinathrow 11y agoIf you disobey robots.txt it's unethical and rude.
- snowwrestler 11y agoI agree, but running a scraper does not necessarily mean disobeying robots.txt.