2 ms·
> It's not about bypassing access restrictions. Yes. It is. You've just made an arbitrary choice not to define it as such.
by messe 7mo ago
> It's not about bypassing access restrictions.
Yes. It is. You've just made an arbitrary choice not to define it as such.
- andrew_zhong 7mo agoI will add a PR to enforce robots.txt before the actual scraping.
- messe 7mo agoOr just follow web standards and define and publish your User-Agent header, so that people can block that as needed. You're creating the wrong kind of value. I really hope your company fails, as its success implies a failure of the web in general. I wish you the best success outside of your current endeavour.