4 ms·
The scrapers should use some discretion. There are some rather obvious optimizations. Content that is not changing is less likely to change in the future.
by rubiquity 8mo ago
The scrapers should use some discretion. There are some rather obvious optimizations. Content that is not changing is less likely to change in the future.
- JohnTHaller 8mo agoThey don't care. It's the reason they ignore robots.txt and change up their useragents when you specifically block them.