4 ms·
A very dishonest company whose scrapers are tough to block. They don't honor robots.txt and tell you that you have to add in special verbiage to get them to st
by shrubble 6y ago
A very dishonest company whose scrapers are tough to block. They don't honor robots.txt and tell you that you have to add in special verbiage to get them to stop. They hop IPs from different ranges so they are difficult to range-ban.
- judge2020 6y agoYep - they get blocked a lot by CF (probably for not honoring robots and crawling random websites for marketing purposes) yet a lot of people use it and wonder why it’s blocked: https://community.cloudflare.com/search?q=Semrush https://community.cloudflare.com/search?q=Semrush