3 ms·
What user agent does it use and do you obey robots.txt? * apparently not as I tested it with LinkedIn.com which blocks most non-google/bing bots and it crawls
by altdataseller 2y ago
What user agent does it use and do you obey robots.txt?
* apparently not as I tested it with LinkedIn.com which blocks most non-google/bing bots and it crawls it OK
- super256 2y ago> FireCrawl is built to navigate common web scraping challenges, including reverse proxies, rate limits, and caching They probably ignore robots.txt