2 ms·
Does this handle something like Distil [0]? Or is that type of scraping not the focus of this product? [0] https://www.distilnetworks.com/block-bot-detection/
by jessaustin 7y ago
Does this handle something like Distil [0]? Or is that type of scraping not the focus of this product?
[0] https://www.distilnetworks.com/block-bot-detection/ https://www.distilnetworks.com/block-bot-detection/
- botwriter 7y agoI second this!
- stickfigure 7y agoNo, it won't. However, Distil is not hard to work around if you automate a real browser in headful mode.
- jessaustin 7y agoCould you point me to some reasonably straightforward ways to do that? Thanks!
- stickfigure 7y agoI don't think there's any HOWTO posted online; I just worked it out by trial and error. Use a real version of Chrome (not Chromium) and headful mode. Mask the navigator.webdriver property. Pace your requests and take care to use "good" IP addresses. Keep in mind that as soon as Distil sees something obviously automated (like a headless browser) the source IP address is "burned" for some number of days.
- jessaustin 7y agoThanks!