3 ms·
One possible problem with this is if one crawler gets a link to the website and then a parallel crawler gets a link to the website as well. Enough parallel cra
by MattGaiser 6y ago
One possible problem with this is if one crawler gets a link to the website and then a parallel crawler gets a link to the website as well.
Enough parallel crawlers and you hit the captcha.
So the crawlers would need to be maintaining a record of sites and associated cookies. I don’t see why it couldn’t be done, but definitely adds to complexity.